I have a very high density virtualized environment with containers, so I'm trying to make each container really small. "Really small" means 87 MB on base Ubuntu 14.04 (Trusty Tahr) without breaking up the package manager compatibility.
So I use LVM as a backing storage for my containers and recently I found very strange numbers. Here they are.
Let's create a 100 MiB (yeah, power of 2) logical volume.
sudo lvcreate -L100M -n test1 /dev/purgatory
I'd like to check the size, so I issue sudo lvs --units k
test1 purgatory -wi-a---- 102400.00k
Sweet, this is really 100 MiB.
Now let's make an ext4 filesystem. And of course, we remember -m 0 parameter, which prevents space waste.
sudo mkfs.ext4 -m 0 /dev/purgatory/test1
mke2fs 1.42.9 (4-Feb-2014)
Filesystem label=
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
Stride=0 blocks, Stripe width=0 blocks
25688 inodes, 102400 blocks
0 blocks (0.00%) reserved for the super user
First data block=1
Maximum filesystem blocks=67371008
13 block groups
8192 blocks per group, 8192 fragments per group
1976 inodes per group
Superblock backups stored on blocks:
8193, 24577, 40961, 57345, 73729
Allocating group tables: done
Writing inode tables: done
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done
Sweet and clean. Mind the block size – our logical volume is small, so mkfs.ext4 decided to make a 1 KiB sized block, not the usual 4 KiB.
Now we will mount it.
sudo mount /dev/purgatory/test1 /mnt/test1
And let's call df without parameters (we would like to see 1 KiB-blocks)
/dev/mapper/purgatory-test1 95054 1550 91456 2% /mnt/test1
Wait, oh shi~
We have 95054 blocks total. But the device itself has 102400 blocks of 1 KiB. We have only 92.8% of our storage. Where are my blocks, man?
Let's look at it on a real block device. A have a 16 GiB virtual disk, 16777216 blocks of 1K, but only 15396784 blocks are in df output. 91.7%, what is it?
Now follows the investigation (spoiler: no results)
-
Filesystem could begin not at the beginning of the device. This is strange, but possible. Luckily, ext4 has magic bytes, let's check their presence.
sudo hexdump -C /dev/purgatory/test1 | grep "53 ef"
This shows superblock:
00000430 a9 10 e7 54 01 00 ff ff 53 ef 01 00 01 00 00 00 |...T....S.......|
Hex 430 = Dec 1072, so somewhere after first kilobyte. Looks reasonable, ext4 skips first 1024 bytes for oddities like VBR, etc.
- This is journal!
No, it is not. Journal take space from Available if df output.
- Oh, we have dump2fs and could check the sizes there!
… a lot of greps …
sudo dumpe2fs /dev/purgatory/test1 | grep "Free blocks"
Ouch.
Free blocks: 93504
Free blocks: 3510-8192
Free blocks: 8451-16384
Free blocks: 16385-24576
Free blocks: 24835-32768
Free blocks: 32769-40960
Free blocks: 41219-49152
Free blocks: 53249-57344
Free blocks: 57603-65536
Free blocks: 65537-73728
Free blocks: 73987-81920
Free blocks: 81921-90112
Free blocks: 90113-98304
Free blocks: 98305-102399
And we have another number. 93504 free blocks.
The question is: what is going on?
- Block device: 102400k (lvs says)
- Filesystem size: 95054k (df says)
- Free blocks: 93504k (dumpe2fs says)
- Available size: 91456k (df says)
Best Answer
Try this:
mkfs.ext4 -N 104 -m0 -O ^has_journal,^resize_inode /dev/purgatory/test1I thinks this does let you understand "what is going on".
-N 104(set the number of iNodes you filesystem should have)-m 0(no reserved blocks)-O ^has_journal,^resize_inode(deactivate the featureshas_journalandresize_inoderesize_inode"costs" free space (most of the 1550 1K-Blocks/2% you see in yourdf- 12K are used for the "lost+found" folder)has_journal"costs" usable space (4096 1K-Blocks in your case)We get
102348out of102400, another 52 blocks unusable (if we have deleted the "lost+found" folder). Therefore we dive intodumpe2fs:and count the used blocks (for Backup superblock, Group descriptors, Block bitmap, Inode bitmap and Inode table) or we
grepand count:which gives us the count of lines which have a single block (in our example) and
which gives us the count of lines which have two blocks (in our example).
So we have (in our example)
13lines with one block each and19lines with two blocks each.which gives us
51blocks which are in use by ext4 itself. Finally there is only one block left. The block 0, which are the skipped1024Bytes at the beginning for things like the boot sector.