Filesystems – Getting Block Size via df vs dumpe2fs

filesystems

These two tools, seems to report different block sizes

root@ubuntu-xenial:~# df
Filesystem     1K-blocks     Used Available Use% Mounted on
udev              498588        0    498588   0% /dev
tmpfs             101584     3116     98468   4% /run
/dev/sda1       10098468  1438288   8643796  15% /
tmpfs             507916        0    507916   0% /dev/shm
tmpfs               5120        0      5120   0% /run/lock
tmpfs             507916        0    507916   0% /sys/fs/cgroup
vagrant        343946960 55977016 287969944  17% /vagrant
tmpfs             101584        0    101584   0% /run/user/1000

root@ubuntu-xenial:~# dumpe2fs /dev/sda1 | grep -i block | grep -i size
dumpe2fs 1.42.13 (17-May-2015)
Block size:               4096
Flex block group size:    16

What is more, fdisk command reports different sector size (aren't the terms sector and block interchangeable?)

root@ubuntu-xenial:~# fdisk -l
Disk /dev/sda: 10 GiB, 10737418240 bytes, 20971520 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x38d40272

Device     Boot Start      End  Sectors Size Id Type
/dev/sda1  *     2048 20971486 20969439  10G 83 Linux


Disk /dev/sdb: 10 MiB, 10485760 bytes, 20480 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

I am on a vagrant machine for what that matters

Best Answer

... reports different sector size (aren't the terms sector and block interchangeable?)

"Block" has been used to mean many different things. You have to look at the specific context.

df stretches this further, saying "1K-blocks" to mean units of 1 Kilobyte, even when the filesystem(s) use different block sizes.

"Sector" originally means the block size a disk uses. But notice your fdisk reports two different types of sector size, "physical" and "logical" :-). So you need to look out for context here as well. [*]

What is the filesystem block size?

Because the first tools you mentioned were df and dumpe2fs, I guessed you might be looking for the filesystem block size. In that case, you want the value which dumpe2fs shows as "Block size".

dumpe2fs is a specific tool for ext2/3/4, but I think that makes for the most robust approach. That said, it means that if you use a different filesystem, you will have to use a different tool and confirm exactly what it uses for "block size".

There may be some exceptions to the normal filesystem block size. Very recent ext4 may support data storage inline in the inode. That is, files smaller than a certain size may be stored in the file's inode, without allocating any data blocks at all. See How to use the new Ext4 Inline Data feature ? (storing data directly in the inode)

Alternative query method: statvfs()

Alternatively, you could run stat -f / to show the statvfs() information for your root filesystem. It was discussed here: stat file system sizes.

stat -f output includes two different fields, "Fundamental block size" and "Block size" :-).

The "fundamental" block size is the granularity that space usage is reported in - this will apply to df. You can hope that this is the same granularity used to allocate file space. At least this is true for your ext2/3/4, and in many other cases.

stat claims the other block size is to be used "for fast transfers". In ext2/3/4, these two block sizes will always be the same.

If ext2 had ever implemented fragments, like UFS from BSD UNIX, it would have been possible for the two sizes to differ. This is reflected in dumpe2fs always showing a "Fragment size" value equal to "Block size". See What can f_bsize be used for? (Is it similar to st_blksize?)

As a more generic tool, stat -f is liable to mislead in some cases.

It is not very widely used either; this might increase the risk. It doesn't help that it reports two different values, where the original meanings are not relevant to native Linux filesystems. FWIW, this site's users hated my question asking to confirm its behaviour (the link above). I usually assume this means they got angry because no-one knows how to answer the question.

stat -f is a feature of GNU coreutils. (It is not supported by stat from FreeBSD 7.2).

Is "block size 'for fast transfers' " meaningful?

This description may be relevant for filesystems which implement fragments. If they do not implement "delayed allocation", it may be more efficient to make sure you use a block-sized write buffer when you increase the length of a file. As always, if you are thinking about making a change to improve performance, you should aim to confirm the improvement with measurements.

On Linux filesystems, the two block sizes will be the same value. In this case, I don't know any reason to use that particular value for optimization.

Most often, userspace doesn't have to worry too much about sizing I/O calls "for fast transfers". The IO scheduler can coalesce adjacent I/Os for efficiency. And un-synced IO through the page cache gets massive performance benefits from write-back caching and automatic read-ahead. You can go a long way just using the page cache size, or hard-coding 4KB :-).

In most transfers, the page cache imposes a minimum size for the physical IO. The page size is 4KB on most systems, and usually file IO goes through the page cache. (Again small files are a special case. Or rather the last page of a file is a special case, if the file size is not an exact multiple of 4KB).

[*] E.g. given modern abstraction layers, you might sometimes see a field that says "sector size" reported as 512, on a modern "advanced format" disk which uses 4KB physical sectors.

Related Solutions

Sparse Files – Understanding File Holes and Block Size

Ext4 can use 1kB, 2kB or 4kB as the block size; as far as I know the default on Ubuntu is 4kB. Note that here, a block is the size of a file chunk, which is constant for a given filesystem. The file you describe has two blocks that are not zeroes: the one containing hello (surrounded by a bunch of zeroes — 3616 before and 474 after), and the one containing here (preceded by a bunch of zeroes, and containing only 3148 bytes, after which the end of the file is reached). The total is two blocks of 4kB.

In the ls output, blocks are an arbitrary unit chosen by the ls command and defaulting to 1kB. There are 2 blocks of 4kB each allocated to contain file data, therefore the allocated size for the file is 8kB.

Your confusion may be due to two things. First, the figure of 2048 bytes for a block is possible, but it's not the default value under Ubuntu (or most modern distributions), and it's apparently not the value on your system. You can check the block size by running tune2fs -l /dev/sdz42 (use the actual path to your filesystem device).

Second, sparse files consist of not storing blocks that are entirely made of zeroes. If a block (which is of necessity aligned on a block size boundary, at least for most filesystems including ext4) contains zeroes and other things, then the full block is stored on the disk. Thus, in that 40012-byte file (how did you get to 40013, by the way), there are 4 all-zero non-stored blocks, then one stored block containing hello surrounded by zeroes, then 4 more all-zero non-stored blocks, and a final partial block containing zeroes and there.

Note that your utility can be written in terms of standard shell commands:

n=20000
while IFS= read -r line; do
  dd bs=1 seek=$n </dev/null
  echo "$line"
done >testfile

Why the partitions don’t show the right capacity on a 4096 byte physical block hard drive

What is happening is that the -b switch is nonsense and should not even be there. The sector numbers recorded in the MBR are always interpreted to be in units of the drive's logical sector size ( 512 bytes ). By using the -b switch, you are causing fdisk to divide all of the sectors it records by 8, so the kernel interprets the partitions to be 1/8th the size you intended.

If you use parted instead of fdisk, it will make sure your partitions are properly aligned automatically. With fdisk, just make sure that the starting sectors are a multiple of 8.