Filesystems – Is ext4 More Expensive Than NTFS?

ext4filesystemsntfsperformance

I have just converted an NTFS partition to ext4, however the total space seems reduced from 421G to 415G. Where did the 6G go? And, the reserved space is grown to 199M in ext4, much larger compared to 78M in NTFS, why?

The partition is mainly used for movies/musics, so most files are very large (>10M each). I want to use ext4 file system, is there any suggestion?

mkfs.ntfs:
    /dev/sdb4             421G   78M  421G   1% /mnt/mmedia

mkfs.ext4:
    /dev/sdb4             415G  199M  393G   1% /mnt/mmedia
                       (415G - 199M == 393G ?)

It's also weird that the remaining size of ext4 is 393G, shouldn't it be 415G or 414G? What happened to the disappeared 22G? Compared to NTFS, ext4 consumed 6.6% of total space, that's really a big deal.

The question is:

What is 6G mainly used for, for journal, for redundancy, or for indexing?
Why the remaining space 393G not 415G? There is a 22G hole which is rather big.
What parameters would you advice if this ext4 partition is used to store movie/music files? It's said ext4 performs better then ext3 for large partitions, is it true? I won't back to ext2 which isn't journal.

Best Answer

What is 6G mainly used for, for journal, for redundancy, or for indexing?

(Not known yet.)

Why the remaining space 393G not 415G? There is a 22G hole which is rather big.

It's 5% reserved blocks for superuser, which is used to avoid fragmentation. You can adjust it to 1% by mkfs.ext4 -m 1.

What parameters would you advice if this ext4 partition is used to store movie/music files?

You can specify a usage option to mkfs.ext4, e.g., mkfs.ext4 -T large_file, this will let mkfs.ext4 to decide parameters for partitions containing large files.

Related Solutions

Linux – How to Create an Ext4 Partition from Console

First of all you need to create a partition, then you can make the filesystem.

Create a new partition

sudo fdisk /dev/sda

This will bring up the fdisk menu. You should familiarize yourself with it, search for tutorials (here's one). The basic options are:

Command action
   a   toggle a bootable flagL
   b   edit bsd disklabel
   c   toggle the dos compatibility flag
   d   delete a partition
   l   list known partition types
   m   print this menu
   n   add a new partition
   o   create a new empty DOS partition table
   p   print the partition table
   q   quit without saving changes
   s   create a new empty Sun disklabel
   t   change a partition's system id
   u   change display/entry units
   v   verify the partition table
   w   write table to disk and exit
   x   extra functionality (experts only)

If all goes well, by pressing N, you will be given the option of creating a new partition in the empty space. If the only unallocated space is the one you want to use, the default choices should be fine and you can just let fdisk choose.

Since you want to create an ext partition, you don't need to do anything. If you wanted to create a different type (swap or NTFS or whatever) you would need to use t to change the partition type. I am mentioning this just in case.
Check that your changes are what you expected by hitting P to print the partition table.
If everything is OK, write the new partition table to the disk (W) and exit (Q).

Now that you have your new, empty partition, you can create its filesystem. So, if you just created /dev/sdaX (where X is the number of the partition you created, for example /dev/sda2), run this:

sudo mkfs.ext4 /dev/sdaX

Ddrescue, “size on disk” lower than total size, with possible impact on performance when writing to NTFS

This answer investigates the behavior of ddrescue to address the main question. If you're not interested in testing procedure then you may skip to my conclusions and interpretation near the end.

Testbed

$ uname -a
Linux foo 4.2.0-27-generic #32~14.04.1-Ubuntu SMP Fri Jan 22 15:32:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

$ cat /etc/issue
Ubuntu 14.04.5 LTS \n \l

$ ddrescue -V
GNU ddrescue 1.17
…

The filesystem is btrfs; it shouldn't matter though as long as it supports sparse files.

Testing

At first I got 8 MiB of random data:

dd if=/dev/urandom of=random.chunk bs=1M count=8

Then I made it a loopback device and remembered its name:

loopdev=`sudo losetup -f --show random.chunk`

Next I created yet another device that consisted of

chunk 0: unreadable, 1 MiB
chunk 1: zeros, 2 MiB
chunk 2: unreadable, 4 MiB
chunk 3: data from random.chunk, 8 MiB
chunk 4: unreadable, 16 MiB

The code (it uses here document syntax):

sudo dmsetup create mydevice << EOF
    0  2048 error
 2048  4096 zero
 6144  8192 error
14336 16384 linear $loopdev 0
30720 32768 error
EOF

I confirmed with gdisk -l /dev/mapper/mydevice that the total size is 31 MiB as it should be.

Actual reading is done with:

ddrescue     /dev/mapper/mydevice  normal.raw  normal.log
ddrescue -R  /dev/mapper/mydevice normalR.raw normalR.log
ddrescue -S  /dev/mapper/mydevice  sparse.raw  sparse.log
ddrescue -RS /dev/mapper/mydevice sparseR.raw sparseR.log

And the results of ls -hls *.raw are

 10M -rw-rw-r-- 1 kamil kamil 15M Sep 10 00:37 normal.raw
 10M -rw-rw-r-- 1 kamil kamil 15M Sep 10 00:37 normalR.raw
8.0M -rw-rw-r-- 1 kamil kamil 15M Sep 10 00:37 sparse.raw
8.0M -rw-rw-r-- 1 kamil kamil 15M Sep 10 00:37 sparseR.raw

To be sure, I confirmed with cmp that all four files are identical when you read them. Four logfiles contained the same map of erroneous and healthy sectors.

Notice that

15 MiB means the last chunk is missing;
10 MiB indicates chunk 1 and chunk 3;
8 MiB indicates chunk 3 only.

Cleaning

sudo dmsetup remove mydevice
sudo losetup -d $loopdev
unset loopdev
rm random.chunk normal.raw normal.log normalR.raw normalR.log sparse.raw sparse.log sparseR.raw sparseR.log

Conclusions

When it comes to file size, it doesn't matter whether you read in reverse (-R) or not.
Unreadable chunk at the very end of the input file doesn't contribute to the overall size of the output file.
Unreadable chunks that do contribute to overall file size are always sparse (if target filesystem supports this, of course).
The -S option only affects blocks of zeros that were actually read from the input file.

Interpretation

Above there were facts. This section is more like my opinion.

It appears ddrescue tries to save you diskspace whenever it can do this without additional work. When you use -S the tool has to do some computations to check if a given data block is all zeros. If there's a read error it doesn't need to compute anything, it can make the fragment sparse in the output file with no cost.

Solution

You wrote:

using the -R switch (“reverse”) at the beginning, figuring that it would allocate the whole size of the input HDD right away

We just saw it's a false assumption. In fact you described what -p does. ddrescue -p will preallocate space on disk for output file. When I did this during my tests the output file had 31 MiB and was not sparse (even with -S).