Ext4 – Moving Millions of Small Files Causes Out of Space Error

ext4file-copymove

I't trying to move around 4.5 million files (size ranges from 100 – 1000 bytes) from one partition to other. The total size of the folder is ~2.4 GB

First I tried to zip it and move the zipped file to the new location. It is able to paste only ~800k files and shows "out of space" error.

Next I tried the mv command and it also resulted in same condition.

Using rsync also resulted in the same error with only ~800k files being moved.

I checked the disk free status and it is way under the limit. ( The new partition has ~700 GB free space and the required space is ~2.4 GB).

I checked the free inode for that partition it is the same. It is using only ~800k out of the maximum possible 191 M inodes. ( I had actually formatted the partition with 'mkfs.ext4 -T small /dev/sdb3' )

I have no idea of what is going wrong here. Everytime it is only able to copy or move ~800k files only.

Best Answer

I have found the reason for the error (found it on a different forum). The error was due to the hashing algorithm used by ext4 which is enabled by "dir_index" parameter. There were too many hashing collisions for me so I disabled it by the following command:

tune2fs -O "^dir_index" /dev/sdb3

The downside is that my partition is slower than before due to no indexing.

For more information on the problem : ext4: Mysterious “No space left on device”-errors

Related Question