Linux – XFS Incorrect statement of “No space left on device”

inodelinuxlvmxfs

I have a server setup of an XFS partition on LVM on a Ubuntu 14.04.1 LTS. While copying files to the home partition, "No space left on device" is displayed.

df -h displays sufficient space:

/dev/mapper/prod--vg-home     35G   21G   15G  60% /home

I also used fallocate to fill the disk almost completely in the same home account to ensure neither quota nor other disk space issues are related:

/dev/mapper/prod--vg-home     35G   34G  1.5G  96% /home

df -i also displays sufficient inodes:

/dev/mapper/prod--vg-home   36700160  379390 36320770    2% /home

I also used a large amount of small random files in the same home account, which replicated the issue:

mkdir 1
cd 1
dd if=/dev/zero of=masterfile bs=10000000 count=1
split -b 10 -a 10 masterfile
cd ..
cp -R 1 2

This resulted in the "No space left on device" again with almost no additional space required (about 30MB) and df -i mentions:

/dev/mapper/prod--vg-home   36700160 1310464 35389696    4% /home

I did perform a second test on the same machine (Ubuntu 14.04.1 LTS).

I've created a new logical volume of 5 GB and filled it with small files using the procedure of dd and split above.

I received a "No space left on device" with the following disk space and inodes available:

/dev/mapper/prod--vg-test    5.0G  4.2G  811M  85% /mnt/test
/dev/mapper/prod--vg-test    4257712  937920  3319792   23% /mnt/test

I've performed this test using root privileges, to ensure the reserved space is still irrelevant here.

I did perform a third test on a different machine (Debian 2.6.32-5).

I've created a new logical volume of 5 GB and filled it with small files using the procedure of dd and split above.

The procedure succeeded with the following disk space and inodes available:

/dev/mapper/data-test      5.0G  4.2G  909M  83% /mnt/test
/dev/mapper/data-test      4721904 1000005 3721899   22% /mnt/test

I've performed this test using root privileges as well, to ensure the reserved space is still irrelevant here.

Does this point at a bug in Ubuntu 14.04.1 LTS?

I did verify the impact of changing the maximum percentage of in odes on the Ubuntu 14.04.1 LTS:

xfs_growfs -m 25 /dev/mapper/prod--vg-home

This amount can easily be decreased and increased.

While experimenting with this setting, I noticed that decreasing it to 3% and increasing it again to 25%, and deleting some files, allows me to add a lot more files again, but still causes the error prior to filling up either storage or inodes.

xfs_info displays:

meta-data=/dev/mapper/prod--vg-home isize=256    agcount=14, agsize=655360 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=9175040, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

Is there any other setting which could cause the "No space left on device" message? Or should I conclude this is a bug?

Thank you

Best Answer

There is a bug with xfs_growfs which causes inodes to not be properly distributed across a partition. The solution is to simply remount with the inode64 option. For example, if this was the /dev/vda1, you would do the following:

mount -o remount,inode64 /dev/vda1

You can find more information about the bug here.

Related Solutions

Linux Ubuntu – Fix ‘No Space Left on Device’ After Resizing Partition

Firstly, you can't simply add up the sizes of all the files on the disk, and expect that to be the total amount used. Every time you store a file, there is some space wasted. It's like putting books on a shelf, if the books vary in size, then you're going to have a space between the top of the book and the bottom of the next shelf.

Secondly, if you have any files which are open but deleted, then the space that is used will still be used until the program with that file either closes it, or exits. That's often used for temporary files, the program doesn't have to worry about cleaning them up, all it needs to do is open a file, then delete it, before working with it. These files used space will show up in df, but you can't find a filename which corresponds to it. If you want to find them, then you'll have to look in /proc/*/fd

Thirdly, and this is your issue here, ext3 file systems have a percentage of reserved space which can only be written to by root. There are two reasons for this, many file systems become inefficient when the disk becomes close to becoming full, the system has to spend more and more time fitting files into the spaces that are left. Also reading and writing to the files is slow, as they end up being badly fragmented. Another reason for reserving space for root is that it allows root to compress files and hopefully recover some space for the users. If the disk was totally full, then that wouldn't be possible.

Therefore, there is nothing wrong, what you are seeing is normal behaviour for a full disk.

Linux – Deleting Files Doesn’t Free Space

The ext3 and ext4 family of filesystems reserve a portion of the disk to keep it from becoming truly "100% full", for stability reasons, and so that the FS can store metadata in some of that space.

You can directly tweak how much of the space is reserved:

tune2fs -m 1 /dev/sdXX

replacing "1" with the percentage (0 to 100) of the disk to reserve, and "XX" with the device node and partition number of the device node, so in your case "XX" would be "b2".

Analogy: you have a full bus, and the only remaining seat is next to a very large person who occupies a seat and a half by himself. A person who is very insistent on getting a seat walks up and demands to sit there. Although most people would consider the second seat taken, this person is insistent. So the large person goes, "Whoa, OK!" and squishes in to allow them to sit. But as soon as 1 person gets off the bus and opens up another seat, even if the passenger next to the large person moves into a seat, most people still consider the bus to be 100% full, because nobody wants to sit next to the large person.

Source

You can also check this to see Reserved Blocks count…

dumpe2fs -h /dev/sdb2

To quote the very intelligent user who nailed the issue in the source above:

You will see "Available" go positive when "Used" is reduced to below 0.95*136236548 blocks = 129424720 blocks roughly.

(we have to adjust the "0.95" to the reserved % in your specific case, and the 136236548 blocks to the total block size of your device).

Best Answer

Related Solutions

Linux Ubuntu – Fix ‘No Space Left on Device’ After Resizing Partition

Linux – Deleting Files Doesn’t Free Space

Related Question