Ubuntu – df shows all space taken, but du doesn’t add up


I'm having a problem with Ubuntu 12.04 LTS. This is the second time I've encountered this problem in the last 3 weeks.
The first time is described in this closed question on StackOverflow.
The TL;DR version is I managed to use all the inodes on a 450G ext4 system compiling and building the Android stack less than 20 times.

I thought I was solving the problem by reformatting the disk as XFS so the inode storage could grow.

This morning after doing a build overnight, I'm down to less than 1GB of free space. There is nothing on this machine other than what is needed to build Android. I've done 5 total builds on the platform sources. The build creates a bunch of files, then I delete them shortly after with make clean. I'm not really less than 1GB but the tools are reporting it that way. I deleted a bunch of temporary files and had about 40GB "freed". A couple hours later, just idling, I was back to less than 1GB free.

Running Ubuntu from a flash drive returns the following for the partition…

$ df
Filesystem     1K-blocks      Used Available Use% Mounted on
/dev/sda5      468521456 468255460    265996 100% /media/f71c77eb-b4cc-

$ df -i
Filesystem      Inodes  IUsed   IFree IUse% Mounted on
/dev/sda5      1691760 624214 1067546   37% /media/f71c77eb-b4cc-

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda5       447G  447G  260M 100% /media/f71c77eb-b4cc-

Here's is the evidence that something is amiss. When I run du (with and without –apparent-size) or the visual Disk Usage Analyzer, I show that I'm really only using about
35GB or so. 98.7% of the used space is in /home/eric, but du doesn't add up to that. The discrepancy is between /home/eric and /home/eric/android

enter image description here

I've read through the relevant questions here and on SO and they usually suggest it's deleted files held by open processes. I rebooted into a flash drive to run this test, so it shouldn't be open files. FWIW, /tmp is empty.

Is there a tool I can install on a flash drive to recover the 'lost' space? I can try to free up memory on the system and run it there, but I'm assuming it's better to do this from a flash drive.

Should I be configuring this system in a different way? I would rather not do another wipe and install, but I need a sustainable Android build system.

FOLLOWUP – I had to nuke the installation last week and re-installed 12.04 in order to get work done. As I go through Android builds again this week, I will keep a watchful eye over the disk usage and give information here as I learn more.


Best Answer

On Oracle Linux it happens when you have (a lot of/large) files which are deleted but still open by a running process. Then stop of the processes or reboot of the machine helps.

Related Question