Ubuntu – df shows all space taken, but du doesn’t add up

disk-usagefilesystem

I'm having a problem with Ubuntu 12.04 LTS. This is the second time I've encountered this problem in the last 3 weeks.
The first time is described in this closed question on StackOverflow.
The TL;DR version is I managed to use all the inodes on a 450G ext4 system compiling and building the Android stack less than 20 times.

I thought I was solving the problem by reformatting the disk as XFS so the inode storage could grow.

This morning after doing a build overnight, I'm down to less than 1GB of free space. There is nothing on this machine other than what is needed to build Android. I've done 5 total builds on the platform sources. The build creates a bunch of files, then I delete them shortly after with make clean. I'm not really less than 1GB but the tools are reporting it that way. I deleted a bunch of temporary files and had about 40GB "freed". A couple hours later, just idling, I was back to less than 1GB free.

Running Ubuntu from a flash drive returns the following for the partition…

$ df
Filesystem     1K-blocks      Used Available Use% Mounted on
/dev/sda5      468521456 468255460    265996 100% /media/f71c77eb-b4cc-

$ df -i
Filesystem      Inodes  IUsed   IFree IUse% Mounted on
/dev/sda5      1691760 624214 1067546   37% /media/f71c77eb-b4cc-

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda5       447G  447G  260M 100% /media/f71c77eb-b4cc-

Here's is the evidence that something is amiss. When I run du (with and without –apparent-size) or the visual Disk Usage Analyzer, I show that I'm really only using about
35GB or so. 98.7% of the used space is in /home/eric, but du doesn't add up to that. The discrepancy is between /home/eric and /home/eric/android

enter image description here

I've read through the relevant questions here and on SO and they usually suggest it's deleted files held by open processes. I rebooted into a flash drive to run this test, so it shouldn't be open files. FWIW, /tmp is empty.

Is there a tool I can install on a flash drive to recover the 'lost' space? I can try to free up memory on the system and run it there, but I'm assuming it's better to do this from a flash drive.

Should I be configuring this system in a different way? I would rather not do another wipe and install, but I need a sustainable Android build system.

FOLLOWUP – I had to nuke the installation last week and re-installed 12.04 in order to get work done. As I go through Android builds again this week, I will keep a watchful eye over the disk usage and give information here as I learn more.

Thanks

Best Answer

On Oracle Linux it happens when you have (a lot of/large) files which are deleted but still open by a running process. Then stop of the processes or reboot of the machine helps.

Related Solutions

Ubuntu – Unable to mount secondary BTRFS filesystem to specific subvolume

Btrfs filesystems do not interact with each other, so you cannot mount the @media subvolume on /dev/sd{b,c}1 because the @media subvolume is currently on /dev/sda1, and you get a cross-device error when you try to snapshot /mnt/media to /mnt/media_snap because /mnt/media is on /dev/sd{b,c}1 whereas /mnt/media_snap is on /dev/sda1.

Instead of having the @media subvolume on the /dev/sda1 btrfs filesystem, you want to create an @media subvolume on the btrfs filesystem on /dev/sd{b,c}1:

sudo mount /dev/sdb1 /mnt/media/
sudo btrfs subvol create /mnt/media/@media
sudo umount /mnt/media/
sudo mount -o subvol=@media /dev/sdb1 /mnt/media/

You can create snapshots of /mnt/media/, but you can place the snapshot only on the /dev/sd{b,c}1 filesystem. So you can create a read-only snapshot like this:

sudo btrfs subvol snap -r /mnt/media/ /mnt/media/<snapshot_name>

But I recommend placing each snapshot of a subvolume outside the subvolume itself, because this simplifies things when you need to revert to a snapshot. To do this, you will need to mount the root of the /dev/sd{b,c}1 filesystem:

sudo mkdir /mnt/media_root/
sudo mount -o subvol=/ /dev/sdb1 /mnt/media_root/
sudo btrfs subvol snap -r /mnt/media_root/@media/ /mnt/media_root/<snapshot_name>

Ubuntu – 2nd drive fills up storage space on root file system

This sounds as if you have written data to one or more of the folders which are the mount-points of your HDD and network drives while nothing was mounted to a mount-point.

You can unmount the drives and then check the contents of this folders to see the contents of this folders while nothing is mounted.

Alternativly (while all drives are mounted) you can use a bind mount to see what is in this folders (mount-points) as if nothing would be mounted to them, to do that, make sure that all drives are mounted properly. Then run

sudo mount --bind / /mnt

and browse to /mnt/media/potaito/..., this folders should be empty. If one or more of this folders are not empty, you can now either delete the files or move them to your HDD or network drive.

Example:

You found files in /mnt/media/potaito/freenas_home. To remove them all use

rm -R /mnt/media/potaito/freenas_home/*

To move them all to the network drive use

mv /mnt/media/potaito/freenas_home/* /media/potaito/freenas_home/

You can also use the file manager to delete/move the files, you'll have to decide yourself which of them to keep.

If you are done, remove the bind mount with

sudo umount /mnt

Best Answer

Related Solutions

Ubuntu – Unable to mount secondary BTRFS filesystem to specific subvolume

Ubuntu – 2nd drive fills up storage space on root file system

Related Question