Linux – What’s eating the disk space

disk-usagelinux-mint

I'm running Linux Mint 14 Nadia. The Linux partition has 10G. When the system starts, du reports 80% usage. Then the usage slowly grows until it reaches 100% and the system becomes unusable. (It can happen on the order of days or weeks). After the reboot the usage resets to 80%.

The strangest thing of all is that du shows no change.

Here's output of those commands (Windows and external drive partitions are elided):

# --- Just after reboot ---

$ df -h     
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       9.8G  7.3G  2.0G  80% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
udev            428M  292K  428M   1% /dev
tmpfs            88M  1.3M   87M   2% /run
none            5.0M     0  5.0M   0% /run/lock
none            437M  288K  437M   1% /run/shm
none            100M   12K  100M   1% /run/user

$ sudo du -x   -d1 -h /
186M    /opt
512M    /var
11M /sbin
556K    /root
1.3G    /home
613M    /lib
8.0K    /media
4.6G    /usr
16K /lost+found
111M    /boot
39M /etc
4.0K    /mnt
60K /tmp
9.1M    /bin
4.0K    /srv
7.3G    /            # <-- note this


# --- After some time ---

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       9.8G  9.1G  199M  98% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
udev            428M  292K  428M   1% /dev
tmpfs            88M  1.3M   87M   2% /run
none            5.0M     0  5.0M   0% /run/lock
none            437M   27M  411M   7% /run/shm
none            100M   28K  100M   1% /run/user

$  sudo du -x   -d1 -h /
186M    /opt
511M    /var
11M /sbin
556K    /root
1.4G    /home
613M    /lib
8.0K    /media
4.6G    /usr
16K /lost+found
111M    /boot
39M /etc
4.0K    /mnt
520K    /tmp
9.1M    /bin
4.0K    /srv
7.3G    /              # <-- note this

(Note: I use hibernation. After the hibernation, the usage stays the same, and after reboot, it resets to 80%.)

How do I track what eats the space?

I've read this question. I'm still in the dark. How do I find out which program is responsible for this behavior?

After edit: found it. The space is claimed by the kernel log, which is seen by dmesg. It fills up because my machine generates errors at the rate 5 a second. (It's related to this bug.) Let the future readers with a similar problem – slowly-filling disk space unseen by du – not forget to try dmesg in searching for the cause.

Best Answer

Repeated execution of

sudo du -x   -d1 -h /

(down the directory tree) should tell you where the space is consumed. That probably explains without further investigation which application is causing that.

invisible files

If du doesn't show these files then one of the possibilities are deleted files. A file (or rather: its name i.e. its entry in a directory) can be deleted while the file is still in use. As long as there is a valid file descriptor pointing at this file it covers space on the volume (if it is not an empty file...).

cat >file &
ls -l file
rm file
ls -l file
# PID of cat is 19834
ls -l /proc/19834/fd
lrwx------ 1 hl hauke 64 11. Feb 19:16 0 -> /dev/pts/0
l-wx------ 1 hl hauke 64 11. Feb 19:16 1 -> /crypto/home/hl/tmp/file (deleted)
lrwx------ 1 hl hauke 64 11. Feb 19:15 2 -> /dev/pts/0

You can find these files with find:

find /proc/ -mindepth 3 -maxdepth 3 \
-regex '/proc/[1-9][0-9]*/fd/[1-9][0-9]*' -type l -lname '*(deleted)' \
-printf '%p\n     %l\n' 2>/dev/null

It may be one single huge file or a bunch of smaller files which cause your problem. There are about 30 such files on my system now (belonging to only five processes). ls -l shows the size of these files but it seems not to be possible to get this value from find.

After killing the process the space becomes available to the file system (df) again.

Related Question