Ubuntu – Find directories with lots of files in

filesysteminode

So a client of mine got an email from Linode today saying their server was causing Linode's backup service to blow up. Why? Too many files. I laughed and then ran:

# df -ih
Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/xvda        2.5M  2.4M   91K   97% /

Crap. 2.4million inodes in use. What the hell has been going on?!

I've looked for the obvious suspects (/var/{log,cache} and the directory where all the sites are hosted from) but I'm not finding anything really suspicious. Somewhere on this beast I'm certain there's a directory that contains a couple of million files.

For context one my my busy servers uses 200k inodes and my desktop (an old install with over 4TB of used storage) is only just over a million. There is a problem.

So my question is, how do I find where the problem is? Is there a du for inodes?

Best Answer

Check /lost+found in case there was a disk problem and a lot of junk ended up being detected as separate files, possibly wrongly.

Check iostat to see if some application is still producing files like crazy.

find / -xdev -type d -size +100k will tell you if there's a directory that uses more than 100kB of disk space. That would be a directory that contains a lot of files, or contained a lot of files in the past. You may want to adjust the size figure.

I don't think there's a combination of options to GNU du to make it count 1 per directory entry. You can do this by producing the list of files with find and doing a little bit of counting in awk. Here is a du for inodes. Minimally tested, doesn't try to cope with file names containing newlines.

#!/bin/sh
find "$@" -xdev -depth | awk '{
    depth = $0; gsub(/[^\/]/, "", depth); depth = length(depth);
    if (depth < previous_depth) {
       # A non-empty directory: its predecessor was one of its files
       total[depth] += total[previous_depth];
       print total[previous_depth] + 1, $0;
       total[previous_depth] = 0;
    }
    ++total[depth];
    previous_depth = depth;
}
END { print total[0], "total"; }'

Usage: du-inodes /. Prints a list of non-empty directories with the total count of entries in them and their subdirectories recursively. Redirect the output to a file and review it at your leisure. sort -k1nr <root.du-inodes | head will tell you the biggest offenders.

Related Question