my current system: 14.04.4 LTS (GNU/Linux 3.13.0-85-generic x86_64)
$ free
total used free shared buffers cached
Mem: 16366288 16090588 275700 4047472 1019652 5253144
-/+ buffers/cache: 9817792 6548496 Swap: 7813116 7308592 504524
$ cat /proc/sys/fs/file-nr
1196103 0 1624594
For a couple of months (and a number of updates) I've got the problem that suddenly after some days of working with my box (and daily suspending it) memory usage and the number of allocated file descriptors shoots up.
All of a sudden the "committed" memory rises, up to around 120 – 160 GB(!). Normally this value sits around my physical RAM. Another symptom is that all my RAM + swap is used up. The result is an unusable system. For now I only could fix this situation by rebooting. Killing nearly all processes after having shut down X did not help to free up the committed value.
A strong indicator seems to be the unreasonably high value of the open files counter in /proc/sys/fs/file-nr — this goes up above 2 million. Lately I tried to limit this to 1624594 but the only result is (of course) that I don't run out of memory but instead out of FDs.
Having Munin in place I can see a strong temporal correlation between FDs and memory usage.
I have some CIFS shares mounted and my user has its gvfsd-fuse. Root FS is ext4.
The results of lsof
and traversing /proc/ list only about 10% of /proc/sys/fs/file-nr.
I suspected MTP to be the cause (when I plug in my Motorola phone) but recently this happened without having used MTP since the last reboot.
Sidenote: I've got radeon drivers for video:
ii libdrm-radeon1:amd64 2.4.64-1~ubuntu14.04.1
ii libdrm-radeon1:i386 2.4.64-1~ubuntu14.04.1
ii radeontool 1.6.3-1
ii xserver-xorg-video-radeon 1:7.3.0-1ubuntu3.1
UPDATE
root:/proc$ (echo -n "0" ; for processid in [0-9]*; do echo -n "+$(ls /proc/$processid/fd/ | wc -l)"; done;echo)|bc
2597
root:/proc$ lsof -n|wc -l
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs
Output information may be incomplete.
84504
root:/proc$ sudo -u mdo lsof -n|wc -l
72170
How can I find out what's eating all these FDs and my memory?
Best Answer
This will show the number of open files per process ID:
(save and execute with
sudo
).Then you can track down what that process ID belongs to.