Ubuntu – Memory usage doesn’t add up over time

kernelmemory usage

The memory usage on my Ubuntu 13.04 system doesn't add up over time. I've tried the 3.8.0-xx kernels and the latest 3.9 from /~kernel-ppa/mainline.

When I first boot, it all adds up correctly but after a couple of days, this is what it looks like.

$ free -k
             total       used       free     shared    buffers     cached
Mem:       7978976    7319920     659056          0      97248    1344356
-/+ buffers/cache:    5878316    2100660
Swap:      2438140          0    2438140

Free is reporting that about 5.8G of resident memory is being used excluding cache and buffers.

When I sum the rss reported by the ps command, the totals differ dramatically.

sudo ps -eo pid,rss,args  > mem.txt

  PID      RSS  COMMAND
6377    848,268  /usr/lib/firefox/firefox
6095    243,664  /usr/bin/gnome-shell
1752    72,300  /usr/sbin/mysqld
6167    71,460  /var/lib/dropbox/.dropbox-dist/dropbox
7608    66,952  /usr/bin/python /usr/bin/deluge-gtk
23320   57,892  /usr/bin/X :3 -background none -verbose -auth /run/gdm/auth-for-gdm-9zmczs/database -nolisten tcp vt7
6445    45,476  /usr/lib/firefox/plugin-container /usr/lib/adobe-flashplugin/libflashplayer.so -greomni /usr/lib/firefox/omni.ja -appomni /usr/lib/firefox/browser/omni.ja -appdir /usr/lib/firefox/browser 6377 true plugin
6159    39,784  nautilus -n
6153    37,276  /usr/bin/python /usr/bin/autokey-gtk
9306    29,432  gnome-terminal
6152    27,152  nm-applet
6066    24,796  /usr/lib/gnome-settings-daemon/gnome-settings-daemon
6434    20,120  /usr/lib/firefox/plugin-container /opt/google/talkplugin/libnpgoogletalk.so -greomni /usr/lib/firefox/omni.ja -appomni /usr/lib/firefox/browser/omni.ja -appdir /usr/lib/firefox/browser 6377 true plugin
9505    17,700  /usr/sbin/apache2 -k start
6154    16,096  update-notifier
6437    13,516  /opt/google/talkplugin/GoogleTalkPlugin
6184    13,192  /usr/lib/evolution/evolution-calendar-factory
5983    12,372  gnome-session --session=gnome
3746    9,468   /usr/bin/python /usr/bin/fail2ban-server -b -s /var/run/fail2ban/fail2ban.sock
9510    8,476   /usr/sbin/apache2 -k start
9511    8,476   /usr/sbin/apache2 -k start
9512    8,476   /usr/sbin/apache2 -k start
9513    8,476   /usr/sbin/apache2 -k start
9514    8,476   /usr/sbin/apache2 -k start
6112    7,732   /usr/lib/evolution/evolution-source-registry
1890    7,212   /usr/bin/atop -a -w /var/log/atop/atop_20130425 600
6106    6,912   /usr/lib/gnome-shell/gnome-shell-calendar-server
1646    6,868   NetworkManager
6073    6,404   /usr/bin/pulseaudio --start --log-target=syslog
6125    6,080   /usr/lib/telepathy/mission-control-5
5588    5,592   /usr/lib/udisks2/udisksd --no-debug
1229    5,568   /usr/lib/colord/colord
6129    5,560   /usr/lib/gvfs/gvfs-udisks2-volume-monitor
1205    5,532   smbd -F
5954    5,220   gdm-session-worker [pam/gdm-password]
1656    5,164   /usr/lib/policykit-1/polkitd --no-debug
9312    5,112   -bash
5965    5,040   /usr/bin/gnome-keyring-daemon --daemonize --login
9531    4,952   -bash
1759    4,900   whoopsie
1606    4,412   /usr/sbin/winbindd -F
3149    4,300   /usr/lib/upower/upowerd
2992    4,068   /usr/sbin/console-kit-daemon --no-daemon
23316   3,952   /usr/lib/gdm/gdm-simple-slave --display-id /org/gnome/DisplayManager/Displays/_3
6299    3,800   /usr/lib/gvfs/gvfsd-metadata
1090    3,748   /usr/sbin/cupsd -F
1721    3,708   /sbin/dhclient -d -sf /usr/lib/NetworkManager/nm-dhcp-client.action -pf /run/sendsigs.omit.d/network-manager.dhclient-eth0.pid -lf /var/lib/NetworkManager/dhclient-681b428f-beaf-8932-dce4-687ed5bae28e-eth0.lease -cf /var/lib/NetworkManager/dhclient-eth0.conf eth0
2988    3,672   /usr/lib/accountsservice/accounts-daemon
6228    3,540   /usr/lib/gvfs/gvfsd-trash --spawner :1.4 /org/gtk/gvfs/exec_spaw/0
6043    3,296   /usr/lib/at-spi2-core/at-spi2-registryd --use-gnome-session
1728    3,288   gdm
6136    3,220   /usr/lib/gvfs/gvfs-gphoto2-volume-monitor
6144    3,212   /usr/lib/gvfs/gvfs-afc-volume-monitor
6036    3,204   /usr/lib/at-spi2-core/at-spi-bus-launcher
1607    3,196   /usr/sbin/modem-manager
1   3,096   /sbin/init
6087    3,044   /usr/lib/pulseaudio/pulse/gconf-helper
6047    2,992   /usr/lib/gvfs/gvfsd
6089    2,900   /usr/lib/x86_64-linux-gnu/gconf/gconfd-2
6274    2,832   /usr/lib/dconf/dconf-service
1248    2,772   /usr/sbin/cups-browsed
1571    2,696   /usr/sbin/sshd -D
6140    2,648   /usr/lib/gvfs/gvfs-mtp-volume-monitor
1484    2,608   dhclient -1 -v -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0
9662    2,600   sudo ps -eo pid,rss,args
3614    2,428   /usr/sbin/afpd -U uams_dhx2.so,uams_clrtxt.so -g nobody -c 50 -n cheetah
771 2,340   dbus-daemon --system --fork
5376    2,124   /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 117:126
6032    1,908   //bin/dbus-daemon --fork --print-pid 5 --print-address 7 --session
1642    1,780   /usr/sbin/winbindd -F
3460    1,760   qmgr -l -t fifo -u
928 1,728   avahi-daemon: running [cheetah.local]
866 1,704   /usr/sbin/bluetoothd
719 1,700   /sbin/udevd --daemon
9637    1,696   man ps
1627    1,684   nmbd -D
762 1,652   rsyslogd -c5
2850    1,636   /usr/lib/postfix/master
1239    1,608   smbd -F
7599    1,608   pickup -l -t fifo -u -c
6040    1,488   /bin/dbus-daemon --config-file=/etc/at-spi2/accessibility.conf --nofork --print-address 3
1760    1,472   /usr/sbin/dnsmasq --no-resolv --keep-in-foreground --no-hosts --bind-interfaces --pid-file=/var/run/NetworkManager/dnsmasq.pid --listen-address=127.0.1.1 --conf-file=/var/run/NetworkManager/dnsmasq.conf --cache-size=0 --proxy-dnssec --enable-dbus=org.freedesktop.NetworkManager.dnsmasq --conf-dir=/etc/NetworkManager/dnsmasq.d
5399    1,368   /sbin/udevd --daemon
5400    1,368   /sbin/udevd --daemon
1233    1,356   rpc.statd -L
3516    1,308   /usr/lib/rtkit/rtkit-daemon
9648    1,188   pager -s
1719    1,044   cron
1675    976 /sbin/getty -8 38400 tty5
1691    976 /sbin/getty -8 38400 tty2
1692    972 /sbin/getty -8 38400 tty3
1695    972 /sbin/getty -8 38400 tty6
3971    972 /sbin/getty -8 38400 tty1
1668    968 /sbin/getty -8 38400 tty4
1216    964 rpcbind -w
716 900 upstart-udev-bridge --daemon
2893    880 /usr/sbin/dnsmasq -x /var/run/dnsmasq/dnsmasq.pid -u dnsmasq -r /var/run/dnsmasq/resolv.conf -7 /etc/dnsmasq.d,.dpkg-dist,.dpkg-old,.dpkg-new
9663    868 ps -eo pid,rss,args
9311    824 gnome-pty-helper
1703    816 acpid -c /etc/acpi/events -s /var/run/acpid.socket
1737    776 /usr/sbin/irqbalance
3606    760 /usr/sbin/cnid_metad -l log_note
1235    732 upstart-socket-bridge --daemon
763 680 rpc.idmapd
6031    592 /usr/bin/dbus-launch --exit-with-session gnome-session --session=gnome
933 472 avahi-daemon: chroot helper
457 420 upstart-file-bridge --daemon
6028    316 /usr/bin/ssh-agent /usr/bin/dbus-launch --exit-with-session gnome-session --session=gnome
1717    168 atd
2   0   [kthreadd]
3   0   [ksoftirqd/0]
5   0   [kworker/0:0H]
7   0   [kworker/u:0H]
8   0   [migration/0]
9   0   [rcu_bh]
10  0   [rcu_sched]
11  0   [watchdog/0]
12  0   [watchdog/1]
13  0   [migration/1]
14  0   [ksoftirqd/1]
16  0   [kworker/1:0H]
17  0   [watchdog/2]
18  0   [migration/2]
19  0   [ksoftirqd/2]
21  0   [kworker/2:0H]
22  0   [watchdog/3]
23  0   [migration/3]
24  0   [ksoftirqd/3]
26  0   [kworker/3:0H]
27  0   [watchdog/4]
28  0   [migration/4]
29  0   [ksoftirqd/4]
31  0   [kworker/4:0H]
32  0   [watchdog/5]
33  0   [migration/5]
34  0   [ksoftirqd/5]
36  0   [kworker/5:0H]
37  0   [khelper]
38  0   [kdevtmpfs]
39  0   [netns]
40  0   [bdi-default]
41  0   [kintegrityd]
42  0   [kblockd]
43  0   [ata_sff]
44  0   [khubd]
45  0   [md]
46  0   [devfreq_wq]
48  0   [irq/72-AMD-Vi]
49  0   [khungtaskd]
50  0   [kswapd0]
51  0   [ksmd]
52  0   [khugepaged]
53  0   [fsnotify_mark]
54  0   [ecryptfs-kthrea]
55  0   [crypto]
66  0   [kthrotld]
74  0   [binder]
93  0   [deferwq]
94  0   [charger_manager]
198 0   [scsi_eh_0]
204 0   [scsi_eh_1]
205 0   [firewire]
219 0   [kworker/u:3]
220 0   [scsi_eh_2]
239 0   [scsi_eh_3]
251 0   [scsi_eh_4]
255 0   [scsi_eh_5]
259 0   [scsi_eh_6]
260 0   [scsi_eh_7]
269 0   [kworker/u:6]
283 0   [scsi_eh_8]
284 0   [scsi_eh_9]
360 0   [kworker/5:1H]
361 0   [jbd2/sda1-8]
362 0   [ext4-dio-unwrit]
643 0   [xfsalloc]
644 0   [xfs_mru_cache]
645 0   [xfslogd]
646 0   [xfs-data/sdb1]
647 0   [xfs-conv/sdb1]
648 0   [xfs-cil/sdb1]
650 0   [xfsaild/sdb1]
695 0   [xfs-data/sdc1]
696 0   [xfs-conv/sdc1]
697 0   [xfs-cil/sdc1]
698 0   [xfsaild/sdc1]
730 0   [rpciod]
743 0   [nfsiod]
922 0   [krfcommd]
944 0   [kworker/4:1H]
1017    0   [edac-poller]
1029    0   [hd-audio0]
1031    0   [kpsmoused]
1060    0   [kvm-irqfd-clean]
1107    0   [hd-audio1]
1157    0   [kworker/3:1H]
1190    0   [kworker/2:1H]
1198    0   [kworker/1:1H]
1793    0   [kauditd]
1845    0   [flush-8:0]
3569    0   [iprt]
4867    0   [kworker/4:1]
5403    0   [nfsv4.0-svc]
5848    0   [kworker/0:1H]
5935    0   [kworker/2:3]
9274    0   [kworker/5:2]
9527    0   [kworker/5:0]
9630    0   [flush-8:32]
10694   0   [kworker/3:3]
13358   0   [kworker/1:0]
18154   0   [kworker/0:0]
19625   0   [kworker/4:0]
23502   0   [kworker/3:2]
31403   0   [kworker/2:1]
32141   0   [kworker/0:2]
32655   0   [kworker/1:1]
    1,939,900   

ps reports that the sum off all the resident memory used by all my processes is about 1.9G.

What processes are using the remaining 3.9G?

EDIT: More information on kernel memory usage. slabtop reports the following.

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME                   
2233988 2233988 100%    0.88K  62057       36   1985824K xfs_inode              
735948 714094  97%    0.22K  20443       36    163544K xfs_ili                
129280 117946  91%    1.00K   4040       32    129280K nfs_inode_cache        
 92344  92344 100%    0.93K   2716       34     86912K ext4_inode_cache       
292866 292866 100%    0.19K  13946       21     55784K dentry                 
 75096  71346  95%    0.55K   2682       28     42912K radix_tree_node     

Does this mean that the kernel is using almost 2GB to cache xfs_inodes, cache that's not reported in the cached column of the free command?

Best Answer

The memory has been found. It was in the xfs_inode cache which wasn't reported by ps or free. slabtop did however show me what I needed. The reason for such a large xfs_inode cache is because of a nightly backup. The memory can be released with the following command.

echo 3 > /proc/sys/vm/drop_caches

From http://www.linuxinsight.com/proc_sys_vm_drop_caches.html

To free pagecache:

echo 1 > /proc/sys/vm/drop_caches

To free dentries and inodes:

echo 2 > /proc/sys/vm/drop_caches

To free pagecache, dentries and inodes:

echo 3 > /proc/sys/vm/drop_caches

More information can be found on this blog.

http://serenadetoacuckooo.blogspot.ca/2012/03/linux-memory-tweaks-for-improved.html

I have to decide now on whether I want to drop the cache after backups, set vfs_cache_pressure back to 100 or just buy more memory.