Linux – Memory not used any more but still allocated

linuxmemory

I have something kind of weird going on with memory.

I have Ubuntu 14.04 servers with Zimbra 8.6 with kernel (for this one) 3.13.0-37-generic. But I have already seen the problem with other kernels.

Memory and swap are full :

$ free
             total       used       free     shared    buffers     cached
Mem:       6112624    5991208     121416         88       4752      79224
-/+ buffers/cache:    5907232     205392
Swap:      3905532    3624768     280764

I thought that Zimbra was eating all my memory but, strangely, it doesn't seems like it :

# ps -A --sort -rss -o comm,pmem | head -n 11
COMMAND         %MEM
java            10.6
clamd            4.7
mysqld           3.0
java             2.0
slapd            1.2
/opt/zimbra/ama  1.1
/opt/zimbra/ama  1.0
/opt/zimbra/ama  1.0
/opt/zimbra/ama  1.0
/opt/zimbra/ama  0.9

All my process take like half of the memory. My buffers and cached take nearly nothing.

When I stop Zimbra, there is still 3.5Gb taken:

# ps -A --sort -rss -o comm,pmem | head -n 12
COMMAND         %MEM
bash             0.0
bash             0.0
bash             0.0
sudo             0.0
rsyslogd         0.0
http             0.0
http             0.0
htop             0.0
init             0.0
ps               0.0

After a reboot, less than 200Mb was used.

Server was up for 139 days and memory use was growing bit by bit each day.

My question is then: what could have taken all the memory?

EDIT1, add some infos :

$ ls -l /dev/shm
lrwxrwxrwx 1 root root 8 mai    2 12:46 /dev/shm -> /run/shm

$ ipcs 
------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status      

------ Semaphore Arrays --------
key        semid      owner      perms      nsems     

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages    


$ df 
Filesystem                  1K-blocks     Used Available Use% Mounted on
udev                 3051744         4    3051740   1% /dev
tmpfs                 611260       496     610764   1% /run
/dev/sda2           14287344   2765996   10772548  21% /
none                       4         0          4   0% /sys/fs/cgroup
none                    5120         0       5120   0% /run/lock
none                 3056288         0    3056288   0% /run/shm
none                  102400         0     102400   0% /run/user

After reboot, (Zimbra started) :

$ free
             total       used       free     shared    buffers     cached
Mem:       6112576    5712908     399668        832     237892    1829424
-/+ buffers/cache:    3645592    2466984
Swap:      3905532          0    3905532

Some graphs of the RAM (I restarted the server around 12:30) :

RAM usage

Second graph shows results of :

ps aux |awk '{s+=$4} END {print s}'

And third, results of :

smem -tw |grep -v Area | sed 's/ //;s/ //'

Best Answer

Cache.

A good summary at http://www.linuxatemyram.com/.

Linux is borrowing unused memory for disk caching. This makes it looks like you are low on memory, but you are not! Everything is fine!

To clear the caches use this command, as root, and then observe resulting memory usage.

echo 3 > /proc/sys/vm/drop_caches
Related Question