Centos – Unable to swapoff but enough physical memory is available

Seeing another of your post I guess you are using zram. So that will be my assumption here.

I did the experience to install zram and consume lot of memory, and I got the same output of smem than you. smem does not take into account zram into its counting, it only uses /proc/meminfo to compute its value, and if you look and try to understand the code you will see that the zram RAM occupation is gets in the end counted under the noncache column of the kernel dynamic memory line.

Further investigations

Following my gut feeling that zram was behind this behavious, I setted up a VM with similar spec as your machine: 4 GB RAM and 2 GB zram swap, no swap file.

I have loaded the VM with heavy weight applications and got the following state:

huygens@ubuntu:~$ smem -wt -K ~/vmlinuz-3.2.0-38-generic.unpacked -R 4096M
Area                           Used      Cache   Noncache 
firmware/hardware            130717          0     130717 
kernel image                  13951          0      13951 
kernel dynamic memory       1063520     922172     141348 
userspace memory            2534684     257136    2277548 
free memory                  451432     451432          0 
----------------------------------------------------------
                            4194304    1630740    2563564 
huygens@ubuntu:~$ free -m
             total       used       free     shared    buffers     cached
Mem:          3954       3528        426          0         79        858
-/+ buffers/cache:       2589       1365
Swap:         1977          0       1977

As you can see free reports 858 MB cache memory and that is also what smem seems to report within the cached kernel dynamic memory.

Then I further stressed the system using Chromium Browser. At the beginning, it was only have 83 MB of swap used. But then after a few more tabs opened, the swap switch quickly to almost it's maximum and I experienced OOM! zram has really a dangerous side where wrongly configured (too big sizes) it can quickly hit you back like a trebuchet-like mechanism.

At that time I had the following outputs:

huygens@ubuntu:~$ smem -wt -K ~/vmlinuz-3.2.0-38-generic.unpacked -R 4096M
Area                           Used      Cache   Noncache 
firmware/hardware            130717          0     130717 
kernel image                  13951          0      13951 
kernel dynamic memory       1355344     124072    1231272 
userspace memory             961004      36456     924548 
free memory                 1733288    1733288          0 
----------------------------------------------------------
                            4194304    1893816    2300488 
huygens@ubuntu:~$ free -m
             total       used       free     shared    buffers     cached
Mem:          3954       2256       1698          0          4        132
-/+ buffers/cache:       2118       1835
Swap:         1977       1750        227

See how the kernel dynamic memory (columns cache and non-cache) look like inverted? It is because in the first case, the kernel had "cached" memory such as reported by free but then it had swap memory held by zram which smem does not know how to compute (check smem source code, zram occupation is not reported in /proc/meminfo, this it is not computed by smem which does simple "total kernel mem" - "type of memory reported by meminfo that I know are cache", what it does not know is that in the computed total kernel mem it has added the size of the swap which is in RAM!)

When I was in this state, I activated a hard disk swap and turned off the zram swap and I reset the zram devices: echo 1 > /sys/block/zram0/reset.

After that the noncache kernel memory melted like snow in summer and returned to "normal" value.

Conclusion

smem does not know about zram (yet) maybe because it is still staging and thus not part of /proc/meminfo which reports global parameters (like (in)active pages size, total memory) and then only report on a few specific parameters. smem identified a few of this specific parameters as "cache", sum them up and compare that to total memory. Because of that zram used memory gets counted in the noncache column.

Note: by the way, in modern kernel, meminfo reports also the shared memory consumed. smem does not take that yet into account, so even without zram the output of smem is to consider carefully esp. if you use application that make big use of shared memory.

References used:

• smem source code
• zram official documentation
• /proc/meminfo explained (though would need an update)
• source code of meminfo output

In /proc/meminfo you find:

CommitLimit:    1551056920 kB
Committed_AS:   1549560424 kB

So you are at the commit limit.

If you have disabled overcommiting of memory (to avoid the OOM-killer) by:

echo 2 > /proc/sys/vm/overcommit_memory

Then the commit limit is computed as:

2   -   Don't overcommit. The total address space commit
        for the system is not permitted to exceed swap + a
        configurable amount (default is 50%) of physical RAM.
        Depending on the amount you use, in most situations
        this means a process will not be killed while accessing
        pages but will receive errors on memory allocation as
        appropriate.

(From: https://www.kernel.org/doc/Documentation/vm/overcommit-accounting)

You can use the full memory by:

echo 100 > /proc/sys/vm/overcommit_ratio

Then you will get out-of-memory when physical RAM and swap is all reserved.

The name overcommit_ratio is in this case a bit misleading: You are not overcommitting anything.

Even with this setup you may see out-of-memory before swap is exhausted. malloc.c:

#include <stdio.h>
#include <malloc.h>
#include <stdlib.h>
#include <unistd.h>

void main(int argc, char **argv) {
  long bytes, sleep_sec;
  if(argc != 3) {
    printf("Usage: malloc bytes sleep_sec\n");
    exit(1);
  }
  sscanf(argv[1],"%ld",&bytes);
  sscanf(argv[2],"%ld",&sleep_sec);
  printf("Bytes: %ld Sleep: %ld\n",bytes,sleep_sec);
  if(malloc(bytes)) {
    sleep(sleep_sec);
  } else {
    printf("Out of memory\n");
    exit(1);
  }
}

Compile as:

gcc -o malloc malloc.c

Run as (reserve 1 GB for 10 seconds):

./malloc 1073741824 10

If you run this you may see OOM even though there is swap free:

# Plenty of ram+swap free before we start
$ free -m
              total        used        free      shared  buff/cache   available
Mem:         515966        2824      512361          16         780      511234
Swap:       1256720           0     1256720

# Reserve 1.8 TB
$ ./malloc 1800000000000 100 &
Bytes: 1800000000000 Sleep: 100

# It looks as if there is plenty of ram+swap free
$ free -m
              total        used        free      shared  buff/cache   available
Mem:         515966        2824      512361          16         780      511234
Swap:       1256720           0     1256720

# But there isn't: It is all reserved (just not used yet)
$ cat /proc/meminfo |grep omm
CommitLimit:    1815231560 kB
Committed_AS:   1761680484 kB

# Thus this fails (as you would expect)
$ ./malloc 180000000000 100
Bytes: 180000000000 Sleep: 100
Out of memory

So while free in practice often will do The Right Thing, looking at CommitLimit and Committed_AS seems to be more bullet-proof.