Swap is high inspite configuration is swappiness=1

kernelmemoryrhelswap

we have moon server – version rhel 7.5

the behavior of consuming swap on this server is very strange

we configured the /proc/sys/vm/swappiness to 1 and restart the server

but we can see that server is eating 15G , when the available is 44G !!

HOW IT CAN BE??

[root@moon01 network-scripts]# more /proc/sys/vm/swappiness
1
[root@moon01 network-scripts]# free -g
              total        used        free      shared  buff/cache   available
Mem:            125          80          38           0           6          44
Swap:            15          15           0

from my understanding only if available is near last few giga ram then swap will increased

but this isn't the situation

Best Answer

Even with swappiness=1 Linux will continue to use swap if available. Your user-space programs do not need to exhaust free RAM for the kernel to start swapping. I first discovered this because I was having problems on an Ubuntu Linux desktop.

In answers and comments to my question, someone pointed out that disk caching is the probable cause. "Free" space in memory is very rarely empty. The kernel will quietly use it for caches including disk caches, safe in the knowledge that it can abandon the cache whenever applications require more memory.

I'm hunting for the reference in the kernel docks. But there's a good description somewhere of the way the majority of programs will have a lot of memory (including code) which is only used during startup and then never again. So particularly on a server, you will have a lot of "junk" sitting in memory, stealing space from useful things like disk caches. Linux knows this and will chose to swap out the junk rather than abandon pages from a disk cache. Over all this has the effect of gradually increasing the swap usage and slowly growing the size of caches. It does this even though "free" memory remains relatively high.

In short, this is expected behaviour and there's no simple way to turn it off.

Related Solutions

Debian – Swap usage too high

You can't swapoff because the amount of swapped memory can't be overtaken by your RAM. You are getting legitimate error message.

Small snippet.

if (!quiet || errno == ENOMEM)
    warn(_("%s: swapoff failed"), orig_special);

return -1;

In my opinion, your workload increases your RAM demand. You are running a workload that requires more memory. Usage of the entire swap indicates that. Also, changing swappiness to 1 might not be a wise decision. Setting swappiness to 1 does not indicate that swapping will not be done. It just indicates how aggressive kernel will be in respect of swapping, it does not eliminate swapping. Swapping will happen if needs to be done.

Also, I don't know why you are trying to disable swap. Unless you have tons and tons of RAM, you should not disable swap.

Of course, you can reboot and swap usage will be zero then. And you can safely swapoff then. But, that doesn't solve the problem in long term.

Would you mind, posting the /proc/meminfo output.

Linux – Spike in swap utilisation without high swap rate

Interesting fact: swap usage increased from 0 to 8GB... meanwhile, not a single byte were written to disk (sar's pswpout/s show 0). So my assumption is that the swap was allocated/reserved, but not consumed/used.

My best guess is that your server uses vm.overcommit_memory=2 (read vm.overcommit_memory documentation, vm overcommit-accounting). In which case, each byte allocated is counted as used. [I didn't checked].

You may also want to read the thread Linux: Total swap used = swap used by processes + ??.

Best Answer

Related Solutions

Debian – Swap usage too high

Linux – Spike in swap utilisation without high swap rate

Related Question