Linux – What’s the Linux kernel’s behaviour when processes in a cgroup hit their memory limit

Someone suggested in your hear cgroups. Well, try to seek that direction as it can provide you with:

• applied to a group of task you choose (thus not system wide but neither per process)
• the limits are set for the group
• the limits are static
• they can enforce hard limit on memory and/or memory+swap

Something like that could bring you closer to your goals:

group limited {
  memory {
    memory.limit_in_bytes = 50M;
    memory.memsw.limit_in_bytes = 50M;
  }
}

This tells that the tasks under this cgroup can use at maximum 50M of memory only and 50M of memory+swap, so when the memory is full, it won't swap, but if the memory is not full and some data could be mapped in swap, this could be allowed.

Here is an excerpt from the cgroup's memory documentation:

By using memsw limit, you can avoid system OOM which can be caused by swap shortage.

I am not sure if this answers your question, but I found this perl script that claims to do exactly what you are looking for. The script implements its own system for enforcing the limits by waking up and checking the resource usage of the process and its children. It seems to be well documented and explained, and has been updated recently.

As slm said in his comment, cgroups can also be used for this. You might have to install the utilities for managing cgroups, assuming you are on Linux you should look for libcgroups.

sudo cgcreate -t $USER:$USER -a $USER:$USER -g memory:myGroup

Make sure $USER is your user.

Your user should then have access to the cgroup memory settings in /sys/fs/cgroup/memory/myGroup.

You can then set the limit to, lets say 500 MB, by doing this:

echo 500000000 > /sys/fs/cgroup/memory/myGroup/memory.limit_in_bytes

Now lets run Vim:

cgexec -g memory:myGroup vim

The vim process and all its children should now be limited to using 500 MB of RAM. However, I think this limit only applies to RAM and not swap. Once the processes reach the limit they will start swapping. I am not sure if you can get around this, I can not find a way to limit swap usage using cgroups.