Systemd – How to Limit CPU Usage with Systemd-Run

cgroupscpulimitsystemdsystemd-run

I have a buggy program which uses 100% CPU even when it's idle. Since fixing it isn't practical at the moment, I'd like to just limit it to be able to use no more than 10% CPU. However no matter what I do, the process always chews up 100% of one CPU.

I found instructions on the Arch Wiki that tell me to create a file containing this:

# cpulimit.slice
[Slice]
CPUQuota=10%

Apparently I can then launch a shell using these limits, like this:

systemd-run --slice=cpulimit.slice --uid=myuser --shell

This seems to work and after entering in my sudo password I get a shell, so I run a simple test that will use 100% CPU and I can stop with Ctrl+C:

while true; do true; done

I expect this to use no more than 10% CPU since it's running inside the slice, however it always uses 100% CPU!

What am I doing wrong?

Best Answer

Just discovered the problem. The .slice file must be placed in /etc/systemd/system/ and you have to run systemctl daemon-reload first. Then it all works.

Unfortunately the systemd-run command doesn't give you any error messages if you use an invalid --slice parameter, it just creates a new slice with no additional restrictions, called whatever name you supplied.

Related Solutions

Systemd Cgroups – Set a Default Resource Limit for All Users

There seems no officially supported way to do that. (This is incorrect. See the bottom) An officially discouraged way (because it manipulates the cgroup) is as follows:

Make the following file as /etc/systemd/system/user@.service.d/set-memhigh.conf

[Service]
Type=simple
ExecStartPost=+/root/set-memoryhigh.sh %i

Then make the following file as "/root/set-memoryhigh.sh"

#!/bin/bash
exec >>/var/tmp/log.txt 2>&1 # for logging
set -x # for logging 
for d in /sys/fs/cgroup /sys/fs/cgroup/user.slice /sys/fs/cgroup/user.slice/user-$1.slice; do
  echo "+memory" >>${d}/cgroup.subtree_control
done
/bin/echo "24G" >> /sys/fs/cgroup/user.slice/user-$1.slice/memory.high

You can see if it works or not by running

cat /sys/fs/cgroup/user.slice/user-${UID}.slice/memory.high

If "/sys/fs/cgroup/user.slice" does not exist, then the unified cgroup hierarchy is not enabled. We have to enable it as https://unix.stackexchange.com/a/452728/297666

Although it works, I am not sure if you like this...

Note added on July 25: Making the following file as /etc/systemd/system/user-1000.slice for each user (replacing 1000 by user's UID) imposes a memory limitation on that user. I verified it on systemd 237 on ubuntu 18.04 and Debian strecth with systemd 237 installed from stretch-backports:

[Slice]
Slice=user.slice
MemoryHigh=24G

The inconvenience is that we have to make the above file for each user. With systemd 239, we can make the above file as /etc/systemd/system/user-.slice.d/memory.conf and the memory limitation is imposed on every user. But there is a bug in systemd 239 (this bug was corrected in 240) and it does not work as intended. To work around the bug, make the following file as user-0.slice and run systemctl enable user-0.slice. We do not have to make the following file for each user.

[Unit]
Before=systemd-logind.service
[Slice]
Slice=user.slice
[Install]
WantedBy=multi-user.target

Ubuntu – Limit total memory usage for multiple instances of systemd service

I finally got this working! The trick was to create my own slice and set it in the service file, like this:

[Service]
# Everything else as in the original question
Slice=my_service_limit.slice

And create a slice unit file /lib/systemd/system/my_service_limit.slice that looks like this:

[Unit]
Description=Slice that limits memory for all my services

[Slice]
# MemoryHigh works only in "unified" cgroups mode, NOT in "hybrid" mode
MemoryHigh=500M
# MemoryMax works in "hybrid" cgroups mode, too
MemoryMax=600M

Note: be careful with the naming of the slice, as - is a hierarchy separator, as explained in https://systemd.io/CGROUP_DELEGATION - a very helpful page for anyone trying to configure this. You can check whether the service is really using the configured slice by looking at the output of systemctl status myservice - it should say:

  CGroup: /my_service_limit.slice/myservice@myinstance.service

It was not necessary to set systemd.unified_cgroup_hierarchy=1 (as per Ryutaroh Matsumoto's answer) for MemoryMax to work, but it was necessary for MemoryHigh - even in "hybrid" mode (the default in Ubuntu 18.04) it's silently ignored.

Also worth noting that these only apply to the physical RAM used - they do not include swap space used. (There is a separate MemorySwapMax setting, but no MemorySwapHigh, it seems.)

Best Answer

Related Solutions

Systemd Cgroups – Set a Default Resource Limit for All Users

Ubuntu – Limit total memory usage for multiple instances of systemd service

Related Question