I'm running some .NET Core processes as systemd services under Ubuntu 16.04 (soon to be 18.04). I have a systemd configuration file (/lib/systemd/system/myservice@.service
) that looks like this:
[Unit]
Description=My Service %i
[Service]
Type=simple
User=myservice
EnvironmentFile=/etc/environment
WorkingDirectory=/home/myservice/instances/%i
ExecStart=/opt/dotnet/dotnet My.Service.dll
[Install]
WantedBy=multi-user.target
I want to limit the total amount of RAM that all instances of this service can use. I don't want to limit the RAM for any individual instance to less than that, so settings like MemoryHigh
and MemoryMax
aren't helpful.
I know that systemd creates a cgroup for the service template, so I want to change the memory limit for that cgroup somehow.
On Ubuntu 18.04 I can manually edit /sys/fs/cgroup/memory/system.slice/system-myservice.slice/memory.limit_in_bytes
and this basically does what I want (processes get killed when the total memory usage exceeds the limit), but there are some issues with this approach:
- This file doesn't always exist on boot until the service is started.
- On Ubuntu 18.04 this file gets overwritten whenever
systemctl daemon-reload
is called. - Trying to write to the file sometimes returns
write error: Device or resource busy
(Under Ubuntu 16.04 the limit seems to be reset whenever my service starts, so it has no effect.)
Is there some way to get systemd itself to set this value, so I don't have to fight against it? Or some other way to limit the total memory use for a group of processes? They all run as the same user, so it would be OK to limit the RAM used by that user, for example.
I even tried manually creating a cgroup (cgcreate -t myservice:myservice -g memory:mycgroup
) and then changing ExecStart
in the service configuration to /usr/bin/cgexec -g memory:mycgroup /opt/dotnet/dotnet My.Service.dll
and this again sort of works, but not reliably: the memory limit I wrote to memory.limit_in_bytes
got reset at some point and I don't know when or why.
Best Answer
I finally got this working! The trick was to create my own slice and set it in the service file, like this:
And create a slice unit file
/lib/systemd/system/my_service_limit.slice
that looks like this:Note: be careful with the naming of the slice, as
-
is a hierarchy separator, as explained in https://systemd.io/CGROUP_DELEGATION - a very helpful page for anyone trying to configure this. You can check whether the service is really using the configured slice by looking at the output ofsystemctl status myservice
- it should say:It was not necessary to set
systemd.unified_cgroup_hierarchy=1
(as per Ryutaroh Matsumoto's answer) for MemoryMax to work, but it was necessary for MemoryHigh - even in "hybrid" mode (the default in Ubuntu 18.04) it's silently ignored.Also worth noting that these only apply to the physical RAM used - they do not include swap space used. (There is a separate MemorySwapMax setting, but no MemorySwapHigh, it seems.)