Yes, you can use cgroups and SELinux/AppArmor exclusively to monitor and control the arbitrary code that you will execute.
With cgroups, you can do the following:
- Limit CPU core usage to 1 CPU with the
cpuset
subsystem
- Set memory usage limits with the
memory
subsystem, tracking even the forks. See https://github.com/gsauthof/cgmemtime for an example.
- Prevent network access to anything that isn't on
lo
with net_prio
subsystem.
And with SELinux/AppArmor, you can limit the process's read/write access.
Note: I am unfamiliar with AppArmor, but it is a Mandatory Access Control (MAC) system, meaning that guarding writing and reading is it's job.
Using these systems is a matter of the writing the proper configurations. Of course, all this is much easier said than done. So here are a few reference links to get you started:
Good Luck!
After a few days of intensive research I found two methods of lowering the cpu usage for processes. Generally if you want to lower the cpu usage of the entire machine, there are probably some programs which use the most long-lasting cpu and you should restrict those rather than burden the entire machine. And if you doing this to save battery life then you might also want to control the hardware's power usage with e.g. tuned or powertop, most distros have tools to help you with that.
- Stop/Continue signals
Signals were around since UNIX. You send a SIGSTOP or SIGTSTP to a process (the difference is the former may crash a process if it must do cleanup work, processes are not forced to stop at the latter, use the one that suits your process) to pause it (freeing the CPU and possibly lowering temperature). Then you send a SIGCONT to the process to resume it, taking the CPU. This method will make a series of "spikes" on the cpu graph and will stop the processor from overheating because you're not giving it enough time for that by pausing processes.
A consequence of this method is that these pauses are not smooth, meaning video playing and even web browsing won't be smooth either, so you may want to use this method with shell commands (multi-process programs or commands like Google Chrome or make won't work well with this method either). Obviously, it's not recommended to pause/resume system processes like systemd.
Although you could do this manually, cpulimit is a nice small program that uses this method (it uses SIGSTOP/SIGCONT). Contrary to the description, the cpu % you specify is between 0 and 100 even if you have multiple cores. And you can always suspend a job in the terminal with Ctrl-Z.
- cpupower (highly recommended)
This one is built in the Linux kernel so most distros should provide it (get it here if you don't have it). This command-line utility manages the CPU frequency so it pretty much controls the entire cpu using governor states (e.g., performance, powersave, etc.), it can also do other things. Unlike the pause/resume method, processes are much smoother with this. You'll need to set the maximum frequency for the processor.
- Run
cpupower frequency-info
to see your available processor states.
- As root, type
cpupower frequency-set -u <frequency>
, start with the lowest one you have and then try to find the highest frequency that doesn't overheat.
- (This is optional) If you want, you can install the lm_sensors package which allows you to see your system temperature. Then run
sensors-detect
and answer 'yes' to all the questions. Last, run sensors
to see the current and critical (beyond which the system overheats) temperatures.
At this point the current temperature should be much lower now.
Be aware though some performance-intensive programs like games might hang after typing the above command, if you get a popup window with that message you should wait for the program rather than force quit. Note you have to type this command every time the system reboots (unless you can get it to run automatically).
See this and this for more information on cpupower.
Best Answer
First of all, most of what comes up in search on the web has been deprecated. For example
cgmanager
is no longer supported on new systemd versions. Don't follow 99% of what comes up in web serches as far as usingcuplimit
,nice
,cgset
or other tools for this job. They either won't work at all as advertised (as in the case of cgroup management tools that expect you to create your own hierarchy) or won't get the job done without resorting to lots of hacks (as in the case of using 'nice' levels to manage whole groups of processes).The good news is that along with those deprecations (and pursuing the traditional all-devouring octopus monster modus operandi of systemd) a default configuration is in place for everything on the system, and tweaking it for systemd services is trivial. Just add an overlay configuration to the service you want to limit:
Add a section with whatever resource control values you want to override. In my case I came up with this:
These values are not all necessary, but the first two answer the question as asked:
CPUWeight
defaults to 100 for all processes on the system. Setting a low value still lets process use the CPU if nothing else is effectively keeping the system responsive for other tasks while not slowing down the results much. This is a arbitrary weight integer.CPUQuota
is an absolute limit on how much CPU time is granted even if nothing else is going on. This is a percent value. In my case it wasn't really necessary to set this to fix the resource hogging issue. I ended up setting it anyway to keep the CPU temperature down when lots of CI jobs pile up.IOWeight
is much the same asCPUWeight
, in this case used to keep disks free for system tasks and only keep them busy with CI jobs when nothing else is going on.MemorySwapMax
also isn't in scope for the question, in my case I ended up adding it because the ray traver (povray
) running in some of the CI jobs seems to think using 30+ Gigs of swap in addition to the 30+ Gigs of RAM in this system is good idea just because it is there. It runs faster if you don't let it use it at all. This is probably something better configured in povray, but this way I don't have to police what happens inside of CI jobs and don't have to disable the system swap.Lastly these values can be changed on the fly without restarting services by running
systemctl daemon-reload
. This is quite handy to watch the effect of the changes right away.