Linux – How to Limit Processes to All But One CPU

While it can be an abuse for memory, it isn't for CPU: when a CPU is idle, a running process (by "running", I mean that the process isn't waiting for I/O or something else) will take 100% CPU time by default. And there's no reason to enforce a limit.

Now, you can set up priorities thanks to nice. If you want them to apply to all processes for a given user, you just need to make sure that his login shell is run with nice: the child processes will inherit the nice value. This depends on how the users log in. See Prioritise ssh logins (nice) for instance.

Alternatively, you can set up virtual machines. Indeed setting a per-process limit doesn't make much sense since the user can start many processes, abusing the system. With a virtual machine, all the limits will be global to the virtual machine.

Another solution is to set /etc/security/limits.conf limits; see the limits.conf(5) man page. For instance, you can set the maximum CPU time per login and/or the maximum number of processes per login. You can also set maxlogins to 1 for each user.

Shares are relative among groups. For example, assigning 25% of CPU means that a cgroup will observe "at least" that much of CPU, but it can use more. From the red hat docs on cgroups:

Note that shares of CPU time are distributed per all CPU cores on multi-core systems. Even if a cgroup is limited to less than 100% of CPU on a multi-core system, it may use 100% of each individual CPU core.

...

The actual amount of CPU time that is available to a cgroup can vary depending on the number of cgroups that exist on the system. If a cgroup has a relative share of 1000 and two other cgroups have a relative share of 500, the first cgroup receives 50% of all CPU time in cases when processes in all cgroups attempt to use 100% of the CPU. However, if another cgroup is added with a relative share of 1000, the first cgroup is only allowed 33% of the CPU (the rest of the cgroups receive 16.5%, 16.5%, and 33% of CPU).

If you want a hard limit on CPU bandwidth, you can use cpu.cfs_quota_us and cpu.cfs_period_us. From the Kernel's CFS docs:

The bandwidth allowed for a group is specified using a quota and period. Within each given "period" (microseconds), a group is allowed to consume only up to "quota" microseconds of CPU time. When the CPU bandwidth consumption of a group exceeds this limit (for that period), the tasks belonging to its hierarchy will be throttled and are not allowed to run again until the next period.

However, neither of these two options will allow you to port a group between distinct CPUs without having to change the cgroups configuration.