Cgroups – How to Limit All Processes Except Whitelist to a Single CPU

cgroupscpu usageresources

There is a guide to cgroups from Red Hat which is maybe sort of kind of helpful (but doesn't answer this question).

I know how to limit a specific process to a specific CPU, during the command to start that process, by:

First, putting the following* in /etc/cgconfig.conf:

mount {
  cpuset =  /cgroup/cpuset;
  cpu =     /cgroup/cpu;
  cpuacct = /cgroup/cpuacct;
  memory =  /cgroup/memory;
  devices = /cgroup/devices;
  freezer = /cgroup/freezer;
  net_cls = /cgroup/net_cls;
  blkio =   /cgroup/blkio;
}

group cpu0only {
  cpuset {
    cpuset.cpus = 0;
    cpuset.mems = 0;
  }
}

And then start a process and assign it specifically to that cgroup by using:

cgexec -g cpuset:cpu0only myprocessname

I can limit all instances of a specific process name automatically by (I think this is correct) putting the following in /etc/cgrules.conf:

# user:process  controller  destination
*:myprocessname cpuset      cpu0only

My question is: How can I do the reverse?

In other words, How can I assign all processes except for a specific set of whitelisted processes and their children to a restricted cgroup?

Based on what I have studied, but haven't tested, I believe that a partial solution would be:

Add an "unrestricted" cgroup:

group anycpu {
  cpuset {
    cpuset.cpus = 0-31;
    cpuset.mems = 0;  # Not sure about this param but it seems to be required
  }
}

Assign my process explicitly to the unrestricted group, and everything else to the restricted group:

# user:process  controller  destination
*:myprocessname cpuset      anycpu
*               cpuset      cpu0only

However, the caveat on this seems to be (from reading the docs, not from testing, so grain of salt) that the children of myprocessname will be reassigned to the restricted cpu0only cgroup.

A possible alternative approach would be to create a user to run myprocessname and have all of that user's processes unrestricted, and everything else restricted. However, in my actual use case, the process needs to be run by root, and there are other processes that also must be run by root which should be restricted.

How can I accomplish this with cgroups?

If this is not possible with cgroups (which I now suspect is the case), are my ideas of partial solutions correct and will they work as I think?

*Disclaimer: This is probably not a minimal code example;I don't understand all the parts so I don't know which are not necessary.

Best Answer

UPDATE: Note that the answer below applies to RHEL 6. In RHEL 7, most cgroups are managed by systemd, and libcgroup is deprecated.

Since posting this question I have studied the entire guide that I linked to above, as well as the majority of the cgroups.txt documentation and cpusets.txt. I now know more than I ever expected to learn about cgroups, so I'll answer my own question here.

There are multiple approaches you can take. Our company's contact at Red Hat (a Technical Architect) recommended against a blanket restriction of all processes in preference to a more declarative approach—restricting only the processes we specifically wanted restricted. The reason for this, according to his statements on the subject, is that it is possible for system calls to depend on user space code (such as LVM processes) which if restricted could slow the system down—the opposite of the intended effect. So I ended up restricting several specifically-named processes and leaving everything else alone.

Additionally, I want to mention some cgroup basic data that I was missing when I posted my question.

Cgroups do not depend on libcgroup being installed. However, that is a set of tools for automatically handling cgroup configuration and process assignments to cgroups and can be very helpful.

I found that the libcgroup tools can also be misleading, because the libcgroup package is built on its own set of abstractions and assumptions about your use of cgroups, which are slightly different than the actual kernel level implementation of cgroups. (I can put examples but it would take some work; comment if you're interested.)

Therefore, before using libcgroup tools (such as /etc/cgconfig.conf, /etc/cgrules.conf, cgexec, cgcreate, cgclassify, etc.) I highly recommend getting very familiar with the /cgroup virtual filesystem itself, and manually creating cgroups, cgroup hierarchies (including hierarchies with multiple subsystems attached, which libcgroup sneakily and leakily abstracts away), reassigning processes to different cgroups by running echo $the_pid > /cgroup/some_cgroup_hierarchy/a_cgroup_within_that_hierarchy/tasks, and other seemingly magical tasks that libcgroup performs under the hood.

Another basic concept I was missing was that if the /cgroup virtual filesystem is mounted on your system at all (or more accurately, if any of the cgroup subsystems aka "controllers" are mounted at all), then every process on your entire system is in a cgroup. There is no such thing as "some processes are in a cgroup and some aren't".

There is what is called the root cgroup for a given hierarchy, which owns all the system's resources for the attached subsystems. For example a cgroup hierarchy that has the cpuset and blkio subsystems attached, would have a root cgroup which would own all the cpus on the system and all the blkio on the system, and could share some of those resources with child cgroups. You can't restrict the root cgroup because it owns all your system's resources, so restricting it wouldn't even make sense.

Some other simple data I was missing about libcgroup:

If you use /etc/cgconfig.conf, you should ensure that chkconfig --list cgconfig shows that cgconfig is set to run at system boot.

If you change /etc/cgconfig.conf, you need to run service cgconfig restart to load in the changes. (And problems with stopping the service or running cgclear are very common when fooling around testing. For debugging I recommend, for example, lsof /cgroup/cpuset, if cpuset is the name of the cgroup hierarchy you are using.)

If you want to use /etc/cgrules.conf, you need to ensure the "cgroup rules engine daemon" (cgrulesengd) is running: service cgred start and chkconfig cgred on. (And you should be aware of a possible but unlikely race condition regarding this service, as described in the Red Hat Resource Management Guide in section 2.8.1 at the bottom of the page.)

If you want to fool around manually and set up your cgroups using the virtual filesystem (which I recommend for first use), you can do so and then create a cgconfig.conf file to mirror your setup by using cgsnapshot with its various options.

And finally, the key piece of info I was missing when I wrote the following:

However, the caveat on this seems to be...that the children of myprocessname will be reassigned to the restricted cpu0only cgroup.

I was correct, but there is an option I was unaware of.

cgexec is the command to start a process/run a command and assign it to a cgroup.

cgclassify is the command to assign an already running process to a cgroup.

Both of these will also prevent cgred (cgrulesengd) from reassigning the specified process to a different cgroup based on /etc/cgrules.conf.

Both cgexec and cgclassify support the --sticky flag, which additionally prevents cgred from reassigning child processes based on /etc/cgrules.conf.

So, the answer to the question as I wrote it (though not the setup I ended up implementing, because of the advice from our Red Hat Technical Architect mentioned above) is:

Make the cpu0only and anycpu cgroup as described in my question. (Ensure cgconfig is set to run at boot.)

Make the * cpuset cpu0only rule as described in my question. (And ensure cgred is set to run at boot.)

Start any processes I want unrestricted with: cgexec -g cpuset:anycpu --sticky myprocessname.

Those processes will be unrestricted, and all their child processes will be unrestricted as well. Everything else on the system will be restricted to CPU 0 (once you reboot, since cgred doesn't apply cgrules to already running processes unless they change their EUID). This is not completely advisable, but that was what I initially requested and it can be done with cgroups.

Related Question