Ensure Exclusive CPU Availability for a Process – Performance Management in Debian

debianperformanceprocessprocess-management

At first, the question seems to be a little bit silly/confusing as the OS does the job of managing process execution.

However, I want to measure how much some processes are CPU/IO-bound and I feel like my OS is interfering on my experiments with, for instance, scheduled OS processes.

Take as an example the following situation:
I ran the process A twice and got the following output from the tool "time" (time columns in seconds):

+---+-------+---------+-----------+---------+
|Run|Process|User Time|System Time|Wall time|
+---+-------+---------+-----------+---------+
|1  |A      |196.3    |5.12       |148.86   |
|2  |A      |190.79   |4.93       |475.46   |
+---+-------+---------+-----------+---------+

As we can see, although the user and sys time are similar, the elapsed time of both drastically changes (diff. of ~5 min). Feels like something in my environment caused some sort of contention.

I want to stop every possible background process/services to avoid any kind of noise during my experiments but I consider myself a novice/intermediate unix-user and I don't know how to guarantee this.

I'm using Linux 4.4.0-45-generic with Ubuntu 14.04 LTS 64 bit.

I really appreciate the assistance. If you guys need any missing information, I will promptly edit my post.

CPU Info

$ grep proc /proc/cpuinfo | wc -l
8
$ lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                8
On-line CPU(s) list:   0-7
Thread(s) per core:    2
Core(s) per socket:    4
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 60
Stepping:              3
CPU MHz:               4002.609
BogoMIPS:              7183.60
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              8192K
NUMA node0 CPU(s):     0-7

Best Answer

You have a kernel option configuration where a CPU won't be used by the OS, it is called isolcpus.

isolcpus — Isolate CPUs from the kernel scheduler.

Synopsis isolcpus= cpu_number [, cpu_number ,...]

Description Remove the specified CPUs, as defined by the cpu_number values, from the general kernel SMP balancing and scheduler algroithms. The only way to move a process onto or off an "isolated" CPU is via the CPU affinity syscalls. cpu_number begins at 0, so the maximum value is 1 less than the number of CPUs on the system.

This configuration I am about to describe how to setup, can have far more uses than for testing.

Meru for instance, uses this technology in their Linux-based AP controllers, to keep the network traffic from interfering with the inner workings of the OS, namely I/O operations.

I also use it in a very busy web frontend, for quite the same reasons: I have found out from life experience that I lost control too regularly for my taste of that server ; had to reboot it forcefully until I separated the front end daemon on it´s own dedicated CPUs.

As you have 8 CPUs, that you can check with the output of the command:

$ grep -c proc /proc/cpuinfo
8

or

$ lscpu | grep '^CPU.s'
CPU(s):                8

Add in Debian/Ubuntu in the file /etc/default/grub to the option GRUB_CMDLINE_LINUX:

GRUB_CMDLINE_LINUX="isolcpus=7"

(it is 7, because it starts in 0, and you have 8 cores)

Then run,

sudo update-grub

This is telling the kernel to not use one of your cores.

Reboot the system.

Then start your process.

Immediately after starting it, you can change for the 8th CPU (7 because 0 is the 1st), and be quite sure you are the only one using that CPU.

For that, use the command:

taskset -cp 7 PID_number

taskset - retrieve or set a processes’s CPU affinity

SYNOPSIS

   taskset [options] [mask | list ] [pid | command [arg]...]

DESCRIPTION

taskset is used to set or retrieve the CPU affinity of a running pro cess given its PID or to launch a new COMMAND with a given CPU affinity. CPU affinity is a scheduler property that "bonds" a process to a given set of CPUs on the system. The Linux scheduler will honor the given CPU affinity and the process will not run on any other CPUs. Note that the Linux scheduler also supports natural CPU affinity: the scheduler attempts to keep processes on the same CPU as long as practical for performance reasons. Therefore, forcing a specific CPU affinity is useful only in certain applications.

For reading more about it, see: isolcpus, numactl and taskset

Also using ps -eF you should see in the PSR column the processor being used.

I have a server with CPU 2 and 3 isolated, and indeed, it can be seen with ps -e the only process in userland as intended, is pound.

# ps -eo psr,command | tr -s " " | grep "^ [2|3]"
 2 [cpuhp/2]
 2 [watchdog/2]
 2 [migration/2]
 2 [ksoftirqd/2]
 2 [kworker/2:0]
 2 [kworker/2:0H]
 3 [cpuhp/3]
 3 [watchdog/3]
 3 [migration/3]
 3 [ksoftirqd/3]
 3 [kworker/3:0]
 3 [kworker/3:0H]
 2 [kworker/2:1]
 3 [kworker/3:1]
 3 [kworker/3:1H]
 3 /usr/sbin/pound

If you compare it with the non-isolated CPUs, they are running many more things (the window below slides):

# ps -eo psr,command | tr -s " " | grep "^ [0|1]"
 0 init [2]
 0 [kthreadd]
 0 [ksoftirqd/0]
 0 [kworker/0:0H]
 0 [rcu_sched]
 0 [rcu_bh]
 0 [migration/0]
 0 [lru-add-drain]
 0 [watchdog/0]
 0 [cpuhp/0]
 1 [cpuhp/1]
 1 [watchdog/1]
 1 [migration/1]
 1 [ksoftirqd/1]
 1 [kworker/1:0]
 1 [kworker/1:0H]
 1 [kdevtmpfs]
 0 [netns]
 0 [khungtaskd]
 0 [oom_reaper]
 1 [writeback]
 0 [kcompactd0]
 0 [ksmd]
 1 [khugepaged]
 0 [crypto]
 1 [kintegrityd]
 0 [bioset]
 1 [kblockd]
 1 [devfreq_wq]
 0 [watchdogd]
 0 [kswapd0]
 0 [vmstat]
 1 [kthrotld]
 0 [kworker/0:1]
 0 [deferwq]
 0 [scsi_eh_0]
 0 [scsi_tmf_0]
 1 [vmw_pvscsi_wq_0]
 0 [bioset]
 1 [jbd2/sda1-8]
 1 [ext4-rsv-conver]
 0 [kworker/0:1H]
 1 [kworker/1:1H]
 1 [bioset]
 0 [bioset]
 1 [bioset]
 1 [bioset]
 1 [bioset]
 1 [bioset]
 1 [bioset]
 1 [bioset]
 0 [jbd2/sda3-8]
 1 [ext4-rsv-conver]
 1 /usr/sbin/rsyslogd
 0 /usr/sbin/irqbalance --pid=/var/run/irqbalance.pid
 1 /usr/sbin/cron
 0 /usr/sbin/sshd
 1 /usr/sbin/snmpd -Lf /dev/null -u snmp -g snmp -I -smux -p /var/run/snmpd.pid
 1 /sbin/getty 38400 tty1
 1 /lib/systemd/systemd-udevd --daemon
 0 /usr/sbin/xinetd -pidfile /run/xinetd.pid -stayalive
 1 [kworker/1:2]
 0 [kworker/u128:1]
 0 [kworker/0:2]
 0 [bioset]
 1 [xfsalloc]
 1 [xfs_mru_cache]
 1 [jfsIO]
 1 [jfsCommit]
 0 [jfsCommit]
 0 [jfsCommit]
 0 [jfsCommit]
 0 [jfsSync]
 1 [bioset]
 0 /usr/bin/monit -c /etc/monit/monitrc
 1 /usr/sbin/pound
 0 sshd: rui [priv]
 0 sshd: rui@pts/0,pts/1
 1 -bash
 1 -bash
 1 -bash
 1 [kworker/u128:0]
 1 -bash
 0 sudo su
 1 su
 1 bash
 0 bash
 0 logger -t cmdline root[/home/rui] 
 1 ps -eo psr,command
 0 tr -s 
 0 grep ^ [0|1]
 0 /usr/bin/vmtoolsd
Related Question