I ran top, and I see lots of rcuos/rcuob processes. Here's a sample of top output:
top - 21:41:15 up 22 days, 19:20, 1 user, load average: 0.00, 0.01, 0.05
Tasks: 332 total, 1 running, 331 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.0%sy, 0.0%ni, 99.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 32910904k total, 2232224k used, 30678680k free, 246636k buffers
Swap: 33517564k total, 0k used, 33517564k free, 1048244k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3098 root 20 0 11.8g 113m 9.9m S 1 0.4 67:57.98 asterisk
12 root 20 0 0 0 0 S 0 0.0 0:53.61 rcuos/3
1130 root 39 19 0 0 0 S 0 0.0 78:26.78 kipmi0
1735 root 20 0 15988 740 540 S 0 0.0 17:34.69 irqbalance
1875 zabbix 20 0 89416 2472 1828 S 0 0.0 12:55.47 zabbix_agentd
1 root 20 0 24316 2288 1336 S 0 0.0 0:05.06 init
2 root 20 0 0 0 0 S 0 0.0 0:00.25 kthreadd
3 root 20 0 0 0 0 S 0 0.0 0:02.40 ksoftirqd/0
5 root 0 -20 0 0 0 S 0 0.0 0:00.00 kworker/0:0H
6 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/u48:0
8 root 20 0 0 0 0 S 0 0.0 4:55.92 rcu_sched
9 root 20 0 0 0 0 S 0 0.0 1:31.03 rcuos/0
10 root 20 0 0 0 0 S 0 0.0 0:54.04 rcuos/1
11 root 20 0 0 0 0 S 0 0.0 0:53.16 rcuos/2
13 root 20 0 0 0 0 S 0 0.0 0:45.96 rcuos/4
14 root 20 0 0 0 0 S 0 0.0 0:41.84 rcuos/5
15 root 20 0 0 0 0 S 0 0.0 0:02.56 rcuos/6
16 root 20 0 0 0 0 S 0 0.0 0:07.25 rcuos/7
17 root 20 0 0 0 0 S 0 0.0 0:12.06 rcuos/8
18 root 20 0 0 0 0 S 0 0.0 0:12.14 rcuos/9
19 root 20 0 0 0 0 S 0 0.0 0:11.21 rcuos/10
20 root 20 0 0 0 0 S 0 0.0 0:05.66 rcuos/11
21 root 20 0 0 0 0 S 0 0.0 1:39.42 rcuos/12
22 root 20 0 0 0 0 S 0 0.0 0:35.66 rcuos/13
23 root 20 0 0 0 0 S 0 0.0 0:26.46 rcuos/14
24 root 20 0 0 0 0 S 0 0.0 0:00.88 rcuos/15
25 root 20 0 0 0 0 S 0 0.0 0:00.30 rcuos/16
26 root 20 0 0 0 0 S 0 0.0 0:31.93 rcuos/17
27 root 20 0 0 0 0 S 0 0.0 0:18.58 rcuos/18
28 root 20 0 0 0 0 S 0 0.0 0:20.53 rcuos/19
29 root 20 0 0 0 0 S 0 0.0 0:44.95 rcuos/20
30 root 20 0 0 0 0 S 0 0.0 0:01.92 rcuos/21
31 root 20 0 0 0 0 S 0 0.0 0:00.24 rcuos/22
32 root 20 0 0 0 0 S 0 0.0 0:05.01 rcuos/23
33 root 20 0 0 0 0 S 0 0.0 0:00.00 rcu_bh
34 root 20 0 0 0 0 S 0 0.0 0:00.00 rcuob/0
35 root 20 0 0 0 0 S 0 0.0 0:00.00 rcuob/1
36 root 20 0 0 0 0 S 0 0.0 0:00.00 rcuob/2
37 root 20 0 0 0 0 S 0 0.0 0:00.00 rcuob/3
38 root 20 0 0 0 0 S 0 0.0 0:00.00 rcuob/4
39 root 20 0 0 0 0 S 0 0.0 0:00.00 rcuob/5
40 root 20 0 0 0 0 S 0 0.0 0:00.00 rcuob/6
41 root 20 0 0 0 0 S 0 0.0 0:00.00 rcuob/7
42 root 20 0 0 0 0 S 0 0.0 0:00.00 rcuob/8
43 root 20 0 0 0 0 S 0 0.0 0:00.00 rcuob/9
44 root 20 0 0 0 0 S 0 0.0 0:00.00 rcuob/10
45 root 20 0 0 0 0 S 0 0.0 0:00.00 rcuob/11
46 root 20 0 0 0 0 S 0 0.0 0:00.00 rcuob/12
47 root 20 0 0 0 0 S 0 0.0 0:00.00 rcuob/13
48 root 20 0 0 0 0 S 0 0.0 0:00.00 rcuob/14
49 root 20 0 0 0 0 S 0 0.0 0:00.00 rcuob/15
50 root 20 0 0 0 0 S 0 0.0 0:00.00 rcuob/16
51 root 20 0 0 0 0 S 0 0.0 0:00.00 rcuob/17
52 root 20 0 0 0 0 S 0 0.0 0:00.00 rcuob/18
53 root 20 0 0 0 0 S 0 0.0 0:00.00 rcuob/19
54 root 20 0 0 0 0 S 0 0.0 0:00.00 rcuob/20
55 root 20 0 0 0 0 S 0 0.0 0:00.00 rcuob/21
56 root 20 0 0 0 0 S 0 0.0 0:00.00 rcuob/22
57 root 20 0 0 0 0 S 0 0.0 0:00.00 rcuob/23
58 root RT 0 0 0 0 S 0 0.0 0:00.87 migration/0
59 root RT 0 0 0 0 S 0 0.0 0:08.47 watchdog/0
60 root RT 0 0 0 0 S 0 0.0 0:08.41 watchdog/1
What are these processes?
Best Answer
The key article explaining this is here: https://lwn.net/Articles/522262/
For faster synchronization, many structures in the kernel are "RCU", or read-copy-update. Many threads can read simultaneously without blocking, and threads that want to alter the structure can do so quickly by copying the relevant data to a new location and changing the reference ("removal"). However, the old data must live for some time longer so that reading threads can finish their work. When the old data no longer has readers working on it, it can finally be deleted ("reclamation").
The rcuos, rcuob threads are there to handle the reclamation part of this process. It used to be handled as part of software IRQ, which meant it could take processor power away from critical user processes at inopportune times. By moving this to a bunch of dedicated rcu** threads, the system can be tuned to make this reclamation process less intrusive.
Reducing OS jitter is not the purpose of these threads, though it may be the main reason they exist.