Linux – a lot of time spent in intel_idle

cpu usagelinux-kernelpower-managementx86

At the time of this writing htop shows two cores at about 50% each in kernel threads (red bars). From previous boots I know, that this will increase to 3 and even 4 cores at 50% each over time. No thread that htop shows has any cpu usage above 1%. This is kind of strange isn't it?

I tried to investigate it further and using perf record -a as root I found that 11.3% of all samples were in a kernel thread of the command swapper in the function intel_idle.

I will assume that this swapper command corresponds to the strange cpu occupation I see in htop, but can somebody explain to me some of the following questions:

  • what caused this? Is it a bug or something with my settings?
  • does this impact the overall performance of my system?
  • how can I get rid of this?

some more details about my system:

$ uname -a
Linux [...] 4.8.0-0.rc2.git2.2.fc26.x86_64 #1 SMP Wed Aug 17 22:16:04 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
$ lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                8
On-line CPU(s) list:   0-7
Thread(s) per core:    2
Core(s) per socket:    4
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 94
Model name:            Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
Stepping:              3
CPU MHz:               799.804
CPU max MHz:           4200.0000
CPU min MHz:           800.0000
BogoMIPS:              8016.00
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              8192K
NUMA node0 CPU(s):     0-7
Flags:                 [...]

update: Under load the behaviour is a bit different. (note first, that I am now at three cores at 50% each and perf will report 27% inside intel_idle)
I put my cpu under stress with sysbench --test=cpu --num-threads=8 --max-requests=1000000 run and checked htop and perf again. Htop reported 3 cores at 100% in kernel (the same cores that show at 50% during idle) and 5 at 100% userland. Perf reported that sysbench occupied 85% of my cpu and does not explain the remaining 15%… I am not sure what this means. Is my cpu effectively reduced to 85% performance?

Best Answer

This answer is mostly speculative, because I know nothing about power management on Intel processors and I haven't looked at the Linux code, but I think it's plausible.

I think derobert's explanation about power management is what's going on. Power management is a compromise between power consumption and performance. When the processor isn't being used at 100% of its peak performance, it's beneficial to reduce its frequency, which makes it slower but cooler.

Linux varies the CPU frequency over time. How it does so is controlled by policies called governors. The general idea is that when the system hasn't used the CPU at its full performance for a while, it reduces the CPU frequency. Conversely, if the CPU is continuously busy for a while, the kernel increases the frequency.

Seeing intel_idle scheduled means that the core isn't executing code, but in fact in a “suspended” mode where it consumes little power. This brings bigger power savings than merely reducing the frequency, but at a higher cost: although the CPU wakes back up when an interrupt occurs, this takes some time (tens of microseconds? more?).

It's perfectly normal to see intel_idle when you aren't fully utilizing all your cores. This saves a lot of power (both for the processor itself and for cooling devices) compared with having the CPU at full speed all the time. The only reason you might have to disable this mechanism is if you need very low latency. If you run a CPU-intensive application, you'll see less and less intel_idle. Making use the CPU idle mode doesn't cost performance except during the transition period where the kernel hasn't fully decided yet that the system needs a lot of CPU power.

If you fully saturate your cores, you'll reach 0% intel_idle. Note that it might be difficult to saturate all the cores (though a specially-designed benchmark can do it) since all the code and data that is being executed doesn't fit in the CPU cache, the limiting factor will be the RAM access speed. “All the code and data” includes everything that's running on the machine, including your user interface; in practice saturating all the cores is rare.

Related Question