I run the following command (which does ls for 10,000 times) in a Ubuntu Linux terminal and use the top command to monitor the CPU usage. While top shows the total CPU usage (line 2) is about 48% (13.1% user + 34.9% kernel), the list below does not reflect the correct CPU usage. Only 6.5% CPU is associated with the bash process. NOTE: I already turned off the Irix mode so the 6.5% is on the same scale as line 2.
for i in {1..10000}; do (ls /tmp/ >/dev/null); done
I also tried htop with 'Hide kernel threads' option unchecked, but got the same result.
Which part of the system is using the CPU (the kernel?) and why it is hidden from top/htop?
Best Answer
You are running a lot of extremely short-lived processes. You aren't going to see them much in the top output.
Top measures system activity periodically (often once per second). At each refresh, it goes through the process list and collects statistics for each process. Depending on the luck of the draw of scheduling, there may be either zero or one
lsprocess at that point. Iftopdoesn't see alsprocess, it won't display one. Even iftopsees a process that isls, getting the statistics isn't atomic; if that process dies between the timetopenumerates the processes and the time it gets around to reading the statistics of that process, the process won't be shown. So pretty often there isn't anylsprocess to list.Even when there is an
lsprocess, that process hasn't run for long, so it accounts for a negligible part of the CPU usage in the last second. It's likely to be somewhere near the bottom of the list.Because CPU usage is not measured atomically and is only an approximation, the CPU total usage statistics collected by the kernel doesn't always match the sum of the usage statistics collected process by process. The sums would match for sufficiently long-running processes, but when there are short-lived processes, as this example illustrates, they can differ a lot.