Linux – Attempting to Abbreviate kSAR’s PDF output

The output you have provided looks different from the standard sar -P ALL or sar -u output. I'm not sure if you hand formatted it, or if you're running it through another tool, but I think there's enough information there to figure this out.

Here's the important piece of information, obtained from the man page for sar

Note: On SMP machines a processor that does not have any activity at all (0.00 for every field) is a disabled (offline) processor.

Since you are running in a cluster, it seems fairly safe to assume that you are using SMP machines.

Note that in example 2 and 3, only 12 of the 24 cores are reporting statistics. If you assume that these cores are disabled, as mentioned in the man page, then the statistics make sense.

Let's update your data as follows, to indicate a disabled core with -

0   | 8       0   | 15      0   | 17
1   | 25      1   | 94      1   | 32
2   | 79      2   | 100     2   | 97
3   | 62      3   | 99      3   | 71
4   | 5       4   | 13      4   | 5
5   | 7       5   | 13      5   | 23
6   | 6       6   | 99      6   | 71
7   | 7       7   | 44      7   | 98
8   | 11      8   | 12      8   | 48
9   | 17      9   | -       9   | -
10  | 33      10  | -       10  | -
11  | 64      11  | -       11  | -
12  | 6       12  | 0       12  | 38
13  | 6       13  | -       13  | -
14  | 6       14  | -       14  | -
15  | 6       15  | -       15  | -
16  | 12      16  | 12      16  | 37
17  | 15      17  | -       17  | -
18  | 62      18  | -       18  | -
19  | 69      19  | -       19  | -
20  | 7       20  | 3       20  | 42
21  | 7       21  | -       21  | -
22  | 6       22  | -       22  | -
23  | 7       23  | -       23  | -

Then we can use the following to calculate the averages (this is a quick oneliner I wrote, I'm sure something better could be written.)

$ awk '{idle = $3; output += idle; if (idle >= 0) {cores += 1} } END {printf "Ave idle%%: %f  Cores: %d\n", output / cores, cores }' input_file
Ave idle%: 22.208333  Cores: 24
$ awk '{idle = $6; output += idle; if (idle >= 0) {cores += 1} } END {printf "Ave idle%%: %f  Cores: %d\n", output / cores, cores }' input_file
Ave idle%: 42.000000  Cores: 12
$ awk '{idle = $9; output += idle; if (idle >= 0) {cores += 1} } END {printf "Ave idle%%: %f  Cores: %d\n", output / cores, cores }' input_file
Ave idle%: 48.250000  Cores: 12

Note that the number of cores in example 2 and 3 is 12, and the averages match what you see in your example output.

It looks like at some point between your first and second case, half your CPU cores have been disabled.

A quick summary on your questions:

1. Missing data probably means that processor core is disabled.
2. The U in the average line is different than the U in that man page. The U referred to in the man page should show up under the processor ID column.
3. The output you've provided is different than the standard sar output and there's not enough information provided to determine what the U or all on the average line refer to. The first number appears to be the idle% on active cores though.