Linux – How to interpret this top command

linuxtop

top - 04:36:16 up 32 days,  2:33,  1 user,  load average: 251.72, 250.54, 231.19
Tasks: 785 total, 249 running, 522 sleeping,   2 stopped,  12 zombie
Cpu(s):  8.9%us, 90.5%sy,  0.4%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.2%si,  0.0%st
Mem:  16313868k total,  7021336k used,  9292532k free,   432196k buffers
Swap:  4194296k total,   295012k used,  3899284k free,   514320k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
18873 nudenude  20   0 97.8m  26m 4944 R  4.0  0.2   0:09.35 php
18140 nudenude  20   0 93984  22m 4940 R  3.7  0.1   0:11.04 php
18178 nudenude  20   0 99.6m  26m 4912 R  3.7  0.2   0:11.11 php
18537 nudenude  20   0  100m  26m 4936 R  3.7  0.2   0:10.31 php
18726 nudenude  20   0 98.3m  25m 4868 R  3.7  0.2   0:09.82 php
18977 nudenude  20   0  101m  25m 4952 R  3.7  0.2   0:09.14 php
19049 nudenude  20   0 99.5m  28m 4920 R  3.7  0.2   0:08.87 php
19852 nudenude  20   0  100m  30m 4912 R  3.7  0.2   0:07.08 php
20504 nudenude  20   0 93024  18m 4880 R  3.7  0.1   0:04.86 php
20562 nudenude  20   0  100m  31m 4940 R  3.7  0.2   0:04.73 php
20681 nudenude  20   0 90912  14m 4940 R  3.7  0.1   0:04.40 php
20685 nudenude  20   0 90656  15m 4928 R  3.7  0.1   0:04.38 php

Some things are strange.

Why system utilize tons of CPU for %sy?

It seems to me the tasks use a lot of virtual memory even though there are still plenty of memory left. So I wonder why?

Also what does 15m, 14m means in res and VIRT. Nothing is shown in man top.

I checked http://unixhelp.ed.ac.uk/CGI/man-cgi?top and non of it is shown.

Note: the load used to be 50%. Moreover, column VIRT is usually 0. However, once in a while it goes to this stuck mode where 90% of CPU is used by kernel. Don't know what the hell is that kernel doing.

very little CPU is used by users. No wonder load is extremely high. But what? What is the kernel doing>

Best Answer

I'm copying this from a man page I wrote for plog, since I was trying to make it clear there:

It is important to understand the difference between virtual address space and physical memory in interpreting some of the above statistics. As the name implies, virtual address space is not real; it’s basically a map of all the memory currently allocated to a process. The limit on the size of this map is the same for each processes (generally, 2-4 GB), and it is not accumulated (ie, you may have dozens or hundreds of processes, each with its own 2-4 GB virtual address space, on a system that only actually has 512 MB of physical memory).

Data cannot actually be stored or retrieved from virtual address space; real data requires real, physical memory. It is the kernel’s job to manage one in relation to another. Virtual space stats (VirtualSz, Data+Stack, and Priv&Write) are useful for considering the structure of a process and the relationship to physical memory use, but with regard to amount of RAM actually used, the physical memory stats (ResidentSz, Share, and Proportion) are what counts.

Top doesn't quite have all those metrics, but the VIRT score is virtual address space, RES refers to physical memory as does SHR. If you are concerned about relative memory usage (ie, one process compared to another), the RES score is more relevant.

Certain parts of VIRT are relevant relative to other processes; visors such as openVZ limit containers based on the total amount of private writable address space, not RSS. Top doesn't report this, but pmap and plog do (see the plog manpage for "Priv&Write"; this was actually part of my motive when writing it).

Related Question