top - 04:36:16 up 32 days, 2:33, 1 user, load average: 251.72, 250.54, 231.19
Tasks: 785 total, 249 running, 522 sleeping, 2 stopped, 12 zombie
Cpu(s): 8.9%us, 90.5%sy, 0.4%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.2%si, 0.0%st
Mem: 16313868k total, 7021336k used, 9292532k free, 432196k buffers
Swap: 4194296k total, 295012k used, 3899284k free, 514320k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
18873 nudenude 20 0 97.8m 26m 4944 R 4.0 0.2 0:09.35 php
18140 nudenude 20 0 93984 22m 4940 R 3.7 0.1 0:11.04 php
18178 nudenude 20 0 99.6m 26m 4912 R 3.7 0.2 0:11.11 php
18537 nudenude 20 0 100m 26m 4936 R 3.7 0.2 0:10.31 php
18726 nudenude 20 0 98.3m 25m 4868 R 3.7 0.2 0:09.82 php
18977 nudenude 20 0 101m 25m 4952 R 3.7 0.2 0:09.14 php
19049 nudenude 20 0 99.5m 28m 4920 R 3.7 0.2 0:08.87 php
19852 nudenude 20 0 100m 30m 4912 R 3.7 0.2 0:07.08 php
20504 nudenude 20 0 93024 18m 4880 R 3.7 0.1 0:04.86 php
20562 nudenude 20 0 100m 31m 4940 R 3.7 0.2 0:04.73 php
20681 nudenude 20 0 90912 14m 4940 R 3.7 0.1 0:04.40 php
20685 nudenude 20 0 90656 15m 4928 R 3.7 0.1 0:04.38 php
Some things are strange.
Why system utilize tons of CPU for %sy?
It seems to me the tasks use a lot of virtual memory even though there are still plenty of memory left. So I wonder why?
Also what does 15m, 14m means in res and VIRT. Nothing is shown in man top.
I checked http://unixhelp.ed.ac.uk/CGI/man-cgi?top and non of it is shown.
Note: the load used to be 50%. Moreover, column VIRT is usually 0. However, once in a while it goes to this stuck mode where 90% of CPU is used by kernel. Don't know what the hell is that kernel doing.
very little CPU is used by users. No wonder load is extremely high. But what? What is the kernel doing>
Best Answer
I'm copying this from a man page I wrote for plog, since I was trying to make it clear there:
Top doesn't quite have all those metrics, but the VIRT score is virtual address space, RES refers to physical memory as does SHR. If you are concerned about relative memory usage (ie, one process compared to another), the RES score is more relevant.
Certain parts of VIRT are relevant relative to other processes; visors such as openVZ limit containers based on the total amount of private writable address space, not RSS. Top doesn't report this, but pmap and plog do (see the plog manpage for "Priv&Write"; this was actually part of my motive when writing it).