htopprocesspsthreadtop
In ps xf
26395 pts/78 Ss 0:00 \_ bash
27016 pts/78 Sl+ 0:04 | \_ unicorn_rails master -c config/unicorn.rb
27042 pts/78 Sl+ 0:00 | \_ unicorn_rails worker[0] -c config/unicorn.rb
In htop, it shows up like:
Why does htop show more process than ps?
A complete re-write of my previous post. Got a bit curious and checked out further.
In short: the reason for the difference is that openSUSE uses a patched version of top and free that adds some extra values to `cached'.
top, free, htop, ...:
Usage is calculated by reading data from /proc/meminfo:
E.g.:
#free:
Row Column | Corresponding /proc/meminfo entry
-----|--------|----------------------------------
Mem:
total : MemTotal
used : MemTotal - MemFree
free : MemFree
shared : MemShared
buffers : Buffers
cached : Cached
-----|--------|----------------------------------
-/+ buffers/cache:
used : (MemTotal - MemFree) - (Buffers + Cached)
free : MemFree + (Buffers + Cached)
#htop:
Used U* : ((MemTotal - MemFree) - (Buffers + Cached)) / 1024
*I'm using the name Used U for memory used by User Mode.
Aka Used minus (Cached + Buffers).
So in reality same calculation is used.
htop display the following in the memory meter:
[Used U % of total | Buffers % of total | Cached % of total ] UsedU MB
(MB is actually MiB.)
The base for free and top on Debian, Fedora, openSuse is is procps-ng.
However, each flavour add their own patches that might, or migh not become
part of the main project.
Under openSUSE we find various additions to the top/free (procps) package. The ones to take notice of here is some additional values used to represent the cache value. (I did not include these in my previous post as my system uses a "clean" procps.)
In /proc/meminfo we have Slab which is in-kernel data structure cache. As a sub category we find SReclaimable which is part of Slab that might be reclaimed for other use both by Kernel and User Mode.
Further we have SwapCached which is memory that once was swapped out, is swapped in but also is in the swap-file. Thus if one need to swap it out again, this is already done.
Lastly there is NFS_Unstable which are pages sent to the server but not yet committed to stable storage.
The following values are added to cache in the openSUSE patched version:
SReclaimable
SwapCached
NFS_Unstable
(In addition there are some checks that total has to be greater then free, used has to be greater then buffers + cache etc.)
Looking at free, as a result the following values are the same:
total, used, free and buffers.
The following are changed:
cached and "+/- buffers".
used = MemTotal - MemFree
old:
cached : Cached
+-/buffers-used: used - (Buffers + Cached)
+/-buffers-free: free + (Buffers + Cached)
patched:
cached : Cached + SReclaimable + SwapCached + NFS_Unstable
+/-buffers-used: used - (Buffers + Cached + SReclaimable +
SwapCached + NFS_Unstable)
+/-buffers-free: free + (Buffers + Cached + SReclaimable +
SwapCached + NFS_Unstable)
The same additions are done to top.
htop is unchanged and thus only aligning with older / or un-patched versions of top / free.
In bash (and dash) the various "job status" messages are not displayed from signal handlers, but require an explicit check. This check is performed only before a new prompt is provided, probably not to disturb the user while he/she is typing a new command.
The message is not shown just before the prompt after the kill is displayed probably because the process is not dead yet - this is particularly probable condition since kill is an internal command of the shell, so it's very fast to execute and doesn't need forking.
Doing the same experiment with killall, instead, usually yields the "killed" message immediately, sign that the time/context switches/whatever required to execute an external command cause a delay long enough for the process to be killed before the control returns to the shell.
matteo@teokubuntu:~$ dash
$ sleep 60 &
$ ps
PID TTY TIME CMD
4540 pts/3 00:00:00 bash
4811 pts/3 00:00:00 sh
4812 pts/3 00:00:00 sleep
4813 pts/3 00:00:00 ps
$ kill -9 4812
$
[1] + Killed sleep 60
$ sleep 60 &
$ killall sleep
[1] + Terminated sleep 60
$
dashFirst of all, I had a look at the dash sources, since dash exhibits the same behavior and the code is surely simpler than bash.
As said above, the point seems to be that job status messages are not emitted from a signal handler (which can interrupt the "normal" shell control flow), but they are the consequence of an explicit check (a showjobs(out2, SHOW_CHANGED) call in dash) that is performed only before requesting new input from the user, in the REPL loop.
Thus, if the shell is blocked waiting for user input no such message is emitted.
Now, why doesn't the check performed just after the kill show that the process was actually terminated? As explained above, probably because it's too fast. kill is an internal command of the shell, so it's very fast to execute and doesn't need forking, thus, when immediately after the kill the check is performed, the process is still alive (or, at least, is still being killed).
bashAs expected, bash, being a much more complex shell, was trickier and required some gdb-fu.
The backtrace for when that message is emitted is something like
(gdb) bt
#0 pretty_print_job (job_index=job_index@entry=0, format=format@entry=0, stream=0x7ffff7bd01a0 <_IO_2_1_stderr_>) at jobs.c:1630
#1 0x000000000044030a in notify_of_job_status () at jobs.c:3561
#2 notify_of_job_status () at jobs.c:3461
#3 0x0000000000441e97 in notify_and_cleanup () at jobs.c:2664
#4 0x00000000004205e1 in shell_getc (remove_quoted_newline=1) at /Users/chet/src/bash/src/parse.y:2213
#5 shell_getc (remove_quoted_newline=1) at /Users/chet/src/bash/src/parse.y:2159
#6 0x0000000000423316 in read_token (command=<optimized out>) at /Users/chet/src/bash/src/parse.y:2908
#7 read_token (command=0) at /Users/chet/src/bash/src/parse.y:2859
#8 0x00000000004268e4 in yylex () at /Users/chet/src/bash/src/parse.y:2517
#9 yyparse () at y.tab.c:2014
#10 0x000000000041df6a in parse_command () at eval.c:228
#11 0x000000000041e036 in read_command () at eval.c:272
#12 0x000000000041e27f in reader_loop () at eval.c:137
#13 0x000000000041c6fd in main (argc=1, argv=0x7fffffffdf48, env=0x7fffffffdf58) at shell.c:749
The call that checks for dead jobs & co. is notify_of_job_status (it's more or less the equivalent of showjobs(..., SHOW_CHANGED) in dash); #0-#1 are related to its inner working; 6-8 is the yacc-generated parser code; 10-12 is the REPL loop.
The interesting place here is #4, i.e. from where the notify_and_cleanup call comes. It seems that bash, unlike dash, may check for terminated jobs at each character read from the command line, but here's what I found:
/* If the shell is interatctive, but not currently printing a prompt
(interactive_shell && interactive == 0), we don't want to print
notifies or cleanup the jobs -- we want to defer it until we do
print the next prompt. */
if (interactive_shell == 0 || SHOULD_PROMPT())
{
#if defined (JOB_CONTROL)
/* This can cause a problem when reading a command as the result
of a trap, when the trap is called from flush_child. This call
had better not cause jobs to disappear from the job table in
that case, or we will have big trouble. */
notify_and_cleanup ();
#else /* !JOB_CONTROL */
cleanup_dead_jobs ();
#endif /* !JOB_CONTROL */
}
So, in interactive mode it's intentional to delay the check until a new prompt is provided, probably not to disturb the user entering commands. As for why the check doesn't spot the dead process when displaying the new prompt immediately after the kill, the previous explanation holds (the process is not dead yet).
Best Answer
By default,
htoplists each thread of a process separately, whilepsdoesn't. To turn off the display of threads, pressH, or use the "Setup / Display options" menu, "Hide userland threads". This puts the following line in your~/.htoprcor~/.config/htop/htoprc(you can alternatively put it there manually):(Also
hide_kernel_threads=1, toggled by pressingK, but it's 1 by default.)Another useful option is “Display threads in a different color” in the same menu (
highlight_threads=1in.htoprc), which causes threads to be shown in a different color (green in the default theme).In the first line of the
htopdisplay, there's a line like “Tasks: 377, 842 thr, 161 kthr; 2 running”. This shows the total number of processes, userland threads, kernel threads, and threads in a runnable state. The numbers don't change when you filter the display, but the indications “thr” and “kthr” disappear when you turn off the inclusion of user/kernel threads respectively.When you see multiple processes that have all characteristics in common except the PID and CPU-related fields (NIce value, CPU%, TIME+, ...), it's highly likely that they're threads in the same process.