Addendum: Apparently ncurses in 10.6.3 is broken
An answer on SO links to a blog entry that describes some other investigations into the problem. One reported workaround is to copy the ncurses library from 10.6.2, but this may have unintended side effects.
ESC is usually the first character of the “control sequence” sent by a terminal (or terminal emulation program like Terminal) for special keys (like the arrow keys, F1, Home, Page Up, etc.).
ESC also serves as a ‘go back’ key in htop (usually only after a delay, so that slow connections have enough time to send a full control sequence that happens to start with ESC).
Please update your question with the answers to the following questions:
What is TERM?
Maybe your TERM environment variable has an odd value? What does echo $TERM report in a window where htop behaves as you have described? TERM should probably be xterm-color, or something similar.
Where did you get htop? MacPorts? Fink? other?
Apps compiled with the system-supplied ncurses will use entries from /usr/share/terminfo/.
MacPorts usually uses /opt/local/ as its prefix, so applications that use ncurses from MacPorts will use entries from /opt/local/share/terminfo/.
Fink usually uses /sw/ as its prefix, so applications that use ncurses from Fink will use entries from /sw/share/terminfo/.
Does the TERMINFO environment variable have a value? If so, ncurses (whatever the source) will look in the directory specified therein.
It could be that you have a partially corrupted terminfo definition for the terminal specified in TERM (‘partially corrupted’ because the output control sequences must be OK or you would see obvious display anomalies).
What are some examples of programs that are not affected?
If you have ncurses-based programs that handle the arrow keys OK (maybe using a different installation of ncurses), you might try using the other terminfo entries with htop and vice versa (by setting TERMINFO to the location of the entries from the other ncurses installation).
What codes does your terminal actually send?
Use cat -v to examine which codes are sent when you use the arrow keys. The following is what I see when I press Up, Down, Right, Left, Enter, Control-D after starting cat -v:
% cat -v
^[[A^[[B^[[D^[[C
^[[A^[[B^[[D^[[C
%
The first set of codes is an echo produced by the tty device. The second set is produced by cat and may be different from the first sent, depending on the exact codes present.
These code can be configured in Terminal, but it is usually not necessary since there are special mode that change the codes (sometimes Up sends ESC [ A, sometimes it sends ESC O A.
Ultimately the problem is that htop is not recognizing whatever codes are being sent as the codes for the arrow keys, so it might be useful to know codes are actually being sent.
Addendum: htop from Homebrew; less is OK
Against which ncurses is htop linked?
It seems that the htop ‘formula’ was changed in late 2009 to not depend on the Homebrew version of ncurses. Presumably this means that versions of htop that were built from this new formula will use the system ncurses. So, which ncurses is your htop binary using?
ls -l "$(which htop)"
otool -L "$(which htop)"
If your htop is older than 2009-12-18 or it is linked against your Homebrew ncurses (probably /usr/local/lib/libncurses.5.dylib), then you might consider rebuilding the htop formula (brew remove htop; brew install htop?). Alternatively, if your existing htop binary is using a Homebrew version of ncurses (or the rebuilt version also still uses the Homebrew ncurses), then you might try rebuilding the ncurses formula.
You mention that less works OK. It seems that there is no Homebrew formula for less, so you are probably using the system-supplied less. It is worth noting which ncurses your less is using for comparison to htop.
otool -L "$(which less)"
Addendum: htop and less use system ncurses
If the problem is being caused by some bit of per-user configuration it should go away if you create a brand new user and try it while logged in as them. If the problem persists under the brand new user you know that it is a system-wide problem (or the problem is in the basic, initial configuration that is given to all new users!). Once you have finished testing, just delete the user to avoid cluttering the login chooser, etc.
Terminal and other VT100-style terminals and terminal emulators have an “Application Cursor Key” mode where the cursor keys send different control sequences to the application.
In “application” mode, the terminal sends ^[OA^[OB^[OC^[OD instead of ^[[A^[[B^[[C^[[D (For Up, Down, Right, Left, both respectively).
The non-“application” mode sequences that the terminal sends are identical to the codes that the terminal would expect to receive as cursor control codes.
It might be possible that your Terminal is not sending the correct application cursor key codes.
This seems unlikely since you say that less works (it also uses “application” mode).
Presumably you mean that the arrow keys work to navigate in less. Due to ‘muscle memory’, I almost always use the vi-style keys (kjlh) to navigate in less.
What “application cursor keys” codes is Terminal sending?
You can check that Terminal is sending the expected “application cursor key” sequences with something like this:
sh -c "$(cat <<\EOF
noecho_appmode() {
stty -echo
printf '\033[?1h'
}
modes="$(stty -g)"
restore_echo_and_appmode() {
stty "$modes"
printf '\033[?1l'
}
printf '\nType <Up> <Down> <Right> <Left> <Control-D> <Control-D>\n'
printf '(no output until after the first <Control-D>, please type "blindly")\n\t'
noecho_appmode ; trap 'restore_echo_and_appmode' 0
cat -v
restore_echo_and_appmode ; trap '' 0
printf '\nExpected:\n\t'
printf 'kcu%c1\n' u d f b | /usr/bin/tput -S | cat -v
printf '\n\n'
EOF
)"
The output labeled “Expected:” is based on the system terminfo entries for your TERM, so it should be what ncurses-based programs expect to see.
If the actual codes differ from the expected codes, then you should investigate the configuration of Terminal to find where the incorrect codes have been configured.
If this does not indicate any problem (the codes from Terminal match the expected codes), I would run a non-setuid copy of htop under ktrace (or maybe a bit of DTrace?) to see the in situ I/O. The nature of the I/O might give someone a clue as the cause of the problem. The full output from kdump will be very large due to reporting the details of loading of shared libraries and supporting data files, but even with that stuff stripped out, I still get nearly 2000 lines and 100kB of output from kdump. This is probably too much to post here.
Best Answer
In Lion, the default terminal type is different, so you should use this line in your
.screenrcinstead: