Bash – Display Control Characters Differently in the Shell

bashcolorsdisplayreadlinespecial characters

When you type control characters in the shell they get displayed using what is called "caret notation". Escape for example gets written as ^[ in caret notation.

I like to customize my bash shell to make it look cool. I have for example changed my PS1 and PS2 to become colorized. I now want control characters to get a unique appearance as well to make them more distinguishable from regular characters.

$ # Here I type CTRL-C to abort the command.
$ blahblah^C
          ^^ I want these two characters to be displayed differently

Is there a way to make my shell highlight control characters differently?

Is it possible to make it display them in a bold font or maybe make them appear in different colors from regular text?

I am using bash shell here but I did not tag the question with bash because maybe there is a solution that applies to many different shells.

Note that I do not know at what level highlighting of control characters takes place. I first thought it was in the shell itself. Now I have heard that it is readline that controls how control characters are in shells like bash. So the question is now tagged with readline and I am still looking for answers.

Best Answer

When you press Ctrl+X, your terminal emulator writes the byte 0x18 to the master side of the pseudo-terminal pair.

What happens next depends on how the tty line discipline (a software module in the kernel that sits in between the master side (under control of the emulator) and the slave side (which applications running in the terminal interact with)) is configured.

A command to configure that tty line discipline is the stty command.

When running a dumb application like cat that is not aware of and doesn't care whether its stdin is a terminal or not, the terminal is in a default canonical mode where the tty line discipline implements a crude line editor.

Some interactive applications that need more than that crude line editor typically change those settings on start-up and restore them on leaving. Modern shells, at their prompt are examples of such applications. They implement their own more advanced line editor.

Typically, while you enter a command line, the shell puts the tty line discipline in that mode, and when you press enter to run the current command, the shell restores the normal tty mode (as was in effect before issuing the prompt).

If you run the stty -a command, you'll see the current settings in use for the dumb applications. You're likely to see the icanon, echo and echoctl settings being enabled.

What that means is that:

icanon: that crude line editor is enabled.
echo: characters you type (that the terminal emulator writes to the master side) are echoed back (made available for reading by the terminal emulator).
echoctl: instead of being echoed asis, the control characters are echoed as ^X.

So, let's say you type A B Backspace-aka-Ctrl+H/? C Ctrl+X Backspace Return.

Your terminal emulator will send: AB\bC\x18\b\r. The line discipline will echo back: AB\b \bC^X\b \b\b \b\r\n, and an application that reads the input from the slave side (/dev/pts/x) will read AC\n.

All the application sees is AC\n, and only when your press Enter so it can't have any control on the output for ^X there.

You'll notice that for echo, the first ^H (^? with some terminals, see the erase setting) resulted in \b \b being sent back to the terminal. That's the sequence to move the cursor back, overwrite with space, move cursor back again, while the second ^H resulted in \b \b\b \b to erase those two ^ and X characters.

The ^X (0x18) itself was being translated to ^ and X for output. Like B, it didn't make it to the application, as we deleted it with Backspace.

\r (aka ^M) was translated to \r\n (^M^J) for echo, and \n (^J) for the application.

So, what are our options for those dumb applications:

disable echo (stty -echo). That effectively changes the way control characters are echoed, by... not echoing anything. Not really a solution.
disable echoctl. That changes the way control characters (other than ^H, ^M... and all the other ones used by the line editor) are echoed. They are then echoed as-is. That is for instance, the ESC character is send as the \e (^[/0x1b) byte (which is recognised as the start of an escape sequence by the terminal), ^G you send a \a (a BEL, making your terminal beep)... Not an option.
disable the crude line editor (stty -icanon). Not really an option as the crude applications would become a lot less usable.
edit the kernel code to change the behaviour of the tty line discipline so the echo of a control character sends \e[7m^X\e[m instead of just ^X (here \e[7m usually enables reverse video in most terminals).

An option could be to use a wrapper like rlwrap that is a dirty hack to add a fancy line editor to dumb applications. That wrapper in effect tries to replace simple read()s from the terminal device to calls to readline line editor (which do change the mode of the tty line discipline).

Going even further, you could even try solutions like this one that hijacks all input from the terminal to go through zsh's line editor (which happens to highlight ^Xs in reverse video) relying on GNU screen's :exec feature.

Now for applications that do implement their own line editor, it's up to them to decide how the echo is done. bash uses readline for that which doesn't have any support for customizing how control characters are echoed.

For zsh, see:

info --index-search='highlighting, special characters' zsh

zsh does highlight non-printable characters by default. You can customize the highlighting with for instance:

zle_highlight=(special:fg=white,bg=red)

For white on red highlighting for those special characters.

The text representation of those characters is not customizable though.

In a UTF-8 locale, 0x18 will be rendered as ^X, \u378, \U7fffffff (two unassigned unicode code points) as <0378>, <7FFFFFFF>, \u200b (a not-really printable unicode character) as <200B>.

\x80 in a iso8859-1 locale would be rendered as ^�... etc.

0.5a - Cyan Tumbleweed Plains

The earlier implementations provided with the question rely on a sequence of commands which uses tr and a set of single byte characters. As was explained in this Q&A, the utility cannot process multibyte characters such as Unicode. But leveraging those characters was quite essential in achieving the desired effect. A "solution" was provided which allows to mix the single byte and multi byte chars in a single stream for rendering. The solution developed there is presented and customized here:

Z1=$(echo -en '\xe2\x97\x98') #◘ 1
Z2=$(echo -en '\xe2\x95\x9a') #╚ 2
Z3=$(echo -en '\xe2\x95\x9c') #╜ 3
Z4=$(echo -en '\xe2\x95\x9d') #╝ 4
Z5=$(echo -en '\xe2\x95\x9e') #╞ 5
Z6=$(echo -en '\xe2\x95\x9f') #╟ 6
Z7=$(echo -en '\xe2\x96\x91') #░ 7
Z8=$(echo -en '\xe2\x96\x92') #▒ 8
Z9=$(echo -en '\xe2\x96\x93') #▓ 9
N1=$(echo -en '\xe2\x94\x80') #─ a
N2=$(echo -en '\xe2\x95\x92') #╒ b
N3=$(echo -en '\xe2\x95\x97') #╗ c
N4=$(echo -en '\xe2\x96\xb6') #▶d
N5=$(echo -en '\xe2\x94\xbc') #┼ e
N6=$(echo -en '\xe2\x94\xa4') #┤ f
N7=$(echo -en '\xe2\x95\xa1') #╡ g
Z11="$(tr -dc '123456789a' < /dev/urandom | head -c 1)" //Z11 to Z13 not
Z12="$(tr -dc '123456789a' < /dev/urandom | head -c 1)" // used here (see
Z13="$(tr -dc '123456789a' < /dev/urandom | head -c 1)" //link)

echo -en $(tr -dcs ' ;",15bdef' ' ' < /dev/urandom | head -c $(echo -en "$[$(tput cols) * $(tput lines)]") | sed -e "s/1/$(echo -en "\033[0;36m$Z1\033[0m")/g" -e "s/5/$(echo -en "\033[0;32m$Z5\033[0m")/g" -e "s/b/$(echo -en "\033[1;36m$N2\033[0m")/g" -e "s/d/$(echo -en "\033[1;36m$N4\033[0m")/g" -e "s/e/$(echo -en "\033[0;32m$N5\033[1;32m")/g" -e "s/f/$(echo -en "\033[0;36m$N7\033[1;32m")/g"); tput cup 1
                 ^set^+^chars^ to implement from pool - here 1,5,b,d,e,f... so_________________________^add the appropriate sed subprocessing units for implemented chars i.e. first one we replace "1" with the value of $Z1 and apply color at the same time, then all the chars move down the pipe to all required blocks - we selected to implement 6 chars here so we have 6 sed blocks. 

[N.B. To remove the blank space from the pattern, remove it from both sets: tr -dcs ';",15bdef' '']

PS1="\[\033[1;36m\] $(echo -en '\xe2\x96\x91')$(echo -en '\xe2\x96\x92')$(echo -en '\xe2\x96\x93')[\t]$(echo -en '\xe2\x96\x93')$(echo -en '\xe2\x96\x92')$(echo -en '\xe2\x96\x91') \[\033[7;36m\]$(echo -en '\xe2\x97\x98')$(echo -en '\xe2\x94\xbc')$(echo -en '\xe2\x94\x80')\W$(echo -en '\xe2\x94\x80')\[\033[0;36m\]$(echo -en '\xe2\x94\x80')$(echo -en '\xe2\x94\x80')$(echo -en '\xe2\x94\x80')@$(echo -en '\xe2\x96\xb6')\[\033[0;36m\]"

PROMPT_COMMAND="echo -en '\033[0;36m$(tr -dc '=' < /dev/urandom | head -c $(tput cols))\n\033[01;46m$(tr -dc '~' < /dev/urandom | head -c $(tput cols))\033[0;36m$(tr -dc '=' < /dev/urandom | head -c $(tput cols))'$(tput cuu 2)"

This implementation no longer renders per line but rather prints the whole sequence in one shot at the end of the sed processing. This appears only at login once or generally when bash is launched. Here is one such random pattern at launch (we can see two shades of green and two shades of cyan):

enter image description here

The screens show the result in the standard linux terminal and it works in xterm too. I've used some of those new pattern chars in the PS1 prompt whereas the PROMPT_COMMAND just takes care of the active line and its 2 line padding which use 1 byte chars.

The pattern also matches well my current distribution which calls archbey in .bashrc.:

enter image description here

It's in for Christmas! Cheers people :)

Bash – Non-printing escape sequence: when

The "non-printing escape sequence" is needed when using non-printing characters in $PS1 because bash needs to know the position of the cursor so that the screen can be updated correctly when you edit the command line. Bash does that by counting the number of characters in the $PS1 prompt and then that's the column number the cursor is in.

However, if you place non-printing sequences in $PS1, then that count is wrong and the line can be messed up if you edit the command line. Hence the introduction of the \[ and \] markers to indicate that the enclosed bytes should not be counted.

Best Answer

Related Solutions

Customizing Terminal Background with ASCII Patterns – How to Guide

0.5a - Cyan Tumbleweed Plains

Bash – Non-printing escape sequence: when

Related Question