Bash displays international characters as escape sequences

bashinputrcreadline

When I press certain keys (e.g. German umlauts) in the bash command line, I get escape sequences. For example, ß gives me \303. These escape sequences are treated as single characters, so one backspace deletes the whole sequence. The character is just displayed wrong on the command line, but it is interpreted correctly, for example typing echo ß gives:

$ echo \303
ß

I guess it has problems with non 7-bit-ASCII characters. In every other place, these work fine however. I can use them in vim or display them with cat, and also unicode characters work fine.

How can I make bash display these characters correctly?

For the record,

TERM=xterm-256color
LANG=en_US.UTF-8

and from my .inputrc:

set input-meta on      # Enable Meta input with eighth bit set
set meta-flag on       # Synonym for the above
set convert-meta off   # Do NOT strip the eighth bit
set output-meta on     # Enable Meta output with eighth bit set

output of stty -a

speed 38400 baud; rows 24; columns 80; line = 0;
intr = ^C; quit = ^\; erase = ^?; kill = ^U; eof = ^D; eol = M-^?; eol2 = M-^?;
swtch = <undef>; start = ^Q; stop = ^S; susp = ^Z; rprnt = ^R; werase = ^W;
lnext = ^V; flush = ^O; min = 1; time = 0;
-parenb -parodd cs8 -hupcl -cstopb cread -clocal -crtscts -cdtrdsr
-ignbrk -brkint -ignpar -parmrk -inpck -istrip -inlcr -igncr icrnl ixon -ixoff
-iuclc ixany imaxbel -iutf8
opost -olcuc -ocrnl onlcr -onocr -onlret -ofill -ofdel nl0 cr0 tab0 bs0 vt0 ff0
isig icanon iexten echo echoe echok -echonl -noflsh -xcase -tostop -echoprt
echoctl echoke

This Stackoverflow question shows the same symptom, but the posted solution doesn't work: https://stackoverflow.com/questions/13094248/how-do-i-get-accented-letters-to-actually-work-on-bash

I'm also confident that my terminal is properly configured, as it works fine with local bash, but only the bash on a certain remote system (via ssh) has this problem.

Best Answer

You can thank someone named Lino Miguel Martins Tinoco from 2004 for this one.

The GNU Readline documentation for .inputrc does not allow in-line comments. Both it and the GNU Bourne Again shell manual say:

Lines beginning with a `#' are comments.

The line

set output-meta on     # Enable Meta output with eighth bit set

is not a line beginning with #. It's a line with a # in the middle. As Lino Miguel Martins Tinoco found, this results in the output-meta option being off, not on, as evident in the output of bind -V when xe ran it:

output-meta is set to `off'

.inputrc is not shell script. As said in the Linux From Scratch tutorial

Note that comments cannot be on the same line as commands.

Related Solutions

Bash – Delete whole argument in current bash command-line

Bash has readline commands that aren't bound by default. You can find them at reference: http://www.gnu.org/software/bash/manual/html_node/Bindable-Readline-Commands.html#Bindable-Readline-Commands

The command you are looking for is called shell-backward-kill-word. You have to select a shortcut first. Let's use Ctrlp, since it's "previous command" - same as up arrow.

bind '"\C-p": shell-backward-kill-word'

Bash Terminal – Where Are Terminal Color Escape Sequences Defined?

There are several aspects at play in what you're asking.

First, bash doesn't define colors. In fact bash has absolutely no idea that colors even exist. All it knows is that you told it to output the characters \033[0;36m. Your terminal emulator (xterm, gnome-terminal, whatever) receives these characters and understands "I need to start outputting in cyan".

Thus it is your terminal emulator that understands colors. Your terminal emulator understands that \033[0;36m is cyan, but another terminal emulator might use an entirely different set of characters for cyan (though no sane terminal emulator would flaunt the standard and do this). This is the reason for tput. When you run tput setaf 6, tput is going to look up your terminal's escape codes for the color 6 (cyan), and output that escape code.
^{(see this question for more info on tput setaf codes)}

Now back to bash. As you may have noticed, when I've been referring to the cyan color, I've been using \033[0;36m, not \[\033[0;36m\]. The square brackets have been missing. The purpose of the square brackets is that when using escape codes (colors) in the prompt, bash has to know which characters are non-printing (zero-width, don't actually show anything). Thus you enclose non-printing characters in \[ \]. If you remove these characters, everything might appear to work just fine at first, but you'll start running into all sorts of weirdness when your command exceeds the terminal width. This is because when typing, bash has to know when the command should wrap to the next line. To do this, it calculates the width of the prompt, and then the width of how much you've typed.

Another note, about tput. CYAN="\[\033[0;36m\]" is not the same thing as CYAN="$(tput setaf 6)". As we just discussed, the square brackets are relevant to bash, and tput is only going to output the terminal escape codes.

Since square brackets are usually only relevant in the prompt, if you're using colors in the output of a script or something, you should not use them. Meaning that if you're going to be using colors for more than prompts, you need to define multiple variables. One with square brackets for using in the prompt, and one without for everything else. Though you could just manually add the square brackets every time you reference a color in the prompt.

So long story short, you probably want to define something like:

local CYAN="$(tput setaf 6)" # OR CYAN="\033[0;36m"
local LIGHT_CYAN="$CYAN$(tput bold)" # OR LIGHT_CYAN="\033[1;36m"
local PROMPT_CYAN="\[$CYAN\]"
local PROMPT_LIGHT_CYAN="\[$LIGHT_CYAN\]"

Best Answer

Related Solutions

Bash – Delete whole argument in current bash command-line

Bash Terminal – Where Are Terminal Color Escape Sequences Defined?

Related Question