Bash – UTF-8 characters are not displayed correctly in Debian

bashcharacter encodingdebianterminal

Short description of my problem:
I ran into an issue lately where I am unable to make bash/nano/irssi/etc display "special" UTF-8 characters like the german umlauts (äüö), the euro sign (€) and some other UTF-8 characters like ß, §, etc.

What I already tried:

dpkg-reconfigure locales and only generated en_US.UTF-8
setting LC_ALL, LANG and LANGUAGE to en_US.UTF-8 within the .bashrc for both my user and root
re-installed locales and libx11-data (which seems to contain all the language data)

Of course I re-logged in via ssh after all these changes and even tried restarting the server even though I know it doesn't solve any problem in Linux in 99,9875% of all cases.

Information on my system:
OS: Debian stretch -> Linux 3.2.0-4-amd64 #1 SMP Debian 3.2.63-2 x86_64 GNU/Linux
locales: v.2.22-7

Output of locale:

LANG=en_US.UTF-8
LANGUAGE=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=en_US.UTF-8

When typing for example ä into the console and press enter I get -bash: $'\344': command not found.
Honestly I am out of ideas, can anyone help me out with this?

Best Answer

You've told bash and other applications that your terminal uses the UTF-8 encoding. That's good only if your terminal actually does use UTF-8. Bash doesn't get to decide that the terminal encoding is, the terminal gets to decide.

If you want to use UTF-8, configure your terminal to use UTF-8. Since you're using SSH, you need to configure whatever terminal you're running the SSH client in to use UTF-8. That's the default on most modern systems, but apparently yours isn't set up this way.

You should avoid setting LC_CTYPE explicitly in a terminal: ideally the terminal will set this. However this doesn't always work, especially over SSH (on many systems, the SSH server forbids the client from setting LC_CTYPE).

If you need to set the environment variable, the right place would be .profile, not .bashrc.

Related Solutions

Corrupted gnu-screen session not displaying UTF-8 correctly

I had the same issue when running:

git clone https://github.com/jwiegley/git-scripts.git
cd git-scripts
perl git-forest

I used this as my test. Basically, you should get nice lines is you have utf8 set properly. If not you will get ugly boxes or characters.

The solution is that you need to set LC_ALL to en_US.UTF-8 BEFORE you start a new screen session. I tired doing it after creating the screen session and had no luck.

Here are the step i followed to get this going:

1) Run locale to view the current setup. I got this (which explains why i was having issues:

LANG=en_US.UTF-8
LC_CTYPE="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_COLLATE="C"
LC_MONETARY="C"
LC_MESSAGES="C"
LC_PAPER="C"
LC_NAME="C"
LC_ADDRESS="C"
LC_TELEPHONE="C"
LC_MEASUREMENT="C"
LC_IDENTIFICATION="C"
LC_ALL=C

2) Before creating a new screen session, you need to redefine LC_ALL to en_US.UTF-8

if you are using csh shell

setenv LC_ALL en_US.UTF-8

if you are using bash shell

export LC_ALL="en_US.UTF-8"

3) Verify LC_ALL was set properly, by running locale again:

LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=en_US.UTF-8

4) Now run a new screen session and run the git-forest test and you should see nice lines

UTF-8 – Can Not Use `cut -c` with UTF-8 Characters?

You haven't said which cut you're using, but since you've mentioned the GNU long option --characters I'll assume it's that one. In that case, note this passage from info coreutils 'cut invocation':

‘-c character-list’
‘--characters=character-list’
Select for printing only the characters in positions listed in character-list. The same as -b for now, but internationalization will change that.

(emphasis added)

For the moment, GNU cut always works in terms of single-byte "characters", so the behaviour you see is expected.

Supporting both the -b and -c options is required by POSIX — they weren't added to GNU cut because it had multi-byte support and they worked properly, but to avoid giving errors on POSIX-compliant input. The same -c has been done in some other cut implementations, although not FreeBSD's and OS X's at least.

This is the historic behaviour of -c. -b was newly added to take over the byte role so that -c can work with multi-byte characters. Maybe in a few years it will work as desired consistently, although progress hasn't exactly been quick (it's been over a decade already). GNU cut doesn't even implement the -n option yet, even though it is orthogonal and intended to help the transition. There are potential compatibility problems with old scripts, which may be a concern, although I don't know definitively what the reason is.

Best Answer

Related Solutions

Corrupted gnu-screen session not displaying UTF-8 correctly

UTF-8 – Can Not Use `cut -c` with UTF-8 Characters?

Related Question