Character Encoding – Fix Umlaut Issues in Terminal and File Access

character encodingspecial characters

On my laptop I have Linux Mint 18.3 installed and right now I am experiencing the following problem:
As a German I happen to have some files (or directories) that contain umlauts in their names (which in itself is not the problem). If I now try to open those files (e.g. PDFs) that either contain an umlaut or are located in a (sub-)directory whose name contains an umlaut it won't open.

For example I have a directory named Übungsblätter and somewhere in that dir I have a few PDF files. If I try to open one of those files from the file manager Okular will tell me it can't open .../ÃbungsblÃ¤tter/... which shows me that it messed up the path by replacing the umlauts with some weird characters.

However if I have Okular opened and use its menu to open the respective file everything works like charm so the problem does only occur when trying to open the files from my file manager (Dolphin).

Furthermore I experienced that I can't type in umlauts in my terminal. I found this question dealing with that specific problem and I hoped that would fix the file-issue as well. However that was not the case although I am now able to type in umlauts in the terminal (though when typing in an umlaut it will insert two characters. For example when typing in ö it will insert the two characters \udcc3\udcb6 (found out by typing in ö and hitting return afterwards). It does get displayed as a single ö though but I have to explicitly delete two characters in order for the command line to be empty again.

Here's the output of locale:

locale: Cannot set LC_ALL to default locale: No such file or directory
LANG=de_DE.UTF-8
LANGUAGE=en_US
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC=de_DE.UTF-8
LC_TIME=nds_DE.UTF-8
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY=de_DE.UTF-8
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER=de_DE.UTF-8
LC_NAME=de_DE.UTF-8
LC_ADDRESS=de_DE.UTF-8
LC_TELEPHONE=de_DE.UTF-8
LC_MEASUREMENT=de_DE.UTF-8
LC_IDENTIFICATION=de_DE.UTF-8
LC_ALL=

EDIT:
I just found out that when opening the respective PDF from the terminal it will also error in a similar way as when opening from the file manager but this time it says it can't open .../bungsbltter/... (It completely removes the umlauts)

EDIT2:
Output of locale -a:

C
C.UTF-8
de_DE.utf8
en_AG
en_AG.utf8
en_AU.utf8
en_BW.utf8
en_CA.utf8
en_DK.utf8
en_GB.utf8
en_HK.utf8
en_IE.utf8
en_IN
en_IN.utf8
en_NG
en_NG.utf8
en_NZ.utf8
en_PH.utf8
en_SG.utf8
en_US.utf8
en_ZA.utf8
en_ZM
en_ZM.utf8
en_ZW.utf8
POSIX

EDIT 3:
/etc/default/locale:

LANGUAGE=en_GB.utf8
LANG=en_US.UTF-8

Best Answer

I see some confusion in the locale settings.

One point is: A mix of de_DE, en_US and en_GB.

The other point is: some locales are declared with .utf8, other with .UTF-8. Though it should mean the same, it may cause trouble.

The error message Cannot set LC_ALL to default locale: No such file or directory is probably because of missing en_US.UTF-8 in locale -a (that has en_US.utf8 instead), but it is declared in /etc/default/locale.

I am a bit confused to see LANG=de_DE.UTF-8 in the output of locale although it is not defined in /etc/default/locale and there is no /etc/locale.conf. Maybe a setting from the desktop environment that overwrites system settings? Or in ~/.bashrc? Also check whether you have a ~/.config/locale.conf. More possible locations where environment variables can be set are listed in Arch wiki. Find out where LANG=de_DE.UTF-8 is set and remove that.

I think the best way is to set everything to german locales. Your locale -a shows installed locales on your system. One of them is de_DE.utf8.

The core setting is in /etc/default/locale. You can edit that manually and I think its content should be just LANG=de_DE.utf8 and nothing else. You can edit the file yourself, or use a tool for this:

update-locale --reset LANG=de_DE.utf8

To keep the system in english, but have umlauts too, it may work to set the locale to en_US.utf8 that is listed in locale -a. utf8 should provide the umlauts, even though the language does not use them:

update-locale --reset LANG=en_US.utf8

Speak a prayer, reboot and check if all is right. If console works fine, but there is still something wrong in desktop environment, check your desktop settings GUI for language entries and set is to de_DE.utf8, too.

Related Solutions

Shell – Bulk Rename Files with Special Characters

I guess you see this � invalid character because the name contains a byte sequence that isn't valid UTF-8. File names on typical unix filesystems (including yours) are byte strings, and it's up to applications to decide on what encoding to use. Nowadays, there is a trend to use UTF-8, but it's not universal, especially in locales that could never live with plain ASCII and have been using other encodings since before UTF-8 even existed.

Try LC_CTYPE=en_US.iso88591 ls to see if the file name makes sense in ISO-8859-1 (latin-1). If it doesn't, try other locales. Note that only the LC_CTYPE locale setting matters here.

In a UTF-8 locale, the following command will show you all files whose name is not valid UTF-8:

grep-invalid-utf8 () {
  perl -l -ne '/^([\000-\177]|[\300-\337][\200-\277]|[\340-\357][\200-\277]{2}|[\360-\367][\200-\277]{3}|[\370-\373][\200-\277]{4}|[\374-\375][\200-\277]{5})*$/ or print'
}
find | grep-invalid-utf8

You can check if they make more sense in another locale with recode or iconv:

find | grep-invalid-utf8 | recode latin1..utf8
find | grep-invalid-utf8 | iconv -f latin1 -t utf8

Once you've determined that a bunch of file names are in a certain encoding (e.g. latin1), one way to rename them is

find | grep-invalid-utf8 |
rename 'BEGIN {binmode STDIN, ":encoding(latin1)"; use Encode;}
        $_=encode("utf8", $_)'

This uses the perl rename command available on Debian and Ubuntu. You can pass it -n to show what it would be doing without actually renaming the files.

Linux Filesystems – Questions About Character Encoding

I have reformulated your questions a bit, for reasons that should appear evident when you read them in sequence.

1. Is it possible to config linux filesystem use fixed character encoding to store file names regardless of LANG/LC_ALL environment?

No, this is not possible: as you mention in your question, a UNIX file name is just a sequence of bytes; the kernel knows nothing about the encoding, which entirely a user-space (i.e., application-level) concept.

In other words, the kernel knows nothing about LANG/LC_*, so it cannot translate.

2. Is it possible to let different file names refer to same file?

You can have multiple directory entries referring to the same file; you can make that through hard links or symbolic links.

Be aware, however, that the file names that are not valid in the current encoding (e.g., your GBK character string when you're working in a UTF-8 locale) will display badly, if at all.

3. Is it possible to patch the kernel to translate character encoding between file-system and current environment?

You cannot patch the kernel to do this (see 1.), but you could -in theory- patch the C library (e.g., glibc) to perform this translation, and always convert file names to UTF-8 when it calls the kernel, and convert them back to the current encoding when it reads a file name from the kernel.

A simpler approach could be to write an overlay filesystem with FUSE, that just redirects any filesystem request to another location after converting the file name to/from UTF-8. Ideally you could mount this filesystem in ~/trans, and when an access is made to ~/trans/a/GBK/encoded/path then the FUSE filesystem really accesses /a/UTF-8/encoded/path.

However, the problem with these approaches is: what do you do with files that already exist on your filesystem and are not UTF-8 encoded? You cannot just simply pass them untranslated, because then you don't know how to convert them; you cannot mangle them by translating invalid character sequences to ? because that could create conflicts...