What charset encoding is used for filenames and paths on Linux

character encodingfilenameslocale

Does it depend on what file system I use? For example, ext2/ext3/ext4 but also what happens when I insert one of those "joliet" CD-ROMs with ISO 9660? I've heard that POSIX contains some sort of spec for the charset encoding of filenames?

Essentially, what I wonder is if I got a UTF-8 encoded filename, what processing/coversion do I need to do before I pass it to a file I/O API in Linux?

Best Answer

As noted by others, there isn't really an answer to this: filenames and paths do not have an encoding; the OS only deals with sequence of bytes. Individual applications may choose to interpret them as being encoded in some way, but this varies.

Specifically, Glib (used by Gtk+ apps) assumes that all file names are UTF-8 encoded, regardless of the user's locale. This may be overridden with the environment variables G_FILENAME_ENCODING and G_BROKEN_FILENAMES.

On the other hand, Qt defaults to assuming that all file names are encoded in the current user's locale. An individual application may choose to override this assumption, though I do not know of any that do, and there is no external override switch.

Modern Linux distributions are set up such that all users are using UTF-8 locales and paths on foreign filesystem mounts are translated to UTF-8, so this difference in strategies generally has no effect. However, if you really want to be safe, you cannot assume any structure about filenames beyond "NUL-terminated, '/'-delimited sequence of bytes".

(Also note: locale may vary by process. Two different processes run by the same user may be in different locales simply by having different environment variables set.)

Related Solutions

Ssh and character encoding

Running the locale command will give you information about your locale settings; the character encoding is given by the LC_CTYPE setting.

Under Ubuntu, the default locale settings are given in /etc/default/locale. You can change the character encoding by setting LC_CTYPE in your ~/.profile on the VPS, e.g.

export LC_CTYPE=en_US.UTF-8

You'll have to make sure that the en_US.UTF-8 locale is available. Ubuntu only generates locale data for requested locales. All English locales should be available if you have the package language-pack-en-base installed. You can manually request their generation with

sudo locale-gen en

You can also add entries to /var/lib/locales/supported.d/local to make sure a particular locale is installed (e.g., add the line en_US.UTF-8 UTF-8).

SSH – Working with Filenames in Different Encoding

Inside a terminal emulator that supports UTF-8, you can use the luit command to run a subshell (or other program) in a different locale. The locale setting that indicates character sets is LC_CTYPE.

LC_CTYPE=ru_RU.KOI8-R luit ls   # run one command
LC_CTYPE=ru_RU.KOI8-R luit      # start a shell (type Ctrl+D or exit to return to the parent shell)

If you have a whole tree of files in a different encoding, I recommend (if possible) mounting it through convmvfs.

mkdir ~/net/ivan@example.com.KOI8-R ~/net/ivan@example.com.UTF-8
sshfs ivan@example.com: ~/net/ivan@example.com.KOI8-R
convmvfs -o srcdir=~/net/ivan@example.com.KOI8-R,icharset=KOI8-R,ocharset=UTF-8 ~/net/ivan@example.com.UTF-8
ls ~/net/ivan@example.com.UTF-8

Best Answer

Related Solutions

Ssh and character encoding

SSH – Working with Filenames in Different Encoding

Related Question