This is just a partial answer, since your question is fairly broad.
C++ defines an "execution character set" (in fact, two of them, a narrow and a wide one).
When your source file contains something like:
char s[] = "Hello";
Then the numeric byte value of the letters in the string literal are simply looked up according to the execution encoding. (The separate wide execution encoding applies to the numeric value assigned to wide character constants L'a'.)
All this happens as part of the initial reading of the source code file into the compilation process. Once inside, C++ characters are nothing more than bytes, with no attached semantics. (The type name char must be one of the most grievous misnomers in C-derived languages!)
There is a partial exception in C++11, where the literals u8"", u"" and U"" determine the resulting value of the string elements (i.e the resulting values are globally unambiguous and platform-independent), but that does not affect how the input source code is interpreted.
A good compiler should allow you to specify the source code encoding, so even if your friend on an EBCDIC machine sends you her program text, that shouldn't be a problem. GCC offers the following options:
-finput-charset: input character set, i.e. how the source code file is encoded-fexec-charset: execution character set, i.e. how to encode string literals-fwide-exec-charset: wide execution character set, i.e. how to encode wide string literals
GCC uses iconv() for the conversions, so any encoding supported by iconv() can be used for those options.
I wrote previously about some opaque facilities provided by the C++ standard to handle text encodings.
Example: take the above code, char s[] = "Hello";. Suppose the source file is ASCII (i.e. the input encoding is ASCII). Then the compiler reads 99, and interprets it as c, and so on. When it comes to the literal, it reads 72, interprets it as H. Now it stores the byte value of H in the array which is determined by the execution encoding (again 72 if that is ASCII or UTF-8). When you write \xFF, the compiler reads 99 120 70 70, decodes it as \xFF, and writes 255 into the array.
Inside a terminal emulator that supports UTF-8, you can use the luit command to run a subshell (or other program) in a different locale. The locale setting that indicates character sets is LC_CTYPE.
LC_CTYPE=ru_RU.KOI8-R luit ls # run one command
LC_CTYPE=ru_RU.KOI8-R luit # start a shell (type Ctrl+D or exit to return to the parent shell)
If you have a whole tree of files in a different encoding, I recommend (if possible) mounting it through convmvfs.
mkdir ~/net/ivan@example.com.KOI8-R ~/net/ivan@example.com.UTF-8
sshfs ivan@example.com: ~/net/ivan@example.com.KOI8-R
convmvfs -o srcdir=~/net/ivan@example.com.KOI8-R,icharset=KOI8-R,ocharset=UTF-8 ~/net/ivan@example.com.UTF-8
ls ~/net/ivan@example.com.UTF-8
Best Answer
If all the files you're searching in have the same encoding:
or in bash or zsh
Or start a child shell in the desired encoding:
Luit (shipped with XFree86 and X.org) runs the program specified on its command line in the locale specified by the
LC_CTYPEsetting, assuming an UTF-8 terminal. So the command runs in the desired locale, and Luit translates its terminal output to UTF-8.Another approach, if you have a directory tree with a lot of files in a different encoding, is to mount a view of that directory tree under a your prefered encoding. I think the fuseflt filesystem can do this (untested).
where the configuration file
iconv-koi8r-utf8.confcontains