This is just a partial answer, since your question is fairly broad.
C++ defines an "execution character set" (in fact, two of them, a narrow and a wide one).
When your source file contains something like:
char s[] = "Hello";
Then the numeric byte value of the letters in the string literal are simply looked up according to the execution encoding. (The separate wide execution encoding applies to the numeric value assigned to wide character constants L'a'
.)
All this happens as part of the initial reading of the source code file into the compilation process. Once inside, C++ characters are nothing more than bytes, with no attached semantics. (The type name char
must be one of the most grievous misnomers in C-derived languages!)
There is a partial exception in C++11, where the literals u8""
, u""
and U""
determine the resulting value of the string elements (i.e the resulting values are globally unambiguous and platform-independent), but that does not affect how the input source code is interpreted.
A good compiler should allow you to specify the source code encoding, so even if your friend on an EBCDIC machine sends you her program text, that shouldn't be a problem. GCC offers the following options:
-finput-charset
: input character set, i.e. how the source code file is encoded
-fexec-charset
: execution character set, i.e. how to encode string literals
-fwide-exec-charset
: wide execution character set, i.e. how to encode wide string literals
GCC uses iconv()
for the conversions, so any encoding supported by iconv()
can be used for those options.
I wrote previously about some opaque facilities provided by the C++ standard to handle text encodings.
Example: take the above code, char s[] = "Hello";
. Suppose the source file is ASCII (i.e. the input encoding is ASCII). Then the compiler reads 99
, and interprets it as c
, and so on. When it comes to the literal, it reads 72
, interprets it as H
. Now it stores the byte value of H
in the array which is determined by the execution encoding (again 72
if that is ASCII or UTF-8). When you write \xFF
, the compiler reads 99 120 70 70
, decodes it as \xFF
, and writes 255
into the array.
My answer is essentially the same as in your other question on this topic:
$ iconv -f UTF-16LE -t UTF-8 myfile.txt | grep pattern
As in the other question, you might need line ending conversion as well, but the point is that you should convert the file to the local encoding so you can use native tools directly.
Best Answer
When Vim reads an existing file, it tries to detect the file encoding. When writing out the file, Vim uses the file encoding that it detected (except when you tell it differently). So a file detected as UTF-8 is written as UTF-8, a file detected as Latin-1 is written as Latin-1, and so on.
By default, the detection process is crude. Every file that you open with Vim will be assumed to be Latin-1, unless it detects a Unicode byte-order mark at the top. A UTF-8 file without a byte-order mark will be hard to edit because any multibyte characters will be shown in the buffer as character sequences instead of single characters.
Worse, Vim, by default, uses Latin-1 to represent the text in the buffer. So a UTF-8 file with a byte-order mark will be corrupted by down-conversion to Latin-1.
The solution is to configure Vim to use UTF-8 internally. This is, in fact, recommended in the Vim documentation, and the only reason it is not configured that way out of the box is to avoid creating enormous confusion among users who expect Vim to operate basically as a Latin-1 editor.
In your
.vimrc
, addset encoding=utf-8
and restart Vim.Now Vim will use UTF-8 to represent the text in the buffer. Plus, it will also make a more determined effort to detect the UTF-8 encoding in a file. Besides looking for a byte-order mark, it will also check for UTF-8 without a byte-order mark before falling back to Latin-1. So it will no longer corrupt a file coded in UTF-8, and it should properly display the UTF-8 characters during the editing session.
For more information on how Vim detects the file encoding, see the
fileencodings
option in the Vim documentation.For more information on setting the encoding that Vim uses internally, see the
encoding
option.If you need to override the encoding used when writing a file back to disk, see the
fileencoding
option.