Windows tool to convert large text file from UTF-8 to “Windows Unicode” (UTF-16)

Using gVim on Windows, I did the following two things:

:set encoding=utf-8
:set guifont=*

The second command brings up a font picker. By choosing the font "@MS Mincho", I got some of the Japanese characters to display, but oddly they were rotated 90 degrees to the left.

Anyway, you'll have to set the encoding before loading or pasting text into gVim (otherwise it might just convert them to all question marks). Then you'll have to find a font that is (a) fixed width, and (b) includes the characters you want to see. I don't seem to have such a font on my system at the moment, but you may.

I don't know a ton about sed, but after some hard Googling it seems to have support for a variety of code pages through the LANG environment variable. I believe UTF-8 is in fact the default in the absence of LANG. I don't know how the Windows port is set up though. I do have a strong suspicion that sed performs no detection processing at all on the input stream.

Sources: https://stackoverflow.com/questions/67410/why-does-sed-fail-with-international-characters-and-how-to-fix http://omgili.com/mailinglist/cygwin/cygwin/com/20100520123926GA1432onderneming10xs4allnl.html

You could also try escape characters as mentioned here: http://forums.whirlpool.net.au/forum-replies-archive.cfm/841095.html That seems very cumbersome though.