character encodingencodingfile-corruptiontext-editorswindows 7
So, I have on a flash drive a txt file generated in Cyrillic (my own work, own pen drive), a few years old. Now I needed to open it, only to see this kind of
mess.
I wonder why is this happening and how can I restore it back to normal.. I tried saving it under Unicode and UTF-8 encoding, even some MS-DOS format (an option from Wordpad) but it makes no difference at all.
I'm guessing you mentioned "batch" simply because you're looking for a way to automate this and thought a batch file might be the best solution. Well it isn't, so unless you specifically need to use a batch file (in which case I'll modify/delete this answer), use a better tool for the job instead such as Notepad++ (or any other comparable full-featured text editor):
There are many other programs whose sole job is text search/replace, for example grepWin:
The odds are that what you see as <92> and <94> are windows-1252 encoded “smart” (curly) apostrophe and “smart” right double quotation mark. They could be just about anything, of course, but in UTF-8, such bytes cannot appear as “standalone”, only as the 2nd or later byte of a multi-byte representation of a character,
Best Answer
What you're seeing is referred to as mojibake. In short, the application you are opening the file with is using the wrong encoding to try and read the file. The standard fix is to use a transcoding tool, either online or offline (though I know of no free ones for Windows which work offline), or open the document in an application that lets you set the encoding and save it through that as the desired encoding.
As a somewhat hacky alternative, if you can save the file without modifying the encoding, you can change the extension to
.eml, format it like an email message, make sure theContent-Typeheader specifies the correct encoding, and then open the resulting file in a good email client (pretty much anything except Outlook or Windows Mail) and copy the text out of there to a text editor and save it.For future reference, the generally accepted method of avoiding this is to save files as either UTF-8 or UTF-16 (UTF-8 is usually preferred, as it's better supported by most platforms other than Windows than UTF-16).
In particular, your file does indeed appear to be encoded using KOI-8 (determined based on the statement that the text is Cyrillic and the apparent distribution of actual characters), with the application apparently interpreting it as ISO-8859-1 or Windows codepage 1252(determined simply based on what is being displayed, plus the fact that these are standard fallback encodings for many devices).