Ubuntu – Convert files to UTF-8 without BOM

encodingutf-8wine

I have a file in Hebrew, which is displayed now as Gibberish. On windows, what I've done at the past, I just converted the file using Notepad++ > Encoding > Convert to UTF-8 without BOM and then Hebrew was displayed.

On Ubuntu if I'm using Notepad++ (running on Wine) when I convert it, it is still displayed as Gibberish (but success to convert it to UTF-8 without BOM).

I wonder if there is a software which converts a file to UTF-8 without BOM at the same way as Notepad++ does (Convert -> Hebrew displayed > save).

I don't know why it still displays GIbberish on Notepad + Wine when I convert it. Wine supports Hebrew as I know (I can write their Hebrew and save it without problem).

Best Answer

Try Kate.

Kate is a multi-document editor. Choose it for viewing HTML sources, editing configuration files, writing new applications or any other text editing task. You still need just one running instance of Kate.

With a multi-view editor like Kate you get a lot of advantages. You can view several instances of the same document and all instances are synchronized. Or you can view more files at the same time for easy reference or simultaneous editing.¹

To install kate, just press Ctrl+Alt+T on your keyboard to open Terminal. When it opens, run the command(s) below:

sudo apt-get install kate

^{¹Source:Kate}

Related Solutions

Ubuntu – gedit can’t recognize character encoding, but gvim can

iconv is probably what you'll want to use. iconv -l will show you the available encodings and then you can use a couple of commands to recode them all:

# all text files are in ./originals/
# new files will be written to ./newversions/

mkdir -p newversions
cd originals
for file in *.txt; do
    cat $file | iconv -f ASCII -t utf-8 > ../newversions/$file;
done

If you want to do this with files you don't the encoding of (because they're all over the place), you want to bring in a few more commands: find, file, awk and sed. The last two are just there to process the output of file.

for file in find . -type f -exec file --mime {} \; | grep "ascii" | awk '{print $1}' | sed s/.$//; do
    ...

I've no idea if this actually works so I certainly wouldn't run it from anything but the least important directory you have (make a testing folder with some known ASCII files in). The syntax of find might preclude it from being within a for loop. I'd hope that somebody else with more bash experience could jump in there and sort it out so it does the right thing.

Ubuntu – Convert Text File Encoding

These Windows files with Persian text are encoded in Windows-1256. So it can be deciphered by command similar to OP tried, but with different charsets. Namely:

~~recode Windows-1256..UTF-8 <Windows_file.txt > UTF8_file.txt~~
(denounced upon original poster’s complaints; see comments)

iconv -f Windows-1256 Windows_file.txt > UTF8_file.txt

This one assumes that the LANG environment variable is set to a UTF-8 locale. To convert to any encoding (UTF-8 or otherwise), regardless of the current locale, one can say:

iconv -f Windows-1256 Windows_file.txt -t ${output_encoding} > ${output_file}

Original poster is also confused with semantic of text recoding tools (recode, iconv). For source encoding (source.. or -f) one must specify encoding with which the file is saved (by the program that created it). Not some (naïve) guesses based on mojibake characters in programs that try (but fail) to read it. Trying either ISO-8859-15 or WINDOWS-1252 for a Persian text was obviously an impasse: these encodings merely do not contain any Persian letter.

Best Answer

Related Solutions

Ubuntu – gedit can’t recognize character encoding, but gvim can

Ubuntu – Convert Text File Encoding

Related Question