How to scan for invalid characters on gedit

gedit

I am having a dilemma whether to edit a javascript file or not. When I open it with gedit, it shows the following warning:

The file you opened has some invalid characters. If you continue
editing this file you could corrupt this document. You can also choose
another character encoding and try again.

The current encoding is UTF-8. As the file has over 100,000 lines of code, is there a quick way to scan for the invalid characters?

Best Answer

As the file is UTF-8 you could run isutf8. An additional utils package. It gives you both line, char and offset for bad bytes.

Then use xxd, hexdump or the like to analyze.

Unfortunately it stops at first crash. But then again it depends on the file. Could be there is only one bad byte ;)

_{Have some C code that does a similar analysis but for entire file. It is on
a disk somewhere long forgotten. Could try to find it if in need.}

Else yes, the quick and not that dirty way would be to do a diff between a copy saved with gedit – as proposed by the good mr. @vonbrand.

Related Solutions

How to write a new syntax highlighter for gedit

GtkSourceView is a portable C library which supports configurable syntax highlighting in gedit.
Create an xml file with custom syntax highlighting for the language.
Save it in /usr/share/gtksourceview*/language-specs as language_name.lang

Refer Language Definition v2.0 Reference for help with syntax highlighting.

Gedit or an other non-commandline editor with the ability to spell-check two languages at the same time

Aspell Author Here.

As I said in an earlier answer, you can't just combine dictionaries from different languages and expect it to work. You need to create a new language that combines the features of the two original languages.

Fortunately for English and German this is fairly easy; however, the suggestion quality will suffer for English words since we will disable the use of soundslike lookup.

Install the aspell-en and the aspell-de dictionary package
Go to a empty directory to keep everything clean. Also to avoid any charset issues change the locale to "C" by setting LC_ALL=C.

Dump the English and German dictionaries into plan wordlists

aspell dump master en > en.txt
aspell dump master de > de.txt

Combine en.dat and de.dat, which you can generally find in usr/lib/aspell.

The English dictionary uses soundslike lookup but that won't with the German dictionary (due to the fact that it is English specific, and more importable it is incompatible with Affix compression) so we will disable it. The English dictionary doesn't use affix compression but the German dictionary does so we will just use the affix file for the German dictionary. (This will avoid having to expand the German dictionary and thus increasing it size).

We will call the language qed, 'q' since very few languages start with q, 'e' for English, and 'g' for German. (The language name should generally be 2 to 3 letters, but aspell doesn't really care, so en-de or some other name might work, but a 2 or 3 letter name is guaranteed to work)

The file will be named qed.dat and contain the following:
```
name qed
charset iso8859-1
special ' -*-
soundslike      none
affix           qed
affix-compress  true
```
Copy de_affix.dat into the current directory and rename it qed_affix.dat.

Create the combined dictionary:

cat en.txt de.txt | aspell create master -l ./qed ./qed.rws

Create the file qed.multi:
```
add qed.rws
```
Test the dictionary by using -d ./qed. the ./ is needed to force aspell to search the current directory.
Install qed.dat qed.rws qed.multi and qed_affix.dat somewhere where aspell will find it. Please see the manual for info on how aspell searches for dictionary and language data files.
Done. Everything should work now. A more sophisticated solution will enable some form of soundslike lookup for better suggestion quality. But that requires special care when used with affix compression (see the Aspell manual for details). As an alternative the German dictionary can be expanded and the English soundlike lookup can be used, but that might not work so well on German words.

The case of combing English and German was easy because they both use the same charset (iso-8859-1) and because only one language used Affix compression. Combining other languages will take more work but it is possible once you know what you are doing. I spelled out the steps here in detail to give readers some idea of how Aspell works so a similar thing can be used for other language combinations.

If both languages use affix compression, either the affix files will need to be combined so there are no conflicting flags, or one of the dictionaries will need to be expanded.

If the two languages use a different 8-bit charset than a compatible charset that can support both languages we need to be used. If a standard one doesn't exist, than a new one can be created. To avoid confusion the wordlists should be converted to utf-8 and Aspell should be instructed to expect all input and output in utf-8 instead of the charset that is used internally, which for historical reasons is the default.

Best Answer

Related Solutions

How to write a new syntax highlighter for gedit

Gedit or an other non-commandline editor with the ability to spell-check two languages at the same time

Related Question