Replace UTF-8 characters with shell perl

perlunicode

How do I get perl to properly replace UTF-8 character from a shell?

The examples use stdin, but I need something that works for perl ... file too.

This is what I expect:

$ echo ABCæøåDEF | perl -CS -pe "s/([æøå])/[\\1]/g"
ABC[æ][ø][å]DEF

This is what I get:

$ echo ABCæøåDEF | perl -CS -pe "s/([æøå])/[\\1]/g"
ABCæøåDEF

Replacing the Unicode characters with ASCII works instantly:

$ echo ABC123DEF | perl -CS -pe "s/([123])/[\\1]/g"
ABC[1][2][3]DEF

My environment:

perl 5.18.2
Bash 3.2.57
LC_ALL=en_US.UTF-8
LANG=en_US.UTF-8

Best Answer

Use this :

 $ echo 'ABCæøåDEF' |
    perl -CSD -Mutf8 -pe 's/([æøå])/[$1]/g'

Works also for files

Output :

ABC[æ][ø][å]DEF

Note :

substitutions: \\1 is for awk, \1 is for sed and in perl we use $1
check perldoc perlrun for -CSD utf8 tricks

Related Solutions

How to do a regex search in a UTF-16LE file while in a UTF-8 locale

My answer is essentially the same as in your other question on this topic:

$ iconv -f UTF-16LE -t UTF-8 myfile.txt | grep pattern

As in the other question, you might need line ending conversion as well, but the point is that you should convert the file to the local encoding so you can use native tools directly.

Linux – Detect how much of Unicode the terminal supports, even through screen

Well, first I guess I would point out that pretty much all terminals these days are "virtual" in the sense you talk about... even if the terminal is at the other end of a bona fide serial port. I mean, the days of VT-100s, Wyse terminals and other "physical", "real" terminals are pretty much gone!

That aside, let's say you want to detect what kind of Unicode support your terminal has. You can do this by writing test characters to is and seeing what happens. (You can make an effort to erase the test characters after you've written then, but the user may still see them briefly, or erasing them might not work properly in the first place.)

The idea is to ask the terminal to tell you its cursor position, output a test character, ask the terminal again to tell you its position, and compare the two positions to see how far the terminal's cursor moved.

To ask the terminal for its position, see here. Essentially:

echo -e "\033[6n"; read -d R foo; echo -en "\nCurrent position: "; echo $foo | cut -d \[ -f 2

Try outputting "é". This character takes 2 bytes in UTF-8 but displays in only one column on the screen. If you detect that outputting "é" causes the cursor to move by 2 positions, then the terminal has no UTF-8 support at all and has probably output some kind of garbage. If the cursor didn't move at all, then then terminal is probably ASCII only. If it moved by 1 position, then congratulations, it can probably display French words.

Try outputing "あ". This character takes 3 bytes in UTF-8 but displays in only two columns on the screen. If the cursor moves by 0 or 3, bad news, similar to above. If it moves by 1, then it looks like the terminal supports UTF-8 but doesn't know about wide characters (in fixed-width fonts). If it moves by 2 columns, all is good.

I'm sure there are other probe characters that you could emit which would lead to useful information. I am not aware of a tool that does this automatically.