“tail -f | iconv -fsjis” does not output anything

buffertail

I want to tail -f a file, but its content is in sjis encoding, so I need to have it converted to the native (utf-8) encoding of my terminal.

When I do

tail -f x | iconv -fsjis

there will be no output. As

tail x | iconv -fsjis

does work, at first I thought it was a buffering issue, but trying unbuffer and stdbuf as described on Turn off buffering in pipe did not help.

In fact, even after more than 10k of data were added to x, there would be no output, so I guess it is not a buffering issue (buffer is 4k, if I'm not mistaken), but iconv will only start outputting when it receives an EOF.

So how can I tail-follow my sjis encoded file?

Best Answer

(take this with a pinch of salt) As far as I remember, the problem lies in the way libiconv works. Multi-byte encodings need a state machine to decode them, and libiconv prefers to receive entire characters, so you can't just give it half a character in one function call and the other half in the next.

I can think of another two solutions, one is a good out-of-band method, the other is an in-band hack.

Change Terminal Emulator encoding (out-of-band): one is to change the character encoding in your terminal emulator, so its native encoding is Shift JIS. I just checked konsole, and is supports this. From the menu, View→Character encoding→Japenese→sjis. You can then just tail -f the file, and konsole will take care of decoding the multibyte characters and matching them up to font glyphs.

Transcode terminal encoding on the fly (in-band; best): courtesy of Gilles, who reminded me of luit after a very long time. Use luit, which should have come with your XOrg distribution (on Debian, it's package x11-utils). Use it like this:

$ luit -encoding SJIS -- tail -f x

This will make the terminal transcode SJIS to/from your terminal encoding, and run tail -f x. The downside of luit is that it doesn't support the wealth of encodings supported by libiconv. The upside is it's available almost everywhere.

Transcode terminal encoding on the fly (in-band; hack): ttyconv is a hack I wrote many years ago (initially in C, later redone in Python) which uses libiconv to transcode terminal I/O. It spawns a new pseudoterminal and (a) transcodes the characters you type from your local encoding into the remote encoding, and (b) transcodes the characters you receive from the remote encoding to your local encoding. I used it to talk to servers that used encodings not supported by the standard Linux terminals. Please note that all of the remote encodings I tested it with were single-byte encodings, so I can't guarantee it'd work for Shift JIS. I don't often find call to use it these days, with most systems switching to Unicode.

This is how you would use it:

$ ttyconv -rsjis -- tail -f x

The downside of ttyconv is that I wrote it, no-one uses it but me, it's probably full of bugs. I excel at this. The upside is that it uses libiconv, so if your encoding is unusual, it's your best bet. At last count, ttyconv --list supports 100 encodings.

Related Solutions

Does tail Read the Whole File? – Understanding tail Command Behavior

No, tail doesn't read the whole file, it seeks to the end then read blocks backwards until the expected number of lines have been reached, then it displays the lines in the proper direction until the end of the file, and possibly stays monitoring the file if the -f option is used.

Note however that tail has no choice but to read the whole data if provided a non seekable input, for example when reading from a pipe.

Similarily, when asked to look for lines starting from the beginning of the file, with using the tail -n +linenumber syntax or tail +linenumber non standard option when supported, tail obviously reads the whole file (unless interrupted).

Why does “tail -f … | tail” fail to produce any output

tail shows the last X lines. tail -f does the same, but essentially in an infinite loop: on start-up, show the last X lines of the file, then using some OS magic (like inotify), monitor and show new lines.

To do its job, tail must be able to locate the end of the file. If tail cannot find the file's end, it cannot show the last X lines, because "last" is undefined. So what does tail do in this case? It waits until it does find the end of the file.

Consider this:

$ chatter() { while :; do date; sleep 1; done; }
$ chatter | tail -f

This never appears to make progress, because there is never a definite end of file from chatter.

You get the same behavior if you ask tail to give you the last lines from a file system pipe. Consider:

$ mkfifo test.pipe
$ tail test.pipe

stdbuf to get around the perceived problem was a noble attempt. The key fact though is that I/O buffering isn't the root cause: the lack of a definite end-of-file is. If you check out the tail.c source code, you'll see the file_lines function comment reads:

END_POS is the file offset of EOF (one larger than offset of last byte).

and that's the magic. You need an end-of-file for tail to work in any configuration. head doesn't have that restriction, it just needs a start of file (which it might not have, try head test.pipe). Stream oriented tools like sed and awk need neither a start or end of file: they work on buffers.

Best Answer

Related Solutions

Does tail Read the Whole File? – Understanding tail Command Behavior

Why does “tail -f … | tail” fail to produce any output

Related Question