Reading and writing a file is not line-atomic, and tail -f
is not line-buffered. So, when tail -f
reads the file while it was still being written, it may reach the end of the file (and therefore stop printing) before the process writing to the file writes the end of the line. When that happens, it prints an incomplete line because that's all it sees.
(take this with a pinch of salt) As far as I remember, the problem lies in the way libiconv
works. Multi-byte encodings need a state machine to decode them, and libiconv
prefers to receive entire characters, so you can't just give it half a character in one function call and the other half in the next.
I can think of another two solutions, one is a good out-of-band method, the other is an in-band hack.
Change Terminal Emulator encoding (out-of-band): one is to change the character encoding in your terminal emulator, so its native encoding is Shift JIS. I just checked konsole
, and is supports this. From the menu, View→Character encoding→Japenese→sjis. You can then just tail -f
the file, and konsole
will take care of decoding the multibyte characters and matching them up to font glyphs.
Transcode terminal encoding on the fly (in-band; best): courtesy of Gilles, who reminded me of luit
after a very long time. Use luit
, which should have come with your XOrg distribution (on Debian, it's package x11-utils
). Use it like this:
$ luit -encoding SJIS -- tail -f x
This will make the terminal transcode SJIS to/from your terminal encoding, and run tail -f x
. The downside of luit
is that it doesn't support the wealth of encodings supported by libiconv
. The upside is it's available almost everywhere.
Transcode terminal encoding on the fly (in-band; hack): ttyconv
is a hack I wrote many years ago (initially in C, later redone in Python) which uses libiconv
to transcode terminal I/O. It spawns a new pseudoterminal and (a) transcodes the characters you type from your local encoding into the remote encoding, and (b) transcodes the characters you receive from the remote encoding to your local encoding. I used it to talk to servers that used encodings not supported by the standard Linux terminals. Please note that all of the remote encodings I tested it with were single-byte encodings, so I can't guarantee it'd work for Shift JIS. I don't often find call to use it these days, with most systems switching to Unicode.
This is how you would use it:
$ ttyconv -rsjis -- tail -f x
The downside of ttyconv
is that I wrote it, no-one uses it but me, it's probably full of bugs. I excel at this. The upside is that it uses libiconv
, so if your encoding is unusual, it's your best bet. At last count, ttyconv --list
supports 100 encodings.
Best Answer
It's not
tail
, it's the piping.mysql
uses a tabular with ASCII boxing output format when its stdout is a terminal device, when it's intended for a user, and reverts to a scripting format when it's not, like when it's a pipe or a regular file.You'd see the same different format with
or
See also the
-r
/--raw
,-s
/--silent
,-B
/--batch
,-N
/--skip-column-names
/--column-names=0
,-H
/--html
,-t
/--table
... that affect the output format.If you want the tabular output even when the output doesn't go to a terminal device, add the
-t
option:But if the point is to remove the header line, just use
-N
, with or without-t
.Here to get the values from the database as raw as possible and without header, I'd use:
That is:
ps
by passing the credentials in a file instead (like yourmy.cnf
) with--defaults-extra-file
.--raw
to avoid the escaping. Assuming the values don't contain newlines as otherwise the output could not be post-processed reliably.--skip-column-names
to remove the header line.