Why does “tail -f … | tail” fail to produce any output

pipetail

Why does the following command not produce any output?

$ tail -f /etc/passwd | tail

After reading about buffering, I tried the following to no avail:

$ tail -f /etc/passwd | stdbuf -oL tail

Note that the following does produce output:

$ tail /etc/passwd | tail

So does this:

$ tail -f /etc/passwd | head

I am using tail version 8.21 (GNU coreutils).

Best Answer

tail shows the last X lines. tail -f does the same, but essentially in an infinite loop: on start-up, show the last X lines of the file, then using some OS magic (like inotify), monitor and show new lines.

To do its job, tail must be able to locate the end of the file. If tail cannot find the file's end, it cannot show the last X lines, because "last" is undefined. So what does tail do in this case? It waits until it does find the end of the file.

Consider this:

$ chatter() { while :; do date; sleep 1; done; }
$ chatter | tail -f

This never appears to make progress, because there is never a definite end of file from chatter.

You get the same behavior if you ask tail to give you the last lines from a file system pipe. Consider:

$ mkfifo test.pipe
$ tail test.pipe

stdbuf to get around the perceived problem was a noble attempt. The key fact though is that I/O buffering isn't the root cause: the lack of a definite end-of-file is. If you check out the tail.c source code, you'll see the file_lines function comment reads:

END_POS is the file offset of EOF (one larger than offset of last byte).

and that's the magic. You need an end-of-file for tail to work in any configuration. head doesn't have that restriction, it just needs a start of file (which it might not have, try head test.pipe). Stream oriented tools like sed and awk need neither a start or end of file: they work on buffers.

Related Question