Note that the problem is not with tail
but with head
here which reads from the pipe more than the first line it is meant to output (so there's nothing left for tail
to read).
And yes, it's POSIX conformant.
head
is required to leave the cursor within stdin just after the last line it has output when the input is seekable, but not otherwise.
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap01.html:
When a standard utility reads a seekable input file and terminates without an error before it reaches end-of-file, the utility shall ensure that
the file offset in the open file description is properly positioned just past the last byte processed by the utility. For files that are not
seekable, the state of the file offset in the open file description for that file is unspecified.
For head
to be able to do that for a non-seekable file would mean it would have to read one byte at a time which would be terribly inefficient¹. That's what the read
or line
utility do or GNU sed
with the -u
option.
So you can replace head -n 20
with gsed -u 20q
if you want that behaviour.
Though here, you'd rather want:
sed -e 1b -e '$b' -e d
instead. Here, only one tool invocation, so no problem with an internal buffer that can't be shared between two tool invocations. Note however that for large files, it's going to be less efficient as sed
reads the whole file, while for seekable files tail
would skip most of it by seeking near the end of the file.
See the related discussion about buffering at Why is using a shell loop to process text considered bad practice?.
Note that tail
must output the tail of the stream on stdin. While, as an optimisation and for seekable files, implementations may seek to the end of the file to get the trailing data from there, it is not allowed to seek back to a point that would be before the initial position at the time tail
was invoked (Busybox tail
used to have that bug).
So for instance in:
{ cat; tail -n 1; } < file
Even though tail
could seek back to the last line of file
, it does not. Its stdin is an empty stream as cat
left the cursor at the end of the file; it's not allowed to reclaim data from that stream by seeking further backward in the file.
(Text above crossed out pending clarification by the Open Group and considering that it's not done correctly by several implementations)
¹ The head
builtin of ksh93
(enabled if you put /opt/ast/bin
ahead of $PATH
), for sockets (a type of non-seekable files) instead peeks at the input (using recvfrom(..., MSG_PEEK)
) prior to actually reading it to see how much it needs to read to make sure it doesn't read too much. And falls back to reading one byte at a time for other types of files. That is slightly more efficient and I believe is the main reason why it implements its pipes with socketpair()
s instead of pipe()
. Note that it's not completely fool proof as there's a race condition that could be triggered if another process read from the socket in between the peek and the read.
Best Answer
tail
shows the last X lines.tail -f
does the same, but essentially in an infinite loop: on start-up, show the last X lines of the file, then using some OS magic (like inotify), monitor and show new lines.To do its job,
tail
must be able to locate the end of the file. Iftail
cannot find the file's end, it cannot show the last X lines, because "last" is undefined. So what doestail
do in this case? It waits until it does find the end of the file.Consider this:
This never appears to make progress, because there is never a definite end of file from
chatter
.You get the same behavior if you ask
tail
to give you the last lines from a file system pipe. Consider:stdbuf
to get around the perceived problem was a noble attempt. The key fact though is that I/O buffering isn't the root cause: the lack of a definite end-of-file is. If you check out the tail.c source code, you'll see thefile_lines
function comment reads:and that's the magic. You need an end-of-file for tail to work in any configuration.
head
doesn't have that restriction, it just needs a start of file (which it might not have, tryhead test.pipe
). Stream oriented tools likesed
andawk
need neither a start or end of file: they work on buffers.