OS X Bash – Less Works on Open File Descriptors, Cat Doesn’t

bashcatfile-descriptorslessosx

In a bash script I'm working on (which has to run on Ubuntu and OS X), I need to redirect the output of hundreds of commands to a file.
Rather than appending &>... to all of them, I simply do

exec 9>&1
exec 5<>/tmp/some-file.txt
exec 1>&5

So far so good, but halfway through all those commands, I need to read everything that has been written so far, while keeping the file descriptor open.
Now, on Ubuntu I can simply do

cat /dev/fd/5

or

tee </dev/fd/5

but on OS X, nothing is printed at all (and the commands exit immediately).
However, using less I can see the contents of the file on both.
I can achieve the above effect (working on both OS'es) by using

less /dev/fd/5 | tee

but that seems like a hack.

So, why can less apparently see stuff that cat can't on OS X? (Or are all BSD descendants affected?)
Or am I doing something wrong?

Best Answer

On OS X, like on all systems where they are supported except Linux, opening /dev/fd/x is like doing a dup(x), the resulting fd more or less points to the same open file description as on fd x and in particular will have the same offset within the file.

Linux is the exception here. On Linux, /dev/fd/x is a symlink to /proc/self/fd/x and /proc/self/fd/x is a pseudo-symlink to the file open on fd x. On Linux when you do a open("/dev/fd/x", somemode), you get a brand new open file description to the same file as open on x. The new fd you obtain is not related to fd x in any way. In particular, the offset will be at the start of the file (except if you open it with O_APPEND of course) and the mode (read/write/append...) can be different from the one on fd x (you can even get something quite different from what's on fd x, like the other end of the pipe when opening it in the opposite mode). (That also means that that doesn't work for sockets for instance which you can't open()).

So, on Linux, when you do

exec 5<> file
echo test >&5

The fd 5's offset is at the end of the file. If you do

cat <&5

You get nothing.

Still when you do:

cat /dev/fd/5

You see test because cat gets a new read-only fd to file unrelated to fd 5.

On other systems, upon

cat /dev/fd/5

cat gets a fd that is a duplicate of fd 5, so still with an offset at the end of the file.

The reason why it works with less is that for some reason, less does a lseek() on that fd to the beginning of the file (does a lseek(1); lseek(0) to determine whether the file is seekable or not).

Here, you probably want to have a fd for reading and one for writing if you want both to have different offsets:

exec 5< file 9>&1 > file

Or you'll have to reopen the file if still there, or do an lseek() as less does.

ksh93 and zsh are the only shells with a builtin lseek() operator though:

cat <&5 <#((0)) # ksh93
{sysseek 0; cat} <&5 # zsh, zmodload zsh/system to enable that builtin

Or:

cat /dev/fd/5 5<#((0))  # ksh93
sysseek -u 5 0; cat /dev/fd/5 # zsh
Related Question