Shell – Two input pipes through file descriptor shuffling and /dev/fd

file-descriptorsio-redirectionkshshell

I want to pipe two programs into one. If my shell supports it, I can use Process substitution.
For example, to list the common lines of two files in indifferent order, I can use

comm -12 <(sort a) <(sort b)

However process substitution doesn't exist in plain sh. I can do it with full POSIX portability by creating a named pipe, but this is cumbersome as it requires finding a directory for the FIFO and cleaning up afterwards. A good practical compromise is to use two shell pipe constructs and use file descriptor shuffling to move one pipe to another file descriptor, then use /dev/fd to designate the pipe, which works on most Unix variants:

sort a | { exec 3<&0; sort b | comm -12 /dev/fd/0 /dev/fd/3; }

This works in dash, bash, BusyBox sh, etc. but not in ksh93 and mksh. Why?

$ mksh -c 'sort a | { exec 3<&0; sort b | comm -12 /dev/fd/0 /dev/fd/3; }'
$ ksh93 -c 'sort a | { exec 3<&0; sort b | comm -12 /dev/fd/0 /dev/fd/3; }'
comm: /dev/fd/0: No such device or address

Best Answer

Unlike redirections on other commands, redirections on the exec builtin may be closed when the shell executes an external program. POSIX allows both behaviors. Ksh (both ATT ksh, and pdksh and mksh) close these descriptors when they execute an external utility (i.e. for a redirection on the exec builtin, after calling dup2 to perform the redirection, they set the FD_CLOEXEC flag on the new descriptor). The Bourne shell, dash, bash, zsh and BusyBox sh treat this redirection like any other redirection.

A more portable solution to the two-input-pipes problem (assuming the existence of /dev/fd) is to perform another redirection on the command that reads the input, moving the file descriptor to a new one. This extra redirection doesn't set the close-on-exec flag on the new descriptor.

sort a | { exec 3<&0; sort b | comm -12 /dev/fd/0 /dev/fd/4 4<&3; }

This works in pdksh/mksh, and in ksh93r but not in recent versions of ksh (93s+ 2008-01-31 or 93u+ 2012-08-01). I don't understand what ksh is doing there.

Related Solutions

Shell – When to Use an Additional File Descriptor

Most commands have a single input channel (standard input, file descriptor 0) and a single output channel (standard output, file descriptor 1) or else operate on several files which they open by themselves (so you pass them a file name). (That's in addition from standard error (fd 2), which usually filters up all the way to the user.) It is however sometimes convenient to have a command that acts as a filter from several sources or to several targets. For example, here's a simple script that separates the odd-numbered lines in a file from the even-numbered ones

while IFS= read -r line; do
  printf '%s\n' "$line"
  if IFS= read -r line; then printf '%s\n' "$line" >&3; fi
done >odd.txt 3>even.txt

Now suppose you want to apply a different filter to the odd-number lines and to the even-numbered lines (but not put them back together, that would be a different problem, not feasible from the shell in general). In the shell, you can only pipe a command's standard output to another command; to pipe another file descriptor, you need to redirect it to fd 1 first.

{ while … done | odd-filter >filtered-odd.txt; } 3>&1 | even-filter >filtered-even.txt

Another, simpler use case is filtering the error output of a command.

exec M>&N redirects a file descriptor to another one for the remainder of the script (or until another such command changes the file descriptors again). There is some overlap in functionality between exec M>&N and somecommand M>&N. The exec form is more powerful in that it does not have to be nested:

exec 8<&0 9>&1
exec >output12
command1
exec <input23
command2
exec >&9
command3
exec <&8

Other examples that may be of interest:

What does “3>&1 1>&2 2>&3” do in a script? (it swaps stdout with stderr)
File descriptors & shell scripting
How big is the pipe buffer?
Bash script testing if a command has run correctly

And for even more examples:

questions tagged io-redirection
questions tagged file-descriptors
search for examples on this site in the Data Explorer (a public read-only copy of the Stack Exchange database)

_{P.S. This is a surprising question coming from the author of the most upvoted post on the site that uses redirection through fd 3!}

File descriptor and fork

When a child is forked then it inherits parent's file descriptors, if child closes the file descriptor what will happen ?

It inherits a copy of the file descriptor. So closing the descriptor in the child will close it for the child, but not the parent, and vice versa.

If child starts writing what shall happen to the file at the parent's end ? Who manages these inconsistencies , kernel or user ?

It's exactly (as in, exactly literally) the same as two processes writing to the same file. The kernel schedules the processes independently, so you will likely get interleaved data in the file.

However, POSIX (to which *nix systems largely or completely conform), stipulates that read() and write() functions from the C API (which map to system calls) are "atomic with respect to each other [...] when they operate on regular files or symbolic links". The GNU C manually also provisionally promises this with regard to pipes (note the default PIPE_BUF, which is part of the proviso, is 64 kiB). This means that calls in other languages/tools, such as use of echo or cat, should be included in that contract, so if two indepenedent process try to write "hello" and "world" simultaneously to the same pipe, what will come out the other end is either "helloworld" or "worldhello", and never something like "hweolrllod".

when a process call close function to close a particular open file through file descriptor.The file table of process decrement the reference count by one.But since parent and child both are holding the same file(there refrence count is 2 and after close it reduces to 1)since it is not zero so process still continue to use file without any problem.

There are TWO processes, the parent and the child. There is no "reference count" common to both of them. They are independent. WRT what happens when one of them closes a file descriptor, see the answer to the first question.

Best Answer

Related Solutions

Shell – When to Use an Additional File Descriptor

File descriptor and fork

Related Question