Dash: Pipe STDIN to multiple commands and their output to STDOUT in defined order

dashfile-descriptorsio-redirectionpipetee

At first I thought this answer was the solution, but now I think I need a temporary file as buffer.

This works unreliably:

#!/bin/sh
echo 'OK' |
{
    {
        tee /dev/fd/3 | head --bytes=1 >&4
    } 3>&1 | tail --bytes=+2 >&4
} 4>&1

When I run this in a terminal, sometimes I get:

OK

and sometimes I get:

K
O

Seems totally random. So as a workaround I'm writing the output of tail to a file and reading it back to stdout after the pipe has finished.

#!/bin/sh
echo 'OK' |
{
    {
        tee /dev/fd/3 | head --bytes=1 >&4
    } 3>&1 | tail --bytes=+2 >file
} 4>&1
cat file

Can this be done in dash without temporary files? Shell variables as buffer aren't an option either, as the output might contain NUL bytes.

Best Answer

If you wanted to run the consumers and producer in parallel, but serialize the output of the consumers, you'd need to delay the output of the second consumer. For that, you'd need to store its output somehow and the best way is with a temporary file.

With zsh:

{cat =(producer > >(consumer1 >&3) | consumer2)} 3>&1

bash has an issue in that it doesn't wait for the process substitution commands, so you'd have to use nasty work arounds there.

Here, we're using the =(...) form of process substitution to store the output of comsumer2 in a temporary file and cat it afterwards. We can't do that for more than 2 consumers. For that, we'd need to create the temp files by hand.

When not using =(...), we'd have to handle the clean up of the tempfiles by hand. We can handle that by creating and deleting them up front so not to have to worry about the cases where the script is killed. Still with zsh:

tmp1=$(mktemp) && tmp2=$(mktemp) || exit
{
  rm -f -- $tmp1 $tmp2
  producer > >(consumer1) > >(consumer2 >&3) > >(consumer3 >&5)
  cat <&4 <&6
} 3> $tmp1 4< $tmp1 5> $tmp2 6< $tmp2

Edit (I initially missed the fact that a solution for dash was required)

For dash (or any POSIX shell that doesn't set the close-on-exec flag on fds above 2 and uses pipes and not socketpairs for |), and on systems with /dev/fd/x support:

tmp1=$(mktemp) && tmp2=$(mktemp) || exit
{
  rm -f -- "$tmp1" "$tmp2"
  {
    {
      {
        producer | tee /dev/fd/4 /dev/fd/6 | consumer1 >&7
      } 4>&1 | consumer2 >&3
    } 6>&1 | consumer3 >&5
  } 7>&1
  cat - /dev/fd/6 <&4
} 3> "$tmp1" 4< "$tmp1" 5> "$tmp2" 6< "$tmp2"

That would work with dash, bash, zsh, mksh, busybox sh, posh on Linux, but not ksh93. That approach can't go beyond 4 consumers as we're limited to fds 0 to 9.

Related Question