Can two asynchronous subshell commands safely write to a shared stdout

concurrencypipe

Can stdout be overwritten by two bourne (or bash, if that matters) subshell commands running asynchronously?

(tail -f ./file1 & tail -f ./file2) | cat

I don’t care about the line order, just that every output line is made up of exactly one input line. I’m worried that some lines could be partially overwritten or interleaved.

I’ve tested this by running four commands that each output a unique line 15 million times each. It appears to work, but I kind of expected it to fail.

Can someone explain how this doesn’t break? Is each subshell buffered and only one subshell gets to write to stdout at a time? or how is this managed.

Is there a better way to do this?

(Never mind that I’m using tail in the above subshells for illustrative purposes. I actually want to run two other commands that output continuously one line at the time to stdout.)

Best Answer

The shells have little involvement there. All they do is create the pipe and start those 3 commands which then run in parallel independently from the shell.

What matters here is that both tail commands write to a file descriptor to the same writing end of the same pipe.

If you do:

printf foo1 >> file1; sleep 1
printf foo2 >> file2; sleep 1
printf 'bar1\n' >> file1; sleep 1
printf 'bar2\n' >> file2

You'll see:

foo1foo2bar1
bar2

Because that's how those were written. You'll want to make sure your commands output one full line at a time, and that those lines be smaller than PIPE_BUF (4096 bytes on Linux) for the write() to be guaranteed to be atomic (it could also write more than one full line at a time provided they're all full and their cumulative size is less than PIPE_BUF).

With GNU grep, you can do that by piping your commands to grep --line-buffered '^'

(tail -f ./file1 | grep --line-buffered '^' &
 tail -f ./file2 | grep --line-buffered '^') | cat

That would guarantee that you get one write() system call for each line of the output of both commands (In the cases where the commands don't terminate their last line of output, grep would add the missing newline)

Related Question