Shell – Why can’t `paste` print stdin next to stderr

io-redirectionpasteshellstderr

Usually paste prints two named (or equivalent) files in adjacent columns like this:

paste <(printf '%s\n' a b) <(seq 2)

Output:

a   1
b   2

But when the two files are /dev/stdin and /dev/stderr, it doesn't seem to work the same way.

Suppose we have blackbbox program which outputs two lines on standard output and two lines on standard error. For illustration purposes, this can be simulated with a function:

bb() { seq 2 | tee >(sed 's/^/e/' > /dev/stderr) ; }

Now run annotate-output, (in the devscripts package on Debian/Ubuntu/etc.), to show that it works:

annotate-output bash -c 'bb() { seq 2 | tee >(sed 's/^/e/' > /dev/stderr) ; }; bb'
22:06:17 I: Started bash -c bb() { seq 2 | tee >(sed s/^/e/ > /dev/stderr) ; }; bb
22:06:17 O: 1
22:06:17 E: e1
22:06:17 O: 2
22:06:17 E: e2
22:06:17 I: Finished with exitcode 0

So it works. Feed bb to paste:

bb | paste /dev/stdin /dev/stderr

Output:

1   e1
e2
^C

It hangs — ^C means pressing Control-C to quit.

Changing the | to a ; also doesn't work:

bb ; paste /dev/stdin /dev/stderr

Output:

1
2
e1
e2
^C

Also hangs — ^C means pressing Control-C to quit.

Desired output:

1    e1
2    e2

Can it be done using paste? If not, why not?

Best Answer

Why you can't use /dev/stderr as a pipeline

The problem isn't with paste, and neither is it with /dev/stdin. It's with /dev/stderr.

All commands are created with one open input descriptor (0: standard input) and two outputs (1: standard output and 2: standard error). Those can typically be accessed with the names /dev/stdin, /dev/stdout and /dev/stderr respectively, but see How portable are /dev/stdin, /dev/stdout and /dev/stderr?. Many commands, including paste, will also interpret the filename - to mean STDIN.

When you run bb on its own, both STDOUT and STDERR are the console, where command output usually appears. The lines go through different descriptors (as shown by your annotate-output) but ultimately end up in the same place.

When you add a | and a second command, making a pipeline...

bb | paste /dev/stdin /dev/stderr

the | tells the shell to connect the output of bb to the input of paste. paste first tries to read from /dev/stdin, which (via some symlinks) resolves to its own standard input descriptor (which the shell just connected up) so the line 1 comes through.

But the shell/pipeline does nothing to STDERR. bb still sends that (e1 e2 etc.) to the console. Meanwhile, paste attempts to read from the same console, which hangs (until you type something).

Your link Why can't I read /dev/stdout with a text editor? is still relevant here because those same restrictions apply to /dev/stderr.

How to make a second pipeline

You have a command that produces both standard output and standard error, and you want to paste those two lines next to each other. That means two concurrent pipes, one for each column. The shell pipeline ... | ... provides one of those, and you're going to need to create the second yourself, and redirect STDERR into that using 2>filename.

mkfifo RHS
bb 2>RHS | paste /dev/stdin RHS

If this is for use in a script, you may prefer to make that FIFO in a temporary directory, and remove it after use.

Related Question