The
echo one; echo two > >(cat); echo three;
command gives unexpected output.
I read this: How process substitution is implemented in bash? and many other articles about process substitution on the internet, but don't understand why it behaves this way.
Expected output:
one
two
three
Real output:
prompt$ echo one; echo two > >(cat); echo three;
one
three
prompt$ two
Also, this two commands should be equivalent from my point of view, but they don't:
##### first command - the pipe is used.
prompt$ seq 1 5 | cat
1
2
3
4
5
##### second command - the process substitution and redirection are used.
prompt$ seq 1 5 > >(cat)
prompt$ 1
2
3
4
5
Why I think, they should be the same? Because, both connects the seq
output to the cat
input through the anonymous pipe – Wikipedia, Process substitution.
Question: Why it behaves this way? Where is my error? The comprehensive answer is desired (with explanation of how the bash
does it under the hood).
Best Answer
Yes, in
bash
like inksh
(where the feature comes from), the processes inside the process substitution are not waited for (before running the next command in the script).for a
<(...)
one, that's usually fine as in:the shell will be waiting for
cmd1
andcmd1
will be typically waiting forcmd2
by virtue of it reading until end-of-file on the pipe that is substituted, and that end-of-file typically happens whencmd2
dies. That's the same reason several shells (notbash
) don't bother waiting forcmd2
incmd2 | cmd1
.For
cmd1 >(cmd2)
, however, that's generally not the case, as it's morecmd2
that typically waits forcmd1
there so will generally exit after.That's fixed in
zsh
that waits forcmd2
there (but not if you write it ascmd1 > >(cmd2)
andcmd1
is not builtin, use{cmd1} > >(cmd2)
instead as documented).ksh
doesn't wait by default, but lets you wait for it with thewait
builtin (it also makes the pid available in$!
, though that doesn't help if you docmd1 >(cmd2) >(cmd3)
)rc
(with thecmd1 >{cmd2}
syntax), same asksh
except you can get the pids of all the background processes with$apids
.es
(also withcmd1 >{cmd2}
) waits forcmd2
like inzsh
, and also waits forcmd2
in<{cmd2}
process redirections.bash
does make the pid ofcmd2
(or more exactly of the subshell as it does runcmd2
in a child process of that subshell even though it's the last command there) available in$!
, but doesn't let you wait for it.If you do have to use
bash
, you can work around the problem by using a command that will wait for both commands with:That makes both
cmd1
andcmd2
have their fd 3 open to a pipe.cat
will wait for end-of-file at the other end, so will typically only exit when bothcmd1
andcmd2
are dead. And the shell will wait for thatcat
command. You could see that as a net to catch the termination of all background processes (you can use it for other things started in background like with&
, coprocs or even commands that background themselves provided they don't close all their file descriptors like daemons typically do).Note that thanks to that wasted subshell process mentioned above, it works even if
cmd2
closes its fd 3 (commands usually don't do that, but some likesudo
orssh
do). Future versions ofbash
may eventually do the optimisation there like in other shells. Then you'd need something like:To make sure there's still an extra shell process with that fd 3 open waiting for that
sudo
command.Note that
cat
won't read anything (since the processes don't write on their fd 3). It's just there for synchronisation. It will do just oneread()
system call that will return with nothing at the end.You can actually avoid running
cat
by using a command substitution to do the pipe synchronisation:This time, it's the shell instead of
cat
that is reading from the pipe whose other end is open on fd 3 ofcmd1
andcmd2
. We're using a variable assignment so the exit status ofcmd1
is available in$?
.Or you could do the process substitution by hand, and then you could even use your system's
sh
as that would become standard shell syntax:though note as noted earlier that not all
sh
implementations would wait forcmd1
aftercmd2
has finished (though that's better than the other way round). That time,$?
contains the exit status ofcmd2
; thoughbash
andzsh
makecmd1
's exit status available in${PIPESTATUS[0]}
and$pipestatus[1]
respectively (see also thepipefail
option in a few shells so$?
can report the failure of pipe components other than the last)Note that
yash
has similar issues with its process redirection feature.cmd1 >(cmd2)
would be writtencmd1 /dev/fd/3 3>(cmd2)
there. Butcmd2
is not waited for and you can't usewait
to wait for it either and its pid is not made available in the$!
variable either. You'd use the same work arounds as forbash
.