Shell – Bash, how to let some background processes run but wait for others

background-processshellshell-script

I have (yet) another wait, &, && control flow question..

Say I have a script something like this where I want to do as much work at the same time as possible:

# may take some hours
something InputA > IrrelevantA &
something InputB > IrrelevantB &

# may take an hour
(
   somethingElse InputA > OutputA &
   somethingElse InputB > OutputB &
)&& combine OutputA OutputB > Result

...morestuff

Question 1: In the script, does combine wait for both somethingElse processes to finish while both something processes continue?

Question 2: If not — and I suspect it doesn't — how do I get combine to wait only for both somethingElse processes whilst something processes above continue to work away in the background?

Best Answer

In your example the combine command will just be run as soon as the subshell exits (and provided the last background process was started without an error). The subshell will exit immediately after the jobs are started since there is no wait command.

If you want to execute a command based on the return value of two or more simultaneous background processes, then I can't see any other way other than to use temporary files for the return values. This is because wait can only return the return value of one of the processes it waits for. Also since the background processes must be run in subshells to get their return values at all, they cannnot be stored in variables. You could do:

something InputA >IrrelevantA &
something InputB >IrrelevantB &

tmp1=$(mktemp)
tmp2=$(mktemp)

( somethingElse InputA >OutputA; echo $? >"$tmp1" ) &
proc1=$!

( somethingElse InputB >OutputB; echo $? >"$tmp2" ) &
proc2=$!

wait "$proc1" "$proc2"

read ret1 <"$tmp1"
read ret2 <"$tmp2"
[ "$ret1" = 0 && "ret2" = 0 ] && combine OutputA OutputB >Result

rm "$tmp1" "$tmp2"

If you don't really care about the return values, you can just start the jobs normally and use wait:

something InputA >IrrelevantA &
something InputB >IrrelevantB &

somethingElse InputA >OutputA &
proc1=$!

somethingElse InputB >OutputB &
proc2=$!

wait "$proc1" "$proc2"
combine OutputA OutputB >Result
Related Question