Suppose I want to execute multiple programs in parallel and combine their outputs to one pipe:
sh -c '
(echo qqq; echo qqq2; echo qqq3)&
(echo www; echo www2; echo www3)&
(echo eee; echo eee2; echo eee3)&
wait; wait; wait'
This shell approach works well for this simple case, but I expect it to fail if programs output more and longer lines in buffered way, like this (constructed):
qqq
qqwww
q2
qqq3www2
wwweee3
eee2
eee3
One of the solution I was hinted to use was tail -f
:
tail -n +0 -q -f <(echo qqq; echo qqq2; echo qqq3) <(echo www; echo www2; echo www3) <(echo eee; echo eee2; echo eee3)
, but this is sub-optimal option: it outputs data sluggishly, it does not terminate; I see outputs not in "sleep" order, but in arguments order in this case:
tail -n +0 -q -f <(sleep 1; echo qqq; sleep 1; echo qqq2; echo qqq3) <(echo www; echo www2; sleep 10; echo www3) <(echo eee; sleep 4; echo eee2; echo eee3) | cat
I've implemented a special little program for this, but believe that there should be some standard good way to do it.
How to do it using standard tools (and without tail -f
disadvantage)?
Best Answer
GNU Parallel.
From release notes dated August 2013:
For example:
parallel --line-buffer <jobs
Where
jobs
contains:short.sh
:long.sh
:Output: