Can GNU parallel output stdout before the program has exited

gnu-parallel

echo 'echo "hello, world!";sleep 3;' | parallel

This command does not output anything until it has completed. Parallel's man page claims:

GNU parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially.

I guess the devil is in the phrasing: you get the same output as if you would run it normally, but not the output the same as if you would run it normally. I've looked for an option that will do this, for example --results /dev/stdout, but that does not work.

My use-case is seeing real-time progress output from the command that I'm running. It's not about how many tasks have completed, which parallel could display for me, but about the progress output of each command individually that I want to see.

I would use a bash loop (for i in $x; do cmd & done;) but I want to be able to stop all tasks with a single Ctrl+C, which parallel allows me to do.

Is it possible to do this in parallel? If not, is there another tool?

Best Answer

I think you're looking for --ungroup. The man page says:

--group  Group output. Output from each jobs is grouped 
         together and is only printed when the command is finished. 

         --group is the default. Can be reversed with -u.

-u of course is a synonym for --ungroup.

Related Solutions

Shell – Combine parallel and sequential commands

Simply with GNU parallel:

parallel ::: 'pip install pipenv && pipenv install --dev' \
             'npm install -g grunt-cli && npm install'

Shell – GNU Parallel doesn’t run the jobs until the program has exited

You are being hit by two issues.

This

(seq 200; sleep 20) | parallel -j10  -k echo

prints:

1
2

and then stalls until the sleep 20 is done.

A partial fix seems to be to move start_more_jobs() outside the while loop:

--- a/src/parallel
+++ b/src/parallel
@@ -4062,9 +4062,8 @@ sub reaper {
        # $stiff = pid of dead process
        if(wantarray) {
            push(@pids_reaped,$stiff);
-       } else {
-           $children_reaped++;
        }
+       $children_reaped++;
         if($Global::sshmaster{$stiff}) {
             # This is one of the ssh -M: ignore
             next;
@@ -4112,12 +4111,12 @@ sub reaper {
             }
         }
        $job->cleanup();
-       start_more_jobs();
        if($opt::progress) {
            my %progress = progress();
            ::status_no_nl("\r",$progress{'status'});
        }
     }
+    if($children_reaped) { start_more_jobs(); }
     $opt::sqlmaster and $Global::sql->run("COMMIT;");
     debug("run", "done ");
     return wantarray ? @pids_reaped : $children_reaped;

This may cost some performance if you have many short lived jobs. I have not measured how much.

The other part of the problem is due to a design decision in GNU Parallel.

Arguments in GNU Parallel are read using the diamond operator (<>). This reads a full line before continuing. Reading from (sleep 20) only generates an end-of-file after sleep finishes, and thus blocks until sleep finishes.

So when GNU Parallel reads the final byte, it has to wait for the sleep to finish to discover that this indeed is the end-of-file.

I see no easy way to change that part of the design.

Luckily this does not block the jobs from being run as you can see if you run date. The jobs are started immediately, it is just the output that is waiting for the sleep:

(seq 20; sleep 5) | parallel -j10  -k 'date;echo'

In other words: Your problem is not related to -N2. You can not see the problem here:

(printf '%s\0' {1..4}; sleep 2) | parallel --null -k --lb -N 2 echo {1} {2}

But you can see the problem here. This pauses before the last 4-8 elements:

(printf '%s\0' {1..40}; sleep 2) | parallel -j4 --null -k --lb -N 2 echo {1} {2}

This pauses before the last 8-10 elements:

(printf '%s\0' {1..40}; sleep 2) | parallel -j8 --null -k --lb -N 2 echo {1} {2}

By running date you can see the problem is not the starting of the jobs - it is only postponing the printing:

(printf '%s\0' {1..40}; sleep 2) | parallel -j4 --null -k --lb -N 2 'date;'echo {1} {2}

Best Answer

Related Solutions

Shell – Combine parallel and sequential commands

Shell – GNU Parallel doesn’t run the jobs until the program has exited

Related Question