Shell – How to use GNU Parallel for this while loop

If you want to parallelize on a machine with multiple cores, you can just use (GNU) xargs, e.g.:

echo seq_[0-9][0-9].gz | xargs -n 1 -P 16 ./crunching

Meaning: xargs starts up to 16 processes in parallel of ./crunching using 1 token from stdin for each process.

You can also use split in combination with xargs.

Or you can create a simple Makefile for Job execution and call make -f mymf -j $CORES (you need temporary files for this solution).

PS: The GNU parallel manual also includes some comparisons with other tools, including xargs and make, interestingly they write:

(Very early versions of GNU parallel were coincidently implemented using make -j).

It is due to GNU Parallel --pipe being slow.

cat bigfile |  parallel --pipe -L1000 --round-robin grep -f regexp.txt -

maxes out at around 100 MB/s.

In the man page example you will also find:

parallel --pipepart --block 100M -a bigfile grep -f regexp.txt

which does close to the same, but maxes out at 20 GB/s on a 64 core system.

parallel --pipepart --block 100M -a bigfile -k grep -f regexp.txt

should give exactly the same result as grep -f regexp.txt bigfile