Poor Man’s GNU Parallel implemented in ksh

gnugnu-parallelkshparallelism

I'd like to use the feature of GNU parallel where it can execute the command and the list it's fed in parallel and spit it out after it's all done, however, I don't want to install GNU parallel across all of our servers.

Or perhaps a parallel version of xargs?

Is there a ksh implementation of what GNU Parallel does? In this case, it doesn't have to be done in order as GNU Parallel does — just as long as all the output can be piped or stored. I'd also like to avoid using temporary files.

Best Answer

If you want to parallelize on a machine with multiple cores, you can just use (GNU) xargs, e.g.:

echo seq_[0-9][0-9].gz | xargs -n 1 -P 16 ./crunching

Meaning: xargs starts up to 16 processes in parallel of ./crunching using 1 token from stdin for each process.

You can also use split in combination with xargs.

Or you can create a simple Makefile for Job execution and call make -f mymf -j $CORES (you need temporary files for this solution).

PS: The GNU parallel manual also includes some comparisons with other tools, including xargs and make, interestingly they write:

(Very early versions of GNU parallel were coincidently implemented using make -j).

Related Question