I'd like to use the feature of GNU parallel where it can execute the command and the list it's fed in parallel and spit it out after it's all done, however, I don't want to install GNU parallel across all of our servers.
Or perhaps a parallel version of xargs
?
Is there a ksh implementation of what GNU Parallel does? In this case, it doesn't have to be done in order as GNU Parallel does — just as long as all the output can be piped or stored. I'd also like to avoid using temporary files.
Best Answer
If you want to parallelize on a machine with multiple cores, you can just use (GNU)
xargs
, e.g.:Meaning:
xargs
starts up to 16 processes in parallel of./crunching
using 1 token from stdin for each process.You can also use
split
in combination with xargs.Or you can create a simple Makefile for Job execution and call
make -f mymf -j $CORES
(you need temporary files for this solution).PS: The GNU parallel manual also includes some comparisons with other tools, including xargs and make, interestingly they write: