How to run GNU parallel in record per job, with 1 process per core

gnugnu-parallelparallelism

What I'm really trying to do is run X number of jobs, with X amount in parallel for testing an API race condition.

I've come up with this

echo {1..10} | xargs -n1 | parallel -m 'echo "{}"';

which prints

7 8 9
10
4 5 6
1 2 3

but what I really want to see is (note order doesn't actually matter). 

1
2
3
4
5
6
7
8
9
10

and those would be processed in parallel 4 at a time (or whatever number of cpus/cores, I have, e.g. --jobs 4). For a total of 10 separate executions.

I tried this

echo {1..10} | xargs -n1 | parallel --semaphore --block 3  -m 'echo -n "{} ";

but it only ever seems to print once. bonus points if your solution doesn't need xargs which seems like a hack around the idea that the default record separator is a newline, but I haven't been able to get a space to work like I want either.

10 is a reasonably small number, but lets say it's much larger, 1000

echo {1..1000} | xargs -n1 | parallel -j1000

prints

parallel: Warning: Only enough file handles to run 60 jobs in parallel.
parallel: Warning: Running 'parallel -j0 -N 60 --pipe parallel -j0' or
parallel: Warning: raising 'ulimit -n' or 'nofile' in /etc/security/limits.conf
parallel: Warning: or /proc/sys/fs/file-max may help.

I don't actually want 1000 processes, I want 4 processes at a time, each process should process 1 record, thus by the time I'm done it will have executed 1000 times.

Best Answer

I want 4 processes at a time, each process should process 1 record

parallel -j4 -k --no-notice 'echo "{}"' ::: {1..10}

  • -j4 - number of jobslots. Run up to 4 jobs in parallel

  • -k - keep sequence of output same as the order of input. Normally the output of a job will be printed as soon as the job completes

  • ::: - arguments

The output:

1
2
3
4
5
6
7
8
9
10
Related Question