Linux – gnu parallel pair argument with file input arguments

gnu-parallellinux

I have a file with many lines, and on each line I have the arguments I want to pass to parallel with a tab separator.

I run this script

cat $hdd_file | grep $ssd | parallel -C '\t' clean_and_destroy

And it works, $hdd_file is the filename, the grep collects the lines which hdds have a certain $ssd as a cache, and then the parallel calls a function which destroys their connection.

Now that I made new partitions to the cleaned ssds, I try to call parallel like this:

cat $hdd_file | grep $ssd | parallel -C '\t' create_new_cache :::+ `seq $partitions_per_ssd`

Which should get the arguments from the pipe and pair them with the numbers given, but it does not.

cat $hdd_file | grep $ssd | parallel -C '\t' create_new_cache ::: {} :::+ `seq $partitions_per_ssd`

I also tried this and it still doesn't work. The {} :::+ are passed as arguments for some reason

Best Answer

GNU parallel solution:

Sample input.txt (for demonstration):

a   b
c   d
e   f

grep '^[ac]' input.txt will be used to emulate command(or pipeline) acting like input source file

parallel -C '\t' echo :::: <(grep '^[ac]' input.txt) ::: $(seq 1 3)

The output:

a b 1
a b 2
a b 3
c d 1
c d 2
c d 3
  • :::: argfiles - treat argfiles as input source. ::: and :::: can be mixed.

To aggregate elements from each input source - add --xapply option:

parallel -C '\t' --xapply echo :::: <(grep '^[ac]' input.txt) ::: $(seq 1 2)

The output:

a b 1
c d 2
Related Question