Bash – How to make a function available to the command `parallel` (GNU)

bashfunctionparallelismshell

In Bash, let's consider a function that does nothing but echo the argument followed by "is an integer".

f () { num="${!1}"; echo $num is an integer; }
number=12
f number
# 12 is an integer

I would like to write on a file a number of commands that uses the function f and then run these commands in parallel using the function parallel (GNU).

# Write Commands to the file `Commands.txt`
rm Commands.txt
touch Commands.txt
for i in $(seq 1 5)
do
   echo "number=$i; f number" >> Commands.txt
done

With source everything work fine

source Commands.txt
1 is an integer
2 is an integer
3 is an integer
4 is an integer
5 is an integer

However, when I try to run the commands in parallel it returns that the function f is not found

parallel :::: Commands.txt
/bin/bash: f: command not found
/bin/bash: f: command not found
/bin/bash: f: command not found
/bin/bash: f: command not found
/bin/bash: f: command not found

Is there a way I can make my function f available for parallel without having to define the function at each line of the file Commands.txt?

Best Answer

You basically have three options:

  1. export -f (which is a non-POSIX bash feature)
  2. Execute a shell defining the function on each invocation
  3. Move the function to a shell script and run that instead

Option 1 is probably the easiest to demonstrate, so I'll show that one:

$ f() { num=$1; echo "$num is an integer"; }
$ export -f f
$ cat Commands.txt 
number=1; f "$number" 
number=2; f "$number" 
number=3; f "$number" 
number=4; f "$number" 
number=5; f "$number" 
$ parallel :::: Commands.txt
1 is an integer
2 is an integer
3 is an integer
4 is an integer
5 is an integer

Note that your population of Commands.txt is likely in error, you need f "$number" in Commands.txt to pass the number, not f number which would pass the literal string "number". Your script to generate it should do echo "number=$i; f \"\$number\"" (note the escapes, which are important to avoid $number being interpreted at echo time, or the "s terminating the string).

Related Question