Bash – Run Multiple Scripts in Parallel Instead of Sequence

background-processbashparallelismshell-script

Suppose that I have three (or more) bash scripts: script1.sh, script2.sh, and script3.sh. I would like to call all three of these scripts and run them in parallel. One way to do this is to just execute the following commands:

nohup bash script1.sh &
nohup bash script2.sh &
nohup bash script3.sh &

(In general, the scripts may take several hours or days to finish, so I would like to use nohup so that they continue running even if my console closes.)

But, is there any way to execute those three commands in parallel with a single call?

I was thinking something like

nohup bash script{1..3}.sh &

but this appears to execute script1.sh, script2.sh, and script3.sh in sequence, not in parallel.

Best Answer

for((i=1;i<100;i++)); do nohup bash script${i}.sh & done

Related Solutions

Bash – How to run multiple scripts in parallel

If you have gnu parallel installed, you could make a script with just the commands, e.g.:

../perlscripts/v2csv.pl -v -F reach results/Heterogeneous*.vec > ./results/csv/reach.csv
../perlscripts/v2csv.pl -v -F roundTrip results/Heterogeneous*.vec > ./results/csv/RT.csv
../perlscripts/v2csv.pl -v -F downlink results/Heterogeneous*.vec > ./results/csv/DL.csv
../perlscripts/v2csv.pl -v -F clusters results/Heterogeneous*.vec > ./results/csv/clusters.csv

and then run them in parallel:

mkdir ./results/csv && parallel :::: myscript.sh

Alternatively, invoking the command and using {} - the default replacement string:

mkdir ./results/csv && parallel ../perlscripts/v2csv.pl -v -F {} \
results/Heterogeneous*.vec '>' ./results/csv/{}.csv ::: reach roundTrip downlink clusters

would run the following commands in parallel:

../perlscripts/v2csv.pl -v -F reach results/Heterogeneous*.vec > ./results/csv/reach.csv
../perlscripts/v2csv.pl -v -F roundTrip results/Heterogeneous*.vec > ./results/csv/roundTrip.csv
../perlscripts/v2csv.pl -v -F downlink results/Heterogeneous*.vec > ./results/csv/downlink.csv
../perlscripts/v2csv.pl -v -F clusters results/Heterogeneous*.vec > ./results/csv/clusters.csv

BASH: parallel run

Summary of the comments: The machine is fast but doesn't have enough memory to run everything in parallel. In addition the problem needs to read a lot of data and the disk bandwidth is not enough, so the cpus are idle most of the time waiting for data.

Rearranging the tasks helps.

Not yet investigated compressing the data to see if it can improve the effective disk I/O bandwidth.

Best Answer

Related Solutions

Bash – How to run multiple scripts in parallel

BASH: parallel run

Related Question