How to run scripts in parallel on a remote machine

parallelism

I can ssh into a remote machine that has 64 cores. Lets say I need to run 640 shell scripts in parallel on this machine. How do I do this?

I can see splitting the 640 scripts into 64 groups each of 10 scripts. How would I then run each of these groups in parallel, i.e. one group on each of one of the available cores.

Would a script of the form

    ./script_A &
    ./script_B &
    ./script_C &
    ...

where script_A corresponds to the first group, script_B to the second group etc., suffice?

The scripts within one group that run on one core are ok to run sequentially, but I want the groups to run in parallel across all cores.

Best Answer

This looks like a job for gnu parallel:

parallel bash -c ::: script_*

The advantage is that you don't have to group your scripts by cores, parallel will do that for you.

Of course, if you don't want to babysit the SSH session while the scripts are running, you should use nohup or screen

Related Question