Unix – How to Limit Background Processes

background-processnohupshell-script

I have to run a script in 100 parts while making sure that only 10 parts run at parallel at any point of time.

Below script will trigger all the 100 processes at the same time :-

for i in {1..100}
do
    nohup ksh my_background_script.ksh -mod ${i} &
done

Best Answer

There are two tools commonly used for this, xargs and GNU parallel:

  1. xargs is older, very often installed by default, but more limited. You would use it like this:

     seq 1 100 | xargs -P 10 -I {} nohup ksh my_background_script.ksh -mod {} &

    The -P 10 is telling it to run 10 processes in parallel.

  2. GNU parallel, which although not installed by default on many systems, should be readily available for any GNU/Linux system and is far more powerful than xargs. You would use it like this (probably, the details depend on what you are running):

     parallel -j 10 nohup ksh my_background_script.ksh -mod ::: {1..100}

Some more details on the two programs and their respective uses can be found here: GNU parallel vs & (I mean background) vs xargs -P

Related Question