Ubuntu – How to run commands in the shell in parallel without disrupting the output

command linescripts

I have the following shell script:

./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 10 0.1 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 10 0.2 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 10 0.5 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 10 1 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 25 0.1 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 25 0.2 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 25 0.5 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 1 0 25 1 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 2 0 10 0.1 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 2 0 10 0.2 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 2 0 10 0.5 &
./run 50 5000 100 100 1.0 2 0.3 0.3 0.05 1 101 0 2 2 0 10 1

And my main function has a structure like this:

int main(int argc, char** argv)
{
    if(argc<18)
    {
        cout<<"Insufficient parameters"<<endl;
        cout<<"loop #ofGen popSize chrLen Pc PmNumerator randPopRate BOAimmigrantsRate Pn algoType #ofBOAsamples mkpfileNo noiseType prb env per sev"<<endl;
        exit(1);
    }

    int algoType;// GA or PBIL
    int mkpfile;
    loop = atoi(argv[1]);
    GA.generation = atoi(argv[2]);
    GA.popSize = atoi(argv[3]);
    GA.chromosomeLength = atoi(argv[4]);
    GA.Pc = atof(argv[5]);
    GA.PmNumerator = atoi(argv[6]);
    GA.randomPopulationRate = atof(argv[7]);
    GA.ImmigrantRateFromBOA = atof(argv[8]);
    DE.Pn = atof(argv[9]);
    algoType = atoi(argv[10]);
    CM.numOfSamples = atoi(argv[11]);
    mkpfile=atoi(argv[12]);
    DE.noiseType=atoi(argv[13]);
    DE.problemType=atoi(argv[14]);
    DE.environmentType = atoi(argv[15]);
    DE.period=atoi(argv[16]);
    DE.severity=atof(argv[17]);

    printf("\nRunning... Problem type: %d...",DE.problemType);
    fflush(stdout);
    for(int i=1; i<=loop; i++)
    {
        myAlgorithm(i,DE.problemType,algoType,mkpfile);
    }
    cout<<"Done!"<<endl;
    return 0;
}

When I run the above code, I want output to first prints the printf() part which doesn't have newline thing, and then prints the cout part:

Running bla bla bla... Done!

It runs correctly if I run only one test case, however when I use shell script to run more than one test cases in parallel, it becomes like this:

    Running... Problem type: 1...Running... Problem type: 1...
Running... Problem type: 1...
Running... Problem type: 1...
Running... Problem type: 1...
Running... Problem type: 1...
Running... Problem type: 1...
Running... Problem type: 1...
Running... Problem type: 2...
Running... Problem type: 2...
Running... Problem type: 2...
Running... Problem type: 2...Done!
Done!
Done!

Running... Problem type: 2...
Running... Problem type: 2...
Running... Problem type: 3...
Running... Problem type: 3...
Running... Problem type: 3...
Running... Problem type: 3...
Running... Problem type: 2...
Running... Problem type: 3...
Running... Problem type: 3...

Running... Problem type: 2...Running... Problem type: 3...
Running... Problem type: 3...Done!
Done!
Done!
Done!
Done!
Done!
Done!
Done!
Done!
Done!
Done!

Is there way to make that happen properly? I am asking this here since I think it is an Ubuntu-related problem.

Best Answer

Meet GNU parallel Install parallel (sudo apt install parallel):

GNU parallel is a shell tool for executing jobs in parallel using one or more computers. (…) [It] makes sure output from the commands is the same output as you would get had you run the commands sequentially. (man parallel)

In your case with repeating arguments you can make use of bash Brace Expansion to easily build the command lines. The basic syntax is parallel COMMAND {} ::: ARGUMENTS, if you want to give each run multiple arguments take care of proper quoting to prevent word splitting, e.g.:

$ parallel echo ./run some args {} ::: {"1 0 "{10,25},"2 0 10"}\ {0.{1,2,5},1}
./run some args 1 0 10 0.1
./run some args 1 0 10 0.2
./run some args 1 0 10 0.5
./run some args 1 0 10 1
./run some args 1 0 25 0.1
./run some args 1 0 25 0.2
./run some args 1 0 25 0.5
./run some args 1 0 25 1
./run some args 2 0 10 0.1
./run some args 2 0 10 0.2
./run some args 2 0 10 0.5
./run some args 2 0 10 1
Related Question