Bash – CPU is free, yet bash script not utilizing all CPU resources

bashperformanceshellshell-scripttime

I ran a simple script to generate a large (10000000 lines) csv file with 6 fields in which some fields changed in each line/row, using a while loop. The machine had all (32) CPUs free, lot of RAM (~31 Gb) was free too.

I timed the script with the command

/usr/bin/time -v bash script.01.sh

After running for around 2 hours, I got the following stats:

Command being timed: "bash script.01.sh"
User time (seconds): 1195.14
System time (seconds): 819.71
Percent of CPU this job got: 27%
Elapsed (wall clock) time (h:mm:ss or m:ss): 2:01:10
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 4976
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 3131983488
Voluntary context switches: 22593141
Involuntary context switches: 10923348
Swaps: 0
File system inputs: 0
File system outputs: 2182920
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0

I want to know why did my script used only 27% of the CPU? The Disk IO was nothing much at all (saw in vmstat output). So what caused the restriction? The code in the script?

Here is the script:

#!/usr/bin/env bash
number=1

while [[ $number -lt 10000001 ]] ; do  
    fname="FirstName LastName $"
    lname=""  
    email="fname.lname.$number@domain.com"  
    password="1234567890"  
    altemail="lname.fname.$number@domain.com"  
    mobile="9876543210"      

    echo "$fname,$lname,$email,$password,$altemail,$mobile" >> /opt/list.csv
    number=$(expr $number + 1)  
done

Best Answer

By using strace, I saw that the line

number=$(expr $number + 1)

causes a fork, path search, and exec of expr. (I'm using bash 4.2.45 on Ubuntu). That filesystem, disk, and process overhead led to bash only getting around 28% of the CPU.

When I changed that line to use only shell builtin operations

((number = number + 1))

bash used around 98% of the CPU and the script ran in a half hour. This was on a single-CPU 1.5GHz Celeron.

The script as is doesn't do anything that runs in parallel, so having 32 free CPUs won't help much. However, you can certainly parallelize it by, for instance, splitting it into 10 1-million-iteration loops that run in parallel, writing to 10 different files, and then using cat to combine them.

The following sample program was added by @Arthur2e5:

max=1000000 step=40000 tmp="$(mktemp -d)"
# Spawning. For loops make a bit more sense in a init-test-incr pattern.
for ((l = 0; l < max; l += step)); do (
    for ((n = l + 1, end = (step + l > max ? max : step + l);
      n <= end; n++)); do
        # Putting all those things into the `buf` line gives you a 1.8x speedup.
        fname="FirstName LastName \$"
        lname=""  
        email="fname.lname.$n@domain.com"  
        password="1234567890"  
        altemail="lname.fname.$n@domain.com"  
        mobile="9876543210"
        buf+="$fname,$lname,$email,$password,$altemail,$mobile"$'\n'
    done
    printf '%s\n' "$buf" > "$tmp/$l" ) &
done # spawning..
wait
# Merging. The filename order from globbing will be a mess,
# since we didn't format $l to some 0-prefixed numbers.
# Let's just do the loop again.
for ((l = 0; l < max; l += step)); do
    printf '%s\n' "$(<"$tmp/$l")" >> /opt/list.csv
done # merging..
rm -rf -- "$tmp" # cleanup
Related Question