Shell Scripting Parallelism – How to Execute a Program on Multiple Files in Parallel

parallelismscriptingshell-script

I have a small script that loops through all files of a folder and executes a (usually long lasting) command. Basically it's

for file in ./folder/*;
do
    ./bin/myProgram $file > ./done/$file
done

(Please Ignore syntax errors, it's just pseudo code).

I now wanted to run this script twice at the same time. Obviously, the execution is unnecessary if ./done/$file exists. So I changed the script to

for file in ./folder/*;
do
    [ -f ./done/$file ] || ./bin/myProgram $file >./done/$file
done

So basically the question is:
Is it possible that both scripts (or in general more than one script) actually are at the same point and check for the existance of the done file which fails and the command runs twice?

it would be just perfect, but I highly doubt it. This would be too easy 😀
If it can happen that they process the same file, is it possible to somehow "synchronize" the scripts?

Best Answer

This is possible and does occur in reality. Use a lock file to avoid this situation. An example, from said page:

if mkdir /var/lock/mylock; then
    echo "Locking succeeded" >&2
else
    echo "Lock failed - exit" >&2
    exit 1
fi

# ... program code ...

rmdir /var/lock/mylock

Related Solutions

Background execution in parallel

With GNU xargs:

seq 1000 | xargs -P4 -n1  sh -c 'exec ./pgm -a "file$1" -b 12 > "out.$1"' sh &

Would run up to 4 ./pgms in parallel.

Otherwise, with pdksh/mksh/oksh:

trap : CHLD
n=0
for f in file*; do
  jobs=$(jobs | wc -l)
  if (($jobs < 4)); then
    ./pgm "$f" > out.$((++n)) &
  else
    wait
  fi
done
trap - CHLD
wait

the details of signal handling vary from one shell to the next. That trick works in pdksh and its derivatives but not in any other shell I tried. You need a shell where one can trap SIGCHLD (excludes bash), where the SIGCHLD handler is executed straight away (not blocked during a wait) (excludes ash, yash), where the SIGCHLD handling interrupts the wait (excludes ksh93 and zsh).

In shells other than bash, you could also look at approaches where jobs are started in the SIGCHLD handler.

Bash Script Execution – Fixing Stuck Bash Scripts

Here's the short and sweet - a heredoc is basically a file streamed to a file-descriptor.

Most people don't denote the 0<<descriptor and so you get it on <&0 stdin. ssh passes stdin to its invoked process so if you feed it a heredoc it will pass through the input to the invoked remote shell.

The one very special quality about heredocs is the difference between a \"'quoted and unquoted heredoc LIMITER. So, <<'THIS' differs from <<THIS. When you do not quote the LIMITER, the contents of the here-document are evaluated for ${shell:+expansion}. Once one ${shell:+expansion} pass is completed, there is very little else to distinguish a here-document from any other file fed as <~/input.

For example:

cat <<\QUOTED >~/file
    $(echo "This is ${NOT:-} expanded.")
#END
QUOTED
cat <~/file
> $(echo "This is ${NOT:-} expanded.")
> #END

But...

cat <<UNQUOTED >~/file
    $(echo "This is ${NOT:-} expanded.")
#END
UNQUOTED
cat <~/file
> This is  expanded.
> #END

You keep using the bash <<< herestring with cat. I don't know exactly how the herestring works but I'm willing to bet cat's already involved. So cat concatenates its <&0stdin with its stdout>&1. That's all it does. So you're unnecessarily complicating <<STDIN when you <<< cat it.

This can be a real problem if cat winds up consuming an input stream which you did not intend it to consume. Run just % cat at your terminal and it will look like nothing is happening because a terminal's stdin and stdout are the same file - your $(tty). But when they differ, cat combines them anyway and that can get pretty messy if you didn't mean it to happen.

It looks to me like some \'quotes are skipping an expansion when $(date) is $expanded. Then possibly the : null shell builtin is invoked and |piped to the next unquoted command after ssh which would be cat >> report.fail which should generate nothing at all in that file. So cat is >>appending /dev/null to report.fail, for as long as it can stand it, I guess. Or, more likely, for as long as exit allows ssh to carry-on proxying the null stream.

Also, have you checked to see if you've got a literal $TMP in your current working directory? I do see ENDSSH at the bottom which looks like a heredoc LIMITER to me so I believe either this is not the entire script or it has been edited by mistake. It would make sense if it were the body of a heredoc to use \$TMP, but as is I think nc will first >truncate then write its stdout to a file named $TMP. Then again, I guess you rm it anyway, so maybe you just didn't notice.

And because you're rming $TMP you may not have realized anyone would be asking this question:

How does $(mktemp) work for you without the filename.xxx argument?

UPDATE I looked closer at your output and definitely >/tmp/tmp.GJ1knZF5Jn means $(mktemp) is working - even the \$TMP part. So you've just taught me that I only need to specify mktemp .xxx if I specify a filename at all. Thank you.

Still I think there is a heredoc at the top somewhere? It could be there is not and \\ is only an attempt to deal with a side-effect of the echo \$TMP <<<herestring, but I dunno... Interesting.

I don't know if I've got this completely right because I don't know where all of these variables come from. But, this is close to how I would do this:

(which actually renders the last two questions irrelevant anyway)

_ssh() ( ssh "$1"@"$2" 'printf %s, `cat` >> '"$3"
) <<-PARAMS   
    "$lastSourceIP" 
    "$lastDestinationIP" 
    "$sourceFqdn" 
    "$fqdn" 
    "$port" "
    "ConnectivityNA" 
    "NA" 
    "NA" 
    "Pas" 
    "$(date)"
PARAMS

nc -z -v -n $lastDestinationIP $port |\
    grep -q "succeeded" && suffix=txt
_ssh user host report.${suffix:-fail}
unset suffix
?ENDSSH?

Note: the "$quotes" in the above are for printf's benefit on the other side of the ssh process - not for anything else. Those "quotes" remain as is even after all of the above PARAMS are evaluated.

There are some things going on up there that I've covered before. For instance the func() ( scope ) I like to think was covered ok here. The ${parameter:-expansion} was also covered there, but also demonstrated pretty well here and here. I've gotten into some of the weird heredoc stuff here, here, and here. Probably there are others - I guess I like messing up my shell or something.

In this case, though, using the function and the heredoc as I have, cat can't get stuck. PARAMS is sent over as stdin and cat will quit when it reaches an EOF (or CTRL-D) so as soon as it consumes PARAMS it's going to stop everytime. This is especially important if you are running this in a heredoc which is also on <&0 because PARAMS will stand in the way of cat eating your script mid-execute.

Anyway, hopefully that helps. If I've missed anything, don't hesitate to ask.

Best Answer

Related Solutions

Background execution in parallel

Bash Script Execution – Fixing Stuck Bash Scripts

Related Question