Bash Script Execution – Fixing Stuck Bash Scripts

bashbash-scriptshell-script

I have a bash script that is reading a CSV file , that contains source IP address , destination IP address and destination port.

My script is basically doing 2 tests:

1 Performance
2 Connectivity.

Under performance test, I am copying files from source to destination (using scp) to calculate the the network IO. After that, copying the same file from one location of Destination server to other location on destination server to calculate Disk I/O.

Under connectivity test I am doing nc from source to destination to check whether port is open or not.

Problem : Sometime while execution of script, it get stuck at some point and then I have to press Ctrl+C to kill that process to continue the further execution.

Here , i am not able to figure it out why it is getting stuck, and even behavior is not consistent , every time it's stuck at different location in the script.

Please help me in resolving this issue. If required I can post my complete script here.

Update :

Observation : Other thing that i observed , if i don't press Ctrl+C , then script resume automatically after 5 mins.

Part of the actual script , where it is getting stuck.

#Connectivity Test
TMP=$(mktemp)
nc -z -v -n $lastDestinationIP $port >\$TMP
if grep -q "succeeded" <<< cat echo \$TMP;then
   echo $x','$lastSourceIP','$lastDestinationIP','$sourceFqdn','$fqdn','$port',Connectivity,NA,NA,NA,Pass,'$(date) | ssh $username@$baseLocation 'cat >> report.txt'
else
  echo $x','$lastSourceIP','$lastDestinationIP','$sourceFqdn','$fqdn','$port',Connectivity,NA,NA,NA,Fail,'$(date) | ssh $username@$baseLocation 'cat >> report.txt'
fi
rm $TMP
fi
exit
ENDSSH

Most of the time it stuck while calling exit.

Here is the snippet where it stuck most of the time (from the console).

21 14:48:16 EDT 2014 | ssh puppet-admin@10.X.X.9 'cat >> report.txt','Fri Mar
> fi
>
> #rm KB_33.txt
> #rm MB_10.txt
> #rm MB_100.txt
> rm -rf dummy
> else
> #Connectivity Test
> nc -z -v -n -w 2 10.X.X.17 1524 >/tmp/tmp.GJ1knZF5Jn
> if grep -q "succeeded" <<< cat echo /tmp/tmp.GJ1knZF5Jn;then
 21 14:48:16 EDT 2014 | ssh puppet-admin@10.X.X.9 'cat >> report.txt','Fri Mar
> else
21 14:48:16 EDT 2014 | ssh puppet-admin@10.X.X.9 'cat >> report.Fail,'Fri Mar
> fi
> fi
> exit
>

Best Answer

Here's the short and sweet - a heredoc is basically a file streamed to a file-descriptor.

Most people don't denote the 0<<descriptor and so you get it on <&0 stdin. ssh passes stdin to its invoked process so if you feed it a heredoc it will pass through the input to the invoked remote shell.

The one very special quality about heredocs is the difference between a \"'quoted and unquoted heredoc LIMITER. So, <<'THIS' differs from <<THIS. When you do not quote the LIMITER, the contents of the here-document are evaluated for ${shell:+expansion}. Once one ${shell:+expansion} pass is completed, there is very little else to distinguish a here-document from any other file fed as <~/input.

For example:

cat <<\QUOTED >~/file
    $(echo "This is ${NOT:-} expanded.")
#END
QUOTED
cat <~/file
> $(echo "This is ${NOT:-} expanded.")
> #END

But...

cat <<UNQUOTED >~/file
    $(echo "This is ${NOT:-} expanded.")
#END
UNQUOTED
cat <~/file
> This is  expanded.
> #END

You keep using the bash <<< herestring with cat. I don't know exactly how the herestring works but I'm willing to bet cat's already involved. So cat concatenates its <&0stdin with its stdout>&1. That's all it does. So you're unnecessarily complicating <<STDIN when you <<< cat it.

This can be a real problem if cat winds up consuming an input stream which you did not intend it to consume. Run just % cat at your terminal and it will look like nothing is happening because a terminal's stdin and stdout are the same file - your $(tty). But when they differ, cat combines them anyway and that can get pretty messy if you didn't mean it to happen.

It looks to me like some \'quotes are skipping an expansion when $(date) is $expanded. Then possibly the : null shell builtin is invoked and |piped to the next unquoted command after ssh which would be cat >> report.fail which should generate nothing at all in that file. So cat is >>appending /dev/null to report.fail, for as long as it can stand it, I guess. Or, more likely, for as long as exit allows ssh to carry-on proxying the null stream.

Also, have you checked to see if you've got a literal $TMP in your current working directory? I do see ENDSSH at the bottom which looks like a heredoc LIMITER to me so I believe either this is not the entire script or it has been edited by mistake. It would make sense if it were the body of a heredoc to use \$TMP, but as is I think nc will first >truncate then write its stdout to a file named $TMP. Then again, I guess you rm it anyway, so maybe you just didn't notice.

And because you're rming $TMP you may not have realized anyone would be asking this question:

How does $(mktemp) work for you without the filename.xxx argument?

UPDATE I looked closer at your output and definitely >/tmp/tmp.GJ1knZF5Jn means $(mktemp) is working - even the \$TMP part. So you've just taught me that I only need to specify mktemp .xxx if I specify a filename at all. Thank you.

Still I think there is a heredoc at the top somewhere? It could be there is not and \\ is only an attempt to deal with a side-effect of the echo \$TMP <<<herestring, but I dunno... Interesting.

I don't know if I've got this completely right because I don't know where all of these variables come from. But, this is close to how I would do this:

(which actually renders the last two questions irrelevant anyway)

_ssh() ( ssh "$1"@"$2" 'printf %s, `cat` >> '"$3"
) <<-PARAMS   
    "$lastSourceIP" 
    "$lastDestinationIP" 
    "$sourceFqdn" 
    "$fqdn" 
    "$port" "
    "ConnectivityNA" 
    "NA" 
    "NA" 
    "Pas" 
    "$(date)"
PARAMS

nc -z -v -n $lastDestinationIP $port |\
    grep -q "succeeded" && suffix=txt
_ssh user host report.${suffix:-fail}
unset suffix
?ENDSSH?

Note: the "$quotes" in the above are for printf's benefit on the other side of the ssh process - not for anything else. Those "quotes" remain as is even after all of the above PARAMS are evaluated.

There are some things going on up there that I've covered before. For instance the func() ( scope ) I like to think was covered ok here. The ${parameter:-expansion} was also covered there, but also demonstrated pretty well here and here. I've gotten into some of the weird heredoc stuff here, here, and here. Probably there are others - I guess I like messing up my shell or something.

In this case, though, using the function and the heredoc as I have, cat can't get stuck. PARAMS is sent over as stdin and cat will quit when it reaches an EOF (or CTRL-D) so as soon as it consumes PARAMS it's going to stop everytime. This is especially important if you are running this in a heredoc which is also on <&0 because PARAMS will stand in the way of cat eating your script mid-execute.

Anyway, hopefully that helps. If I've missed anything, don't hesitate to ask.

Related Solutions

Linux – What Happens If You Edit a Script During Execution?

In Unix, most editors work by creating a new temporary file containing the edited contents. When the edited file is saved, the original file is deleted and the temporary file renamed to the original name. (There are, of course, various safeguards to prevent dataloss.) This is, for example, the style used by sed or perl when invoked with the -i ("in-place") flag, which is not really "in-place" at all. It should have been called "new place with old name".

This works well because unix assures (at least for local filesystems) that an opened file continues to exist until it is closed, even if it is "deleted" and a new file with the same name is created. (It's not coincidental that the unix system call to "delete" a file is actually called "unlink".) So, generally speaking, if a shell interpreter has some source file open, and you "edit" the file in the manner described above, the shell won't even see the changes since it still has the original file open.

[Note: as with all standards-based comments, the above is subject to multiple interpretations and there are various corner-cases, such as NFS. Pedants are welcome to fill the comments with exceptions.]

It is, of course, possible to modify files directly; it's just not very convenient for editing purposes, because while you can overwrite data in a file, you cannot delete or insert without shifting all following data, which would imply quite a lot of rewriting. Furthermore, while you were doing that shifting, the contents of the file would be unpredictable and processes which had the file open would suffer. In order to get away with this (as with database systems, for example), you need a sophisticated set of modification protocols and distributed locks; stuff which is well beyond the scope of a typical file editing utility.

So, if you want to edit a file while its being processed by a shell, you have two options:

You can append to the file. This should always work.
You can overwrite the file with new contents of exactly the same length. This may or may not work, depending on whether the shell has already read that part of the file or not. Since most file I/O involves read buffers, and since all the shells I know read an entire compound command before executing it, it is pretty unlikely that you can get away with this. It certainly wouldn't be reliable.

I don't know of any wording in the Posix standard which actually requires the possibility of appending to a script file while the file is being executed, so it might not work with every Posix compliant shell, much less with the current offering of almost- and sometimes-posix-compliant shells. So YMMV. But as far as I know, it does work reliably with bash.

As evidence, here's a "loop-free" implementation of the infamous 99 bottles of beer program in bash, which uses dd to overwrite and append (the overwriting is presumably safe because it substitutes the currently executing line, which is always the last line of the file, with a comment of exactly the same length; I did that so that the end result can be executed without the self-modifying behaviour.)

#!/bin/bash
if [[ $1 == reset ]]; then
  printf "%s\n%-16s#\n" '####' 'next ${1:-99}' |
  dd if=/dev/stdin of=$0 seek=$(grep -bom1 ^#### $0 | cut -f1 -d:) bs=1 2>/dev/null
  exit
fi

step() {
  s=s
  one=one
  case $beer in
    2) beer=1; unset s;;
    1) beer="No more"; one=it;;
    "No more") beer=99; return 1;;
    *) ((--beer));;
  esac
}
next() {
  step ${beer:=$(($1+1))}
  refrain |
  dd if=/dev/stdin of=$0 seek=$(grep -bom1 ^next\  $0 | cut -f1 -d:) bs=1 conv=notrunc 2>/dev/null
}
refrain() {
  printf "%-17s\n" "# $beer bottles"
  echo echo ${beer:-No more} bottle$s of beer on the wall, ${beer:-No more} bottle$s of beer.
  if step; then
    echo echo Take $one down, pass it around, $beer bottle$s of beer on the wall.
    echo echo
    echo next abcdefghijkl
  else
    echo echo Go to the store, buy some more, $beer bottle$s of beer on the wall.
  fi
}
####
next ${1:-99}   #

bash – Parallel Execution of Bash Script

You can try GNU parallel. Many examples can be found here and here.

I am a member of the Swift team and we do exactly these kinds of things for large research projects.

Best Answer

Related Solutions

Linux – What Happens If You Edit a Script During Execution?

bash – Parallel Execution of Bash Script

Related Question