SSH Shell Script – Why SSH Causes while Loop to Stop

inputio-redirectionshell-scriptssh

I have finally managed to boil down a problem I have been struggling with for a few weeks. I use SSH with "authorized keys" to run commands remotely. All is fine except when I do it in a while loop. The loop terminates after completing any iteration with an ssh command.

For a long time I thought this was some kind of ksh weirdness, but I now discovered bash does in fact behave identically.

A small sample program to reproduce the problem. This is distilled from a larger implementation which takes snapshots and replicates them amongst the nodes in a cluster.

#!/bin/bash

set -x

IDTAG=".*zone"
MARKER="mark-$(date +%Y.%m.%d.%H.%M.%S)"
REMOTE_HOST=sol10-target
ZFSPARENT=rpool

ssh $REMOTE_HOST zfs list -t filesystem -rHo name,mounted $ZFSPARENT | grep "/$IDTAG    " > /tmp/actionlist

#for RMT_FILESYSTEM in $(cat /tmp/actionlist)
cat /tmp/actionlist | while read RMT_FILESYSTEM ISMOUNTED
do
   echo ${RMT_FILESYSTEM}@${MARKER}
   [ "$ISMOUNTED" = "yes" ] && ssh $REMOTE_HOST zfs snapshot -r ${RMT_FILESYSTEM}@${MARKER}
   echo Remote Command Return Code: $?
done

(Note there is a TAB character in the grep search expression as per the definition of the behaviour of the zfs list "-H" option.)

My sample have some ZFS filesystems for the root where all the "zones" have their root file system on a dataset named similar to

POOL/zones/app1zone
POOL/zones/group2/app2zone

etc.

The above loop should create a snapshot for each of the selected datasets, but in stead it operates only on the first one and then exits.

That the program finds the right number of datasets can be easily confirm by checking the "/tmp/actionlist" file after the script exists.

If the ssh command is replaced by, for example, an echo command, then the loop iterates through all the input lines. Or my favourite – prepend "echo" to the offending command.

If I use a for loop in stead then it also works, but due to the potential size of the list of datasets this could cause problems with the maximum expanded command line length.

I am now 99.999% sure that only those loops with ssh commands in them give me problems!

Note that the iteration in which the ssh command runs, completes! It is as if the data pipped in to the while loop is suddenly lost… If the first few input lines don't perform an ssh command, then the loop goes on until it actually runs the SSH command.

On my laptop where I am testing this I have two Solaris 10 VMs with only about two or three sample datasets, but the same is happening on the large SPARC systems where this is meant to go live, and there are many datasets.

Best Answer

SSH might be reading from standard input, eating up your actionlist. Try to redirect ssh's standard input to /dev/null:

ssh $REMOTE_HOST zfs snapshot -r ${RMT_FILESYSTEM}@${MARKER} </dev/null

As a general rule, when running commands that may interfere with standard input under a while read-style loop, I like to wrap the whole loop body into braces:

cat /tmp/uuoc | while read RMT_FILESYSTEM ISMOUNTED
do {
    echo ${RMT_FILESYSTEM}@${MARKER}
    [ "$ISMOUNTED" = "yes" ] && ssh $REMOTE_HOST zfs snapshot -r ${RMT_FILESYSTEM}@${MARKER}
    echo Remote Command Return Code: $?
} < /dev/null; done

Related Solutions

Reading Lines from a File in Bash – for vs. while Loop Comparison

The for loop is fine here. But note that this is because the file contains machine names, which do not contain any whitespace characters or globbing characters. for x in $(cat file); do … does not work to iterate over the lines of file in general, because the shell first splits the output from the command cat file anywhere there is whitespace, and then treats each word as a glob pattern so \[?* are further expanded. You can make for x in $(cat file) safe if you work on it:

set -f
IFS='
'
for x in $(cat file); do …

Related reading: Looping through files with spaces in the names?; How can I read line by line from a variable in bash?; Why is while IFS= read used so often, instead of IFS=; while read..? Note that when using while read, the safe syntax to read lines is while IFS= read -r line; do ….

Now let's turn to what goes wrong with your while read attempt. The redirection from the server list file applies to the whole loop. So when ssh runs, its standard input comes from that file. The ssh client can't know when the remote application might want to read from its standard input. So as soon as the ssh client notices some input, it sends that input to the remote side. The ssh server there is then ready to feed that input to the remote command, should it want it. In your case, the remote command never reads any input, so the data ends up discarded, but the client side doesn't know anything about that. Your attempt with echo worked because echo never reads any input, it leaves its standard input alone.

There are a few ways you can avoid this. You can tell ssh not to read from standard input, with the -n option.

while read server; do
  ssh -n $server "uname -a"
done < /home/kenny/list_of_servers.txt

The -n option in fact tells ssh to redirect its input from /dev/null. You can do that at the shell level, and it'll work for any command.

while read server; do
  ssh $server "uname -a" </dev/null
done < /home/kenny/list_of_servers.txt

A tempting method to avoid ssh's input coming from the file is to put the redirection on the read command: while read server </home/kenny/list_of_servers.txt; do …. This will not work, because it causes the file to be opened again each time the read command is executed (so it would read the first line of the file over and over). The redirection needs to be on the whole while loop so that the file is opened once for the duration of the loop.

The general solution is to provide the input to the loop on a file descriptor other than standard input. The shell has constructs to ferry input and output from one descriptor number to another. Here, we open the file on file descriptor 3, and redirect the read command's standard input from file descriptor 3. The ssh client ignores open non-standard descriptors, so all is well.

while read server <&3; do
  ssh $server "uname -a"
done 3</home/kenny/list_of_servers.txt

In bash, the read command has a specific option to read from a different file descriptor, so you can write read -u3 server.

bash – Writing a For Loop That Iterates Over String Values in Bash Shell Script

The correct syntax is as follows:

#!/bin/bash

for fname in a.txt b.txt c.txt
do
  echo $fname
done

Best Answer

Related Solutions

Reading Lines from a File in Bash – for vs. while Loop Comparison

bash – Writing a For Loop That Iterates Over String Values in Bash Shell Script

Related Question