Bash while loop read from colon-delimited list of paths using IFS

bashshellstring

I am trying to write a bash function that behaves similarly to the where builtin in tcsh. In tcsh, where lists all the builtins, aliases, and the absolute paths to executables on the PATH with a given name, even if they are shadowed, e.g.

tcsh> where tcsh
/usr/bin/tcsh
/bin/tcsh

As part of this I want to loop over everything in the $PATH and see if an executable file with the appropriate name exists.

The following bash snippet is intended to loop over a colon-delimited list of paths and print each component followed by a newline, however, it just seems to print the entire contents of $PATH all on one line

#!/bin/bash
while IFS=':' read -r line; do
    printf "%s\n" "$line"
done <<< "$PATH"

As is stands now, bash where and ./where just print /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games

So, how do I set up my while loop so that the value of the loop variable is each segment of the colon-separated list of paths in turn?

Best Answer

read uses IFS to separate the words in the line it reads, it doesn't tell read to read until the first occurrence of any of the characters in it.

IFS=: read -r a b

Would read one line, put the part before the first : in $a, and the rest in $b.

IFS=: read -r a

would put the whole line (the rest) in $a (except if that line contains only one : and it's the last character on the line).

If you wanted to read until the first :, you'd use read -d: instead (ksh93, zsh or bash only).

printf %s "$PATH" | while IFS= read -rd: dir || [ -n "$dir" ]; do
  ...
done

(we're not using <<< as that adds an extra newline character).

Or you could use standard word splitting:

IFS=:; set -o noglob
for dir in $PATH""; do
  ...
done

Now beware of few caveats:

An empty $PATH component means the current directory.
An empty $PATH means the current directory (that is, $PATH contains one component which is the current directory, so the while read -d: loop would be wrong in that case).
//file is not necessary the same as /file on some system, so if $PATH contains /, you need to be careful with things like $dir/$file.
An unset $PATH means a default search path is to be used, it's not the same as a set but empty $PATH.

Now, if it's only the equivalent of tcsh/zsh's where command, you could use bash's type -a.

Related Solutions

Reading Lines from a File in Bash – for vs. while Loop Comparison

The for loop is fine here. But note that this is because the file contains machine names, which do not contain any whitespace characters or globbing characters. for x in $(cat file); do … does not work to iterate over the lines of file in general, because the shell first splits the output from the command cat file anywhere there is whitespace, and then treats each word as a glob pattern so \[?* are further expanded. You can make for x in $(cat file) safe if you work on it:

set -f
IFS='
'
for x in $(cat file); do …

Related reading: Looping through files with spaces in the names?; How can I read line by line from a variable in bash?; Why is while IFS= read used so often, instead of IFS=; while read..? Note that when using while read, the safe syntax to read lines is while IFS= read -r line; do ….

Now let's turn to what goes wrong with your while read attempt. The redirection from the server list file applies to the whole loop. So when ssh runs, its standard input comes from that file. The ssh client can't know when the remote application might want to read from its standard input. So as soon as the ssh client notices some input, it sends that input to the remote side. The ssh server there is then ready to feed that input to the remote command, should it want it. In your case, the remote command never reads any input, so the data ends up discarded, but the client side doesn't know anything about that. Your attempt with echo worked because echo never reads any input, it leaves its standard input alone.

There are a few ways you can avoid this. You can tell ssh not to read from standard input, with the -n option.

while read server; do
  ssh -n $server "uname -a"
done < /home/kenny/list_of_servers.txt

The -n option in fact tells ssh to redirect its input from /dev/null. You can do that at the shell level, and it'll work for any command.

while read server; do
  ssh $server "uname -a" </dev/null
done < /home/kenny/list_of_servers.txt

A tempting method to avoid ssh's input coming from the file is to put the redirection on the read command: while read server </home/kenny/list_of_servers.txt; do …. This will not work, because it causes the file to be opened again each time the read command is executed (so it would read the first line of the file over and over). The redirection needs to be on the whole while loop so that the file is opened once for the duration of the loop.

The general solution is to provide the input to the loop on a file descriptor other than standard input. The shell has constructs to ferry input and output from one descriptor number to another. Here, we open the file on file descriptor 3, and redirect the read command's standard input from file descriptor 3. The ssh client ignores open non-standard descriptors, so all is well.

while read server <&3; do
  ssh $server "uname -a"
done 3</home/kenny/list_of_servers.txt

In bash, the read command has a specific option to read from a different file descriptor, so you can write read -u3 server.

Bash – How to prevent the caller’s shell from being used in sudo

Then answer is that sudo has a bug. First, the workaround: I put this in my /etc/sudoers.d/zabbix file:

zabbix    ALL=(root) NOPASSWD:       /bin/env SHELL=/bin/sh /usr/local/bin/zabbix_raid_discovery

and now subcommands called from zabbix_raid_discovery work.

A patch to fix this will be in sudo 1.8.15. From the maintainer, Todd Miller:

This is just a case of "it's always been like that".  There's not
really a good reason for it.  The diff below should make the behavior
match the documentation.

 - todd

diff -r adb927ad5e86 plugins/sudoers/env.c
--- a/plugins/sudoers/env.c     Tue Oct 06 09:33:27 2015 -0600
+++ b/plugins/sudoers/env.c     Tue Oct 06 10:04:03 2015 -0600
@@ -939,8 +939,6 @@
            CHECK_SETENV2("USERNAME", runas_pw->pw_name,
                ISSET(didvar, DID_USERNAME), true);
        } else {
-           if (!ISSET(didvar, DID_SHELL))
-               CHECK_SETENV2("SHELL", sudo_user.pw->pw_shell, false, true);
            /* We will set LOGNAME later in the def_set_logname case. */
            if (!def_set_logname) {
                if (!ISSET(didvar, DID_LOGNAME))
@@ -984,6 +982,8 @@
            if (!env_should_delete(*ep)) {
                if (strncmp(*ep, "SUDO_PS1=", 9) == 0)
                    ps1 = *ep + 5;
+               else if (strncmp(*ep, "SHELL=", 6) == 0)
+                   SET(didvar, DID_SHELL);
                else if (strncmp(*ep, "PATH=", 5) == 0)
                    SET(didvar, DID_PATH);
                else if (strncmp(*ep, "TERM=", 5) == 0)
@@ -1039,7 +1039,9 @@
     if (reset_home)
        CHECK_SETENV2("HOME", runas_pw->pw_dir, true, true);

-    /* Provide default values for $TERM and $PATH if they are not set. */
+    /* Provide default values for $SHELL, $TERM and $PATH if not set. */
+    if (!ISSET(didvar, DID_SHELL))
+       CHECK_SETENV2("SHELL", runas_pw->pw_shell, false, false);
     if (!ISSET(didvar, DID_TERM))
        CHECK_PUTENV("TERM=unknown", false, false);
     if (!ISSET(didvar, DID_PATH))

Best Answer

Related Solutions

Reading Lines from a File in Bash – for vs. while Loop Comparison

Bash – How to prevent the caller’s shell from being used in sudo

Related Question