Why does a failed filename generation make zsh stop processing a script

zsh

I was trying to write a short script which would write all the executable programs found in $PATH:

for dir in $(tr ':' ' ' <<<"${PATH}"); do
  for pgm in $dir/*; do
    if command -v "${pgm}" >/dev/null 2>&1; then
      echo "${pgm}"
    fi
  done
done | sort >file

In bash, it works as expected, but zsh stops processing the script as soon as a filename generation fails in the inner loop:

for pgm in $dir/*; do
           ^^^^^^
  ...
done

As a result, since my $PATH contains a directory which doesn't contain any file (/usr/local/sbin), in zsh, the script fails to write the executables found in the directories afterwards.

Here's another code showing the same issue:

for f in /not_a_dir/*; do
  echo 'in the loop'
done
echo 'after the loop'

In bash, this command outputs:

in the loop
after the loop

And exits with the code 0.

While in zsh, the same command outputs:

no matches found: /not_a_dir/*

And exits with the code 1.

The difference of behavior between the shells seems to come from the nomatch option, which is described in man zshoptions:

NOMATCH (+3) <C> <Z>

If a pattern for filename generation has no matches, print an error, instead of leaving it unchanged
in the argument list. This also applies to file expansion of an initial ~ or =.

And also explained in man zshexpn (section FILENAME GENERATION):

The word is replaced with a list of sorted filenames that match the pattern. If no matching pattern is found, the shell gives an error message, unless the NULL_GLOB option is set, in which case the word is deleted; or unless the NOMATCH option is unset, in which case the word is left unchanged.

Because if I unset nomatch, zsh behaves like bash:

unsetopt nomatch
for f in /not_a_dir/*; do
  echo 'in the loop'
done
echo 'after the loop'

Now I understand the difference of behaviors between bash and zsh, and why the script raises an error in zsh, but I want to understand why a failed filename generation makes zsh immediately stop processing a script.
So, I tried to reproduce the same issue by replacing the failed filename generation with a failed command (by executing not_a_cmd):

for f in ~/*; do
  not_a_cmd
done
echo 'after the loop'

But the output of this script is almost identical in both shells (apart from the error messages due to not_a_cmd). In particular, both shells print:

after the loop

And both shells exit with the code 0.

Why does a failed filename generation (like for f in /not_a_dir/*) make zsh stop processing a script, but not a failed command (like not_a_cmd)?

I'm using zsh 5.6.2-dev-0 (x86_64-pc-linux-gnu).

Best Answer

It's a feature, not a bug. ?

As it says in Zsh's manual, section "Errors":

Certain errors are treated as fatal by the shell: in an interactive shell, they cause control to return to the command line, and in a non-interactive shell they cause the shell to be aborted. […]

Fatal errors found in non-interactive shells include:

[…]

File generation errors where not caught by the option BAD_PATTERN

[…]

File generation failures where not caused by NO_MATCH or similar options.

As for the reasoning behind it, I found this email exchange in the Zsh mailing list archives:

In other shells this is implemented in a very simple way, if the wildcard can be expanded, then it is expanded, otherwise an asterisk is sent as an argument to the application.

Yes, other shells do it that way, but that has its downsides. For instance, if you run "touch *,c" it's better to get "no match" rather than creating a useless *,c file. Or if you type a command like this: "rsync -avP *.html *.shmtl *.php host:/dest/" (note the misspelling), it is better to get a "no match" error right away than to get a single open-error embedded in all the output that you then have to figure out at the end of the transfer. I much prefer to simply put single quotes around any wild-carded file names that should not be expanded rather than to configure the shell to include a wildcard that didn't match anything. YMMV, of course, but if you give it a try, you may find that you come to prefer it as well.

Related Solutions

Bash – Compatibility scripting: Save $? for use later

This answer works in bash-like shells and csh-like shells, and it doesn’t require perl:

sh -c '<< other code >>; [ "$0" != "" ] && exit "$0"; exit "$1"' "$?" "$status"

or you can add a dummy/pad parameter:

sh -c '<< other code >>; [ "$1" != "" ] && exit "$1"; exit "$2"' foo "$?" "$status"

and the test can be changed to [ ! -z "$0" ] (or "$1"). I don’t know whether it will work in zsh or fish. Every shell that I’m familiar with will evaluate $? and $status when the command is parsed, and pass them to the sh -c command as parameters. The sh shell assigns them to the last two positional parameters. When the “other code” finishes, the shell will check the positional parameter that got its value from $? to see whether it is null or non-null, and exit with the appropriate positional parameter value.

This will fail if the ambient shell uses something other than $? or $status to hold the exit status. It will also fail if the shell allows $? to be set (non-null) if it is not the exit status, or if $? is an invalid character sequence that causes the entire command to be rejected as a syntax error.

Bash – way to make the local command work portably to dash ksh bash and zsh

Note ksh88 and all its clones do dynamic scoping and have been supporting local for it since at least 1990 for ksh88 and 1994 for pdksh (and 1989 for bash which (now) implements a lot of the ksh88 API).

The ksh you're referring to is ksh93, a newer implementation from scratch by David Korn with a slightly different and incompatible API.

It's a good idea to refer to that shell as ksh93 instead of just ksh as ksh alone has meant the ksh88 API for a few decades (the one all but the ksh93 implementations implement). ksh93 has not been in wide use until a few years after 2000 when its code was released as open source.

ksh93's typeset only does static scoping. POSIX had objected to specifying ksh88's typeset/local on the ground that it was dynamic scoping (though, even though most languages like C do static scoping, dynamic scoping comes more naturally in shells as it's what you get in effect with subshells or with the environment), which explains why ksh93 was rewritten to do static coping.

Later, most other shells implemented scoping a la ksh88, so ksh93 is now the odd one out. And now ironically, POSIX's only reasonable option would be to specify a dynamic scope. While David Korn initially refused to implement dynamic scoping in ksh93, he had said he could consider it with the local keyword/builtin on the POSIX mailing list, but he has retired before that fully happened.

The ksh93v- beta and final version from AT&T can be compiled with an experimental "bash" mode (actually enabled by default) that does dynamic scoping (in bother forms of functions, including with local and typeset) when ksh93 is invoked as bash. That bash mode will be disabled by default in ksh2020 though the local/declare aliases to typeset will be retained even when the bash mode is not compiled in (though still with static scoping).

Now if we leave that beta release aside and its bash mode, ksh93 only does static scoping and only in the functions declared with the ksh-style syntax (function name { code; }). You're getting confused because the functions declared with the Bourne-style syntax (f() command) don't do scoping at all. It's the same with sourced files or ksh functions invoked with . name [args]. In those, typeset doesn't declare a new variable in the function's scope (that function has no scope), it just updates the type of the variable in the current scope which is going to be either the global scope or the scope of a ksh-style function if that Bourne-style was (eventually) called from within a ksh-style function.

The code of Bourne-style functions is run as if embedded/copy-pasted/sourced wherever they're invoked.

var=global
function ksh_function {
  typeset var=private
  echo "ksh1: $var"
  bourne_function
  echo "ksh2: $var"
  other_ksh_function other
  echo "ksh3: $var"
  . other_ksh_function other_invoked_with_dot
  echo "ksh4: $var"
}
bourne_function() {
  typeset var=set-from-bourne-function
}
function other_ksh_function {
  echo "other: $var"
  var=$1
}
ksh_function
echo "global: $var"

Gives:

ksh1: private
ksh2: set-from-bourne-function
other: global
ksh3: set-from-bourne-function
other: set-from-bourne-function
ksh4: other_invoked_with_dot
global: other

It's not possible in ksh93 to have dynamic scoping other than by using subshells or implementing a variable stack by yourself like in that locvar proof of concept or you export the variable for it to be passed to every command (including functions declared with the ksh-style functions, including external commands via the environment).

In your particular case where only the testlocal function (and not testdescend) need a local scope, you can use the shdef+kshdef approach as described there or do something like:

case $KSH_VERSION in
  (*" 93"*)
    fn_with_local_scope() {
      alias local=typeset
      eval "function $1 {
        $(cat)
      }"
    }
  ;;
  (*)
    fn_with_local_scope() {
      eval "$1() {
        $(cat)
      }"
    }
  ;;
esac

And then declare your functions as:

fn_with_local_scope testlocal << '}'
  local IFS
  IFS=123
  echo "internal IFS = $IFS"
  testdescend
}

testdescend(){
  echo "descended IFS = $IFS"
}

IFS=abc
testlocal
echo "external IFS = $IFS"

(and only use local in functions declared with fn_with_local_scope).

Which gives

internal IFS = 123
descended IFS = 123
external IFS = abc

in all shells (note that you need a recent version of yash (2.48 or above) for it to support local).

Or if you're OK for the local variables to also be exported (in ksh93 only):

case $KSH_VERSION in
  (*" 93"*)
    fn() {
      alias local='typeset -x'
      eval "function $1 {
        $(cat)
      }"
    }
  ;;
  (*)
    fn() {
      eval "$1() {
        $(cat)
      }"
    }
  ;;
esac

fn testlocal << '}'
  local IFS
  IFS=123
  echo "internal IFS = $IFS"
  testdescend
}

fn testdescend << '}'
  echo "descended IFS = $IFS"
}

IFS=abc
testlocal
echo "external IFS = $IFS"

Now, if you were to do something like that in a language with static scoping like C or ksh93, you'd do something like:

function testlocal {
  typeset IFS
  IFS=123
  echo "internal IFS = $IFS"
  testdescend "$IFS"
}

function testdescend {
  typeset IFS="$1" # explicitly get the value $IFS from the caller
  echo "descended IFS = $IFS"
}

Which seems to me like a better design, and that code would work OK as well in shells that do dynamic scoping (you'd still need to address the different function definition syntax).

Best Answer

Related Solutions

Bash – Compatibility scripting: Save $? for use later

Bash – way to make the local command work portably to dash ksh bash and zsh

Related Question