Bash – the rationale behind $array not expanding the whole array in ksh and bash

arraybashksh

bash$ a=(1 2 3)
bash$ echo $a
1

but

zsh% a=(1 2 3)
zsh% echo $a
1 2 3
zsh% printf '%s\n' $a
1
2
3

(the last part demonstrates that the array expands into separate arguments, equivalent to "${a[@]}" and not "${a[*]}")

The bash behavior (which matches ksh) is extremely counter-intuitive. How is "first element only" a reasonable reaction to "expand this array variable"?

In other areas where zsh is divergent, it's because ksh and bash are sticking closer to the original Bourne shell. But Bourne had no user-defined array variables.

Why did bash make this strange decision? If it was copying ksh, why did ksh make this strange decision?

Continuing after a long string of comments:

This should not be a question of criticizing or praising zsh. zsh is nothing but an easily accessible example of how things might have been done differently.

One of the possibilities to explain a design decision is backward compatibility. And backward compatibility isn't an opinion. It's an objective fact.

If you can show a script (a full script, not an out-of-context excerpt) that behaves in a known way in Bourne shell (i.e. does not just bomb with a syntax error), and behaves different in the hypothetical "Korn shell with full $array expansion", then you win! It's a backward compatibility issue.

No such script has been given. This isn't one:

a=(1 2 3)
printf '%s\n' $a

because it's a syntax error in Bourne shell. Giving new meaning to something that used to be a syntax error is a way to create new features while keeping backward compatibility.

As far as I can tell, the fact that a=(...) was originally a syntax error creates a clean separation between scripts that (attempt to) use arrays and those that don't. In the first category, backward compatibility can't be invoked as a reason for anything, because those scripts wouldn't run in the old shell anyway. In the second category, backward compatibility holds regardless of your array variable expansion rules, because there are no arrays to expand!

This is not a proof since I'm relying partially on intuition to decide that there's no way to smuggle an array into a script without =( and therefore no script exists that would exhibit incompatible behavior. The nice thing about a claim of nonexistence is that you only have to show one counterexample to end it.

The a=$@ thing that was brought up in the comments does look like it could contribute to an explanation. If it creates an array variable a then this script:

a=$@
printf '%s\n' $a

should show the difference. In my tests, though, that doesn't happen. All shells (heirloom sh, modern ksh, bash, and zsh) seem to handle the first line the same way. a is not an array, it's just a string with spaces in it. (zsh diverges on the second line because it doesn't do word-splitting on the value of $a, but that has nothing to do with array variables)

Best Answer

I can't give an answer but suggest some possible explanations.

It's true that except for ksh and its clones (pdksh and further derivatives and bash), all other shells with arrays (csh, tcsh, rc, es, akanga, fish, zsh, yash) have $array expand to all the members of the array.

But in both yash and zsh (when in sh emulation), the two Bourne-like shells in that list, that expansion is still subject to split+glob (and still empty removal even in zsh even when not in sh emulation), so you still need to use the awkward "${array[@]}" syntax (or "${(@)array}" or "$array[@]" in zsh which are hardly easier to type) to preserve the list (csh and tcsh have similar issues). That split+glob and empty removal is the Bourne heritage (itself to some extent caused by the Thompson shell heritage where $1 was more like macro expansion).

rc and fish are two examples of later shells that don't have the Bourne baggage and with a cleaner approach. They acknowledge the fact that a shell being a command line interpreter, the primary things they deal with is lists (the list of arguments to commands), so list/array is the primary data type (there's only one type and it's lists in rc) and got rid of the split+glob-upon-expansion bug/misfeature of the Bourne shell (which is no longer needed now that the primary type is array).

Still, that doesn't explain why David Korn chose not to have $array expand to all elements but to the element of index 0.

Now, apart from csh/tcsh, all those shells are much newer than ksh, developed in the early 80s only a few years after the Bourne shell and Unix V7 were released. Unix V7 was the one that also introduced the environ. That was the fancy new thing at time. The environment is neat and useful but environment variables can't contain arrays unless you use some form of encoding.

That's only conjecture but I suspect one reason for David Korn to choose that approach was so that the interface with the environment was not modified.

In ksh88, like rc, all variables were arrays (sparse though; a bit like associative arrays with keys limited to positive integers which is another oddity compared to other shells or programming languages, and you could tell it hadn't been completely thought through as it was impossible for instance to retrieve the list of keys). In that new design, var=value became short for var[0]=value. You could still export all your variables, but export var exports the element of index 0 of the array to the environment.

rc does put all its variables in the environment, fish supports exporting arrays, but to do that for arrays with more than one element, (at least for the port to Unix for rc which comes from plan9), they have to resort to some form of encoding which is only understood by them.

csh, tcsh, zsh don't support exporting arrays (though nowadays that may not sound like a big limitation). You can export arrays in yash, but they're exported as an environment variable that's the array elements joined with : (so (a "" "" b) and (a : b) are exported to the same value) and there's no converting back to array on importing.

Another possible justification might have been the consistency with Bourne's $@/$* (but then why have array indices start at 0 instead of 1 (another oddity compared to other shells/languages of the time)?). ksh was not free software, it was a commercial enterprise, one of the requirements was Bourne compatibility. ksh did remove the field splitting done on every non-quoted word in list context (as that was clearly not useful in the Bourne shell) but had to keep it for expansions (as scripts did use things like var="file1 file2"; cmd $var as the Bourne shell had no array but "$@"). Keeping that in a shell that otherwise has arrays makes little sense, but Korn had little other option if Ksh was to still be able to interpret scripts of the consumer base. If $scalar was subject to split+glob, $array would have to be for consistency, and so "${array[@]}" as a generalisation of "$@" made some sense. zsh had no similar constraint so was free to remove the split+glob upon expansions at the same time as adding arrays (but paid a price for breaking Bourne backward compatible).

Another explanation as offered by @Arrow might have been that he didn't want to overload the existing operators to make them behave differently for different types of variables (for instance ${#var} vs ${#array} though the Bourne shell didn't have that one or ${var-value}, ${var#pattern}) which can cause confusion for users (in zsh it's not always obvious how some operators work with array vs scalar).

Some related reading:

https://www.usenix.org/legacy/publications/library/proceedings/vhll/full_papers/korn.ksh.a where David Korn explains the development of ksh (arrays first added as patches over the Bourne shell for a form entry system).
https://news.slashdot.org/story/01/02/06/2030205/david-korn-tells-all

As to the a=$@ case in your edit, that's actually one case where ksh broke compatibility with the Bourne shell.

In the Bourne shell, $@ and $* contained the concatenation of the positional parameters with space characters. Only $@ when quoted was special as it expanded to the same as "$*" but with the inserted spaces not quoted (with special cases for the empty list in the newer versions where it has been addressed like on Solaris). You'll notice that if you remove space from $IFS, "$@" expands to just one argument in list contexts (0 for an empty list in the fixed versions mentioned above). When not quoted, $* and $@ behave like any other variable (split upon characters of $IFS, not necessarily on the original positional parameters). For instance, in the Bourne shell:

'set' 'a:b'   'c'
IFS=:
printf '<%s>\n' $@
printf '[%s]\n' "$@"

Would output:

<a>
<b c>
[a:b c]

Ksh88 changed that so that $@ and $* were joined with the first character of $IFS. "$@" in list context separates the positional parameters except when $IFS is empty.

When $IFS is empty, $* are joined on space, except for $* when quoted which is joined on with no separator.

Examples:

$ set a b
$ IFS=:
$ a=$@ b=$* c="$@" d="$*"
$ printf '<%s>\n' "$a" "$b" "$c" "$d" $@ $* "$@" "$*"
<a:b>
<a:b>
<a:b>
<a:b>
<a>
<b>
<a>
<b>
<a>
<b>
<a:b>
$ IFS=
$ a=$@ b=$* c="$@" d="$*"
$ printf '<%s>\n' "$a" "$b" "$c" "$d" $@ $* "$@" "$*"
<a b>
<a b>
<a b>
<ab>
<a b>
<a b>
<a b>
<ab>

You'll see a lot of variations in the different Bourne/Korn-like shells including ksh93 vs ksh88. There are also some variations in cases like:

set --
cmd ''"$@"
cmd $empty"$@"

Or when $IFS contains multi-byte characters, or bytes not forming valid characters.

Related Solutions

BASH reading txt file and storing in array

IFS=$'\n' a=($(cat phonebook.txt))
for i in $(seq ${#a[*]}); do
    [[ ${a[$i-1]} = $name ]] && echo "${a[$i]}"
done

In Bash 4 IFS=$'\n' a=($(cat phonebook.txt)) can be replaced with mapfile -t a < phonebook.txt.

grep -A1 prints one line after the match. -x disables regex like -F but it only matches complete lines.

grep -x "$name" -A1 phonebook.txt | tail -n1

Shell – Arrays in Unix Bourne Shell

/bin/sh is hardly ever a Bourne shell on any systems nowadays (even Solaris which was one of the last major system to include it has now switched to a POSIX sh for its /bin/sh in Solaris 11). /bin/sh was the Thompson shell in the early 70s. The Bourne shell replaced it in Unix V7 in 1979.

/bin/sh has been the Bourne shell for many years thereafter (or the Almquist shell, a free reimplementation on BSDs).

Nowadays, /bin/sh is more commonly an interpreter or another for the POSIX sh language which is itself based on a subset of the language of ksh88 (and a superset of the Bourne shell language with some incompatibilities).

The Bourne shell or the POSIX sh language specification don't support arrays. Or rather they have only one array: the positional parameters ($1, $2, $@, so one array per function as well).

ksh88 did have arrays which you set with set -A, but that didn't get specified in the POSIX sh as the syntax is awkward and not very usable.

Other shells with array/lists variables include: csh/tcsh, rc, es, bash (which mostly copied the ksh syntax the ksh93 way), yash, zsh, fish each with a different syntax (rc the shell of the once to-be successor of Unix, fish and zsh being the most consistent ones)...

In standard sh (also works in modern versions of the Bourne shell):

set '1st element' 2 3 # setting the array

set -- "$@" more # adding elements to the end of the array

shift 2 # removing elements (here 2) from the beginning of the array

printf '<%s>\n' "$@" # passing all the elements of the $@ array 
                     # as arguments to a command

for i do # looping over the  elements of the $@ array ($1, $2...)
  printf 'Looping over "%s"\n' "$i"
done

printf '%s\n' "$1" # accessing individual element of the array.
                   # up to the 9th only with the Bourne shell though
                   # (only the Bourne shell), and note that you need
                   # the braces (as in "${10}") past the 9th in other
                   # shells (except zsh, when not in sh emulation and
                   # most ash-based shells).

printf '%s\n' "$# elements in the array"

printf '%s\n' "$*" # join the elements of the array with the 
                   # first character (byte in some implementations)
                   # of $IFS (not in the Bourne shell where it's on
                   # space instead regardless of the value of $IFS)

(note that in the Bourne shell and ksh88, $IFS must contain the space character for "$@" to work properly (a bug), and in the Bourne shell, you can't access elements above $9 (${10} won't work, you can still do shift 1; echo "$9" or loop over them)).

Best Answer

Related Solutions

BASH reading txt file and storing in array

Shell – Arrays in Unix Bourne Shell

Related Question