Bash – Difference Between Double-Quoting and Not Double-Quoting an Array

arraybashbrace-expansionshellshellcheck

While tracking down an error in my shellscript, I found the following behavior in this code snippet:

declare -a filelist
readarray filelist < <(ls -A)
readonly filelist
for file in "${filelist[@]}"; do
  sha256sum ${filelist[$file]} | head -c 64
done

When the array filelist is not in double quotes, the command succeeds. I've been using ShellCheck to try to improve my coding, which recommends-

Double quote to prevent globbing and word splitting.

I'm not worried about word splitting in this case, but in a lot of other cases I am, so I'm trying to keep my code consistent. However, when I double quote the array, the command fails. Simplifying the code to a single element gives the following:

bash-5.0# sha256sum ${filelist[0]} | head -c 64
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

bash-5.0# sha256sum "${filelist[0]}" | head -c 64
sha256sum: can't open 'file1
': No such file or directory

I can obviously just… not double quote because in this instance word splitting isn't a concern. But I wanted to post because in the future it might be.

My question has two parts:

  1. Is there a "best-practices" way to prevent word splitting other than double quoting the array as above?
  2. Where are the single quotes coming from in the array? Edit: there are no single quotes. The single quotes are the error showing the name of the file that cannot be opened.

Also, just out of curiosity, why does echo ${filelist[0]} not contain an additional newline but echo "${filelist[0]}" does?

Best Answer

There is absolutely no problem with quoting an array expansion.

And, of course, there is no problem with no quoting it either as long as you know and accept the consequences. Any non-quoted expansion is subject to splitting and globbing. And, in your code, the ${filelist[…]} is subject to IFS character removal (and splitting if the string contains any <space>, <tab>, or <newline>).

That is what having the expansion un-quoted do, remove trailing <newline>.

What creates this problem is that you are using readarray without removing the trailing delimiter from each array element.
Doing that keeps a trailing <newline> that is reflected on the error message.

What you could have used is:

readarray -t filelist < <(ls -A)

The -t option will remove all the trailing newlines of each file name.

-t Remove a trailing delim (default newline) from each line read.

But your code has some additional issues.

  • There is no need to declare or empty the array filelist. It gets done by default by readarray. It needs to be done in some other cases.

  • There is no need to parse the output of ls, in fact, that is a bad idea. The easiest way to get a list of files in an array is simply:

    filelist=( ./* )

    And, to make it even better, it would be a good idea to avoid directories:

    for file in ./*; do
  [[ -f $file ]] && filelist+=( "$file" )
done

  • In the loop, the value of the var $file is what should be used:

    for file in "${filelist[@]}"; do
  sha256sum "$file" | head -c 64
done

    Unless you use for file in "${!filelist[@]}"; do which will list the keys of the array.

  • The whole list could be processed with just one call to sha256sum:

    sha256sum "${filelist[@]}" | cut -c -64

The improved script is:

filelist=()              # declare filelist as an array and empty it.
for file in ./*; do
    if [[ -f $file ]]; then
        filelist+=( "$file" )
    fi
done
declare -r filelist      # declare filelist as readonly.
sha256sum "${filelist[@]}" | cut -c -64
Related Question