Shell – POSIX compliant way to work with a list of filenames possibly with whitespace

filenamesposixquotingshell-scriptwhitespace

I have seen Bash scripting guides suggesting the use of array for working with filenames containing whitespace. DashAsBinSh however suggests that arrays are not portable so I am looking for a POSIX compliant way of working with lists of filenames that may contain whitespace.

I am looking to modify the below example script so that it would echo

foo/target/a.jar
foo/target/b.jar
bar/target/lol whitespace.jar

Here is the script

#!/usr/bin/env sh

INPUT="foo/target/a.jar
foo/target/b.jar
bar/target/b.jar
bar/target/lol whitespace.jar"
# this would be produced by a 'ls' command
# We can execute the ls within the script, if it helps

dostuffwith() { echo $1; };

F_LOCATIONS=$INPUT
ALL_FILES=$(for f in $F_LOCATIONS; do echo `basename $f`; done)
ALL_FILES=$(echo "$ALL_FILES" | sort | uniq)

for f in $ALL_FILES
do
    fpath=$(echo "$F_LOCATIONS" | grep -m1 $f)
    dostuffwith $fpath
done

Best Answer

POSIX shells have one array: the positional parameters ($1, $2, etc., collectively refered to as "$@").

set -- 'foo/target/a.jar' 'foo/target/b.jar' 'bar/target/b.jar' 'bar/target/lol whitespace.jar'
set -- "$@" '/another/one at the end.jar'
…
for jar do
  dostuffwith "$jar"
done

This is inconvenient because there's only one, and it destroys any other use of the positional parameters. Positional parameters are local to a function, which is sometimes a blessing and sometimes a curse.

If your file names are guaranteed not to contain newlines, you can use newlines as the separator. When you expand the variable, first turn off globbing with set -f and set the list of field splitting characters IFS to contain only a newline.

INPUT="foo/target/a.jar
foo/target/b.jar
bar/target/b.jar
bar/target/lol whitespace.jar"
…
set -f; IFS='
'                           # turn off variable value expansion except for splitting at newlines
for jar in $INPUT; do
  set +f; unset IFS
  dostuffwith "$jar"        # restore globbing and field splitting at all whitespace
done
set +f; unset IFS           # do it again in case $INPUT was empty

With items in your list separated by newlines, you can use many text processing commands usefully, in particular sort.

Remember to always put double quotes around variable substitutions, except when you explicitly want field splitting to happen (as well as globbing, unless you've turned that off).

Related Question