Shell – How to Generate Arguments to Another Command via Command Substitution

argumentscommand-substitutionshell

Following on from:
unexpected behaviour in shell command substitution

I have a command which can take a huge list of arguments, some of which can legitimately contain spaces (and probably other things)

I wrote a script which can generate those arguments for me, with quotes, but I must copy and paste the output e.g.

./somecommand
<output on stdout with quoting>
./othercommand some_args <output from above>

I tried to streamline this by simply doing

./othercommand $(./somecommand)

and ran into the unexpected behaviour mentioned in question above. The question is — can command substitution be reliably used to generate the arguments to othercommand given that some arguments require quoting and this cannot be changed?

Best Answer

I wrote a script which can generate those arguments for me, with quotes

If the output is properly quoted for the shell, and you trust the output, then you could run eval on it.

Assuming you have a shell that supports arrays, it would be best to use one to store the arguments you get.

If ./gen_args.sh produces output like 'foo bar' '*' asdf, then we could run eval "args=( $(./gen_args.sh) )" to populate an array called args with the results. That would be the three elements foo bar, *, asdf.

We can use "${args[@]}" as usual to expand the array elements individually:

$ eval "args=( $(./gen_args.sh) )"
$ for var in "${args[@]}"; do printf ":%s:\n" "$var"; done
:foo bar:
:*:
:asdf:

(Note the quotes. "${array[@]}" expands to all elements as distinct arguments unmodified. Without quotes the array elements are subject to word splitting. See e.g. the Arrays page on BashGuide.)

However, eval will happily run any shell substitutions, so $HOME in the output would expand to your home directory, and a command substitution would actually run a command in the shell running eval. An output of "$(date >&2)" would create a single empty array element and print the current date on stdout. This is a concern if gen_args.sh gets the data from some untrusted source, like another host over the network, file names created by other users. The output could include arbitrary commands. (If get_args.sh itself was malicious, it wouldn't need to output anything, it could just run the malicious commands directly.)

An alternative to shell quoting, which is hard to parse without eval, would be to use some other character as separator in the output of your script. You'd need to pick one that is not needed in the actual arguments.

Let's choose #, and have the script output foo bar#*#asdf. Now we can use unquoted command expansion to split the output of the command to the arguments.

$ IFS='#'                          # split on '#' signs
$ set -f                           # disable globbing
$ args=( $( ./gen_args3.sh ) )     # assign the values to the array
$ for var in "${args[@]}"; do printf ":%s:\n" "$var"; done
:foo bar:
:*:
:asdf:

You'll need to set IFS back later if you depend on word splitting elsewhere in the script (unset IFS should work to make it the default), and also use set +f if you want to use globbing later.

If you're not using Bash or some other shell that has arrays, you could use the positional parameters for that. Replace args=( $(...) ) with set -- $(./gen_args.sh) and use "$@" instead of "${args[@]}" then. (Here, too, you need quotes around "$@", otherwise the positional parameters are subject to word splitting.)

What happens without quotes

Note that without double quotes, two things happen.

First, the result of the expansion (the value of the variable for a parameter substitution like ${foo}, or the output of the command for a command substitution like $(foo)) is split into words wherever it contains whitespace.
More precisely, the result of the expansion is split at each character that appears in the value of the IFS variable (separator character). If a sequence of separator characters contains whitespace (space, tab or newline), the whitespace is counts as a single character; leading, trailing or repeated non-whitespace separators lead to empty fields. For example, with IFS=" :", :one::two : three: :four produces empty fields before one, between one and two, and (a single one) between three and four.
Each field that results from splitting is interpreted as a glob (a wildcard pattern) if it contains one of the characters \[*?. If that pattern matches one or more file names, the pattern is replaced by the list of matching file names.

An unquoted variable expansion $foo is colloquially known as the “split+glob operator”, in contrast with "$foo" which just takes the value of the variable foo. The same goes for command substitution: "$(foo)" is a command substitution, $(foo) is a command substitution followed by split+glob.

Where you can omit the double quotes

Here are all the cases I can think of in a Bourne-style shell where you can write a variable or command substitution without double quotes, and the value is interpreted literally.

On the right-hand side of an assignment.
```
var=$stuff
a_single_star=*
```
Note that you do need the double quotes after export, because it's an ordinary builtin, not a keyword. This is only true in some shells such as dash, zsh (in sh emulation), yash or posh; bash and ksh both treat export specially.
```
export VAR="$stuff"
```
In a case statement.
```
case $var in …
```
Note that you do need double quotes in a case pattern. Word splitting doesn't happen in a case pattern, but an unquoted variable is interpreted as a pattern whereas a quoted variable is interpreted as a literal string.
```
a_star='a*'
case $var in
  "$a_star") echo "'$var' is the two characters a, *";;
   $a_star) echo "'$var' begins with a";;
esac
```
Within double brackets. Double brackets are shell special syntax.
```
[[ -e $filename ]]
```
Except that you do need double quotes where a pattern or regular expression is expected: on the right-hand side of = or == or != or =~.
```
a_star='a*'
if [[ $var == "$a_star" ]]; then echo "'$var' is the two characters a, *"
elif [[ $var == $a_star ]]; then echo "'$var' begins with a"
fi
```
You do need double quotes as usual within single brackets [ … ] because they are ordinary shell syntax (it's a command that happens to be called [). See Single or double brackets
In a redirection in non-interactive POSIX shells (not bash, nor ksh88).
```
echo "hello world" >$filename
```
Some shells, when interactive, do treat the value of the variable as a wildcard pattern. POSIX prohibits that behaviour in non-interactive shells, but a few shells including bash (except in POSIX mode) and ksh88 (including when found as the (supposedly) POSIX sh of some commercial Unices like Solaris) still do it there (bash does also attempt splitting and the redirection fails unless that split+globbing results in exactly one word), which is why it's better to quote targets of redirections in a sh script in case you want to convert it to a bash script some day, or run it on a system where sh is non-compliant on that point, or it may be sourced from interactive shells.
Inside an arithmetic expression. In fact, you need to leave the quotes out in order for a variable to be parsed as an arithmetic expression.
```
expr=2*2
echo "$(($expr))"
```
However, you do need the quotes around the arithmetic expansion as they are subject to word splitting in most shells as POSIX requires (!?).

In an associative array subscript.

typeset -A a
i='foo bar*qux'
a[foo\ bar\*qux]=hello
echo "${a[$i]}"

An unquoted variable and command substitution can be useful in some rare circumstances:

When the variable value or command output consists of a list of glob patterns and you want to expand these patterns to the list of matching files.
When you know that the value doesn't contain any wildcard character, that $IFS was not modified and you want to split it at whitespace characters.
When you want to split a value at a certain character: disable globbing with set -f, set IFS to the separator character (or leave it alone to split at whitespace), then do the expansion.

Zsh

In zsh, you can omit the double quotes most of the times, with a few exceptions.

$var never expands to multiple words, however it expands to the empty list (as opposed to a list containing a single, empty word) if the value of var is the empty string. Contrast:
```
var=
print -l $var foo        # prints just foo
print -l "$var" foo      # prints an empty line, then foo
```
Similarly, "${array[@]}" expands to all the elements of the array, while $array only expands to the non-empty elements.
The @ parameter expansion flag sometimes requires double quotes around the whole substitution: "${(@)foo}".
Command substitution undergoes field splitting if unquoted: echo $(echo 'a'; echo '*') prints a * (with a single space) whereas echo "$(echo 'a'; echo '*')" prints the unmodified two-line string. Use "$(somecommand)" to get the output of the command in a single word, sans final newlines. Use "${$(somecommand; echo _)%?}" to get the exact output of the command including final newlines. Use "${(@f)$(somecommand)}" to get an array of lines from the command's output.

Bash – How to pass files found by find as arguments

Use arrays.

If you don't need to handle the possibility of newlines in your filenames, then you could get away with

mapfile -t ABC_FILES < <(find -L some/dir -name \*.abc | sort)
mapfile -t XYZ_FILES < <(find -L other/dir -name \*.xyz | sort)

then

./program --abc-files "${ABC_FILES[@]}" --xyz-files "${XYZ_FILES[@]}"

If you do need to handle newlines within filenames, and have bash >= 4.4, you can use -print0 and -d '' to null-terminate the names during array construction:

mapfile -td '' ABC_FILES < <(find -L some/dir -name \*.abc -print0 | sort -z)

(and similarly for the XYZ_FILES). If you don't have the newer bash, then you could use a null-terminated read loop to append filenames to the arrays e.g.

ABC_FILES=()
while IFS= read -rd '' f; do ABC_FILES+=( "$f" ); done < <(find -L some/dir -name \*.abc -print0 | sort -z)

Best Answer

Related Solutions

When and Why Double-Quoting is Necessary in Shell Scripts

What happens without quotes

Where you can omit the double quotes

Zsh

Bash – How to pass files found by find as arguments

Related Question