Bash – Why do options in a quoted variable fail, but work when unquoted

When instructed to echo commands as they are executed ("execution trace"), both bash and ksh add single quotes around any word with meta-characters (*, ?, ;, etc.) in it.

The meta-characters could have gotten into the word in a variety of ways. The word (or part of it) could have been quoted with single or double quotes, the characters could have been escaped with a \, or they remained as the result of a failed filename matching attempt. In all cases, the execution trace will contain single-quoted words, for example:

$ set -x
$ echo foo\;bar
+ echo 'foo;bar'

This is just an artifact of the way the shells implement the execution trace; it doesn't alter the way the arguments are ultimately passed to the command. The quotes are added, printed, and discarded. Here is the relevant part of the bash source code, print_cmd.c:

/* A function to print the words of a simple command when set -x is on. */
void
xtrace_print_word_list (list, xtflags)
...
{
  ...
  for (w = list; w; w = w->next)
    {
      t = w->word->word;
      ...
      else if (sh_contains_shell_metas (t))
        {
          x = sh_single_quote (t);
          fprintf (xtrace_fp, "%s%s", x, w->next ? " " : "");
          free (x);
        }

As to why the authors chose to do this, the code there doesn't say. But here's some similar code in variables.c, and it comes with a comment:

/* Print the value cell of VAR, a shell variable.  Do not print
   the name, nor leading/trailing newline.  If QUOTE is non-zero,
   and the value contains shell metacharacters, quote the value
   in such a way that it can be read back in. */
void
print_var_value (var, quote)
...
{
  ...
  else if (quote && sh_contains_shell_metas (value_cell (var)))
    {
      t = sh_single_quote (value_cell (var));
      printf ("%s", t);
      free (t);
    }

So possibly it's done so that it's easier to copy the command lines from the output of the execution trace and run them again.

Short answer: see BashFAQ #50: I'm trying to put a command in a variable, but the complex cases always fail!.

Long answer: the shell does variable expansion partway through the process of parsing a command line -- notably, after it processes quotes and escapes. As a result, putting quotes and escapes in a variable doesn't do the same thing as having them directly on the command line.

The solution in your answer (doubling the escape characters) will work (in most cases), but not for the reason you think it's working, and that makes me rather nervous. The command:

cmd="freebcp ... -t "," -r "\\r\\n" -c"

Gets parsed into the double-quotesd string freebcp ... -t, followed by the unquoted string , followed by the double-quoted string -r, followed by the unquoted string '\\r\\n' (the fact that it's unquoted is why you needed to double the escapes), followed by the double-quoted string ' -c'. The double-quotes you meant to be part of the string aren't treated as part of the string, they're treated as delimiters that change how different parts of the string are parsed (and actually have pretty much the reverse of the intended effect). The reason this works is that the double-quotes actually weren't having much effect in the original command, so reversing their effect didn't do much. It would actually be better to remove them (just the internal ones, though), because it'd be less misleading about what's really going on. That'd work, but it'd be fragile -- the only reason it works is that you didn't really need the double-quotes to begin with, and if you had a situation (say, a password or filename with a space in it) where you actually needed quotes, you'd be in trouble.

There are several better options:

• Don't store the command in a variable at all, just execute it directly. Storing commands is tricky (as you're finding), and if you don't really need to, just don't.

• Use a function. If you're doing something like executing the same command over & over, define it as a function and use that:

  loaddb() {
      freebcp "${db_name}.dbo.${table_name}" in "${p_file}" -S "${server_name}" -U "${username}" -P "${password}" -t \",\" -r \"\r\n\" -c"
  }
  loaddb
  

  Note that I used double-quotes around all of the variable references -- this is generally good scripting hygiene, in case any of them contain whitespace, wildcards, or anything else that the shell does parse in variable values.

• Use an array instead of a plain variable. If you do this properly, each command argument gets stored as a separate element of the array, and you can then expand it with the idiom "${arrayname[@]}" to get it out intact:

  cmdarray=(freebcp "${db_name}.dbo.${table_name}" in "${p_file}" -S "${server_name}" -U "${username}" -P "${password}" -t \",\" -r \"\r\n\" -c")
  "${cmdarray[@]}"
  

  Again, note the prolific use of double-quotes; here they're being used to make sure the array elements are defined properly, not as part of the values stored in the array. Also, note that arrays aren't available in all shells; make sure you're using bash or zsh or something similar.

A couple of final notes: when you use something like:

`somecommand`

the backquotes aren't doing what you seem to think they are, and in fact they're potentially dangerous. What they do is execute the command, capture its output, and try to execute that output as another command. If the command doesn't print anything, this doesn't matter; but if it does print something, it's unlikely the output will be a valid command. Just lose the backquotes.

Lastly, giving a password as a command argument is insecure -- command arguments published in the process table (for example, the ps command can see them), and publishing passwords in public locations is a really bad idea. freebcp doesn't seem to have any alternative way to do this, but I found a patch that'll let it read the password from stdin (echo "$password" | freebcp -P - ... -- note that echo is a shell builtin, so its arguments don't show up in the process table). I make no claims about the correctness, safety, etc of the patch (especially since it's rather old), but I'd check it out if I were you.