Shell Quoting – How to Use Variables Inside Single Quotes

quotingshellvariable

I have an application which takes as an input attributes in double quotes embedded in single quotes. Take for example this right command:

command -p 'cluster="cl1"'

In order to automate it, I created a bash file using $CLUSTER as a variable.
How should be my command? In other words, what should I put instead of cl1?

Please note that, If I modified the above command, it won't be accepted. For instance: command -p "cluster=cl1" is not accepted

Best Answer

It looks like your command is maybe setting environment variables based on arguments given it on the command-line. It may be you can do:

CLUSTER=cl1; cluster=$CLUSTER command

...and set its environment for it at invocation.

Otherwise, shell quotes typically delimit arguments or escape other special shell characters from shell interpretation. You can contain (and therefore escape) different kinds of shell-quotes within other kinds based on various rules:

"''''" - a soft-quoted string can contain any number of hard-quotes.
"\"" - a \ backslash can escape a "soft-quote within a "soft-quoted string.
- In this context a \\backslash also escapes itself, the \$expansion token, and \newlines as noted below, but is otherwise treated literally.
"${expand} and then some" - a soft-quoted string can contain an interpreted shell $expansion.
'"\' - a 'hard-quoted string can contain any character other than a 'hard-quote.
\ - an unquoted backslash will escape any following character for literal interpretation - even another backslash - excepting a \newline.
- In a \\newline case both the \ backslash and the \newline are completely removed from the resulting interpreted command.
${parameter+expand "$parameter"} - quotes resulting from a shell expansion almost never serve as delimiter markers excepting a few special cases. I won't venture to describe these further here.

I consider it odd that any application would interpret quotes in its command-line args. Such a practice doesn't make a lot of sense in that - for shells, at least - the primary purpose of a quote is generally to delimit an argument. At invocation, however, arguments are always already delimited with \0NUL characters and so a quote cannot serve much purpose.

Even a shell will typically only bother to interpret quotes in one of its invocation arguments when it is called with a -c switch - which denotes that its first operand is actually a shell script that it should run upon invocation. This is a case of twice evaluated input.

All that said, you can do a number of things to pass literal quotes via arguments on the command-line. For example:

CLUSTER='"cl1"'; command -p "cluster=$CLUSTER"

As I noted in a comment before, you can contain the " quotes within an expansion that is itself " quoted.

CLUSTER=cl1; command -p "cluster=\"$CLUSTER\""

You can escape the " with a \backslash within the " quoted string.

CLUSTER=cl1; command -p cluster='"'"$CLUSTER"'"'

You can alternate and concatenate quoting styles to arrive at your desired end result as @jimmij notes above.

CLUSTER=cl1; ( set -f; IFS=; command -p cluster=\"$CLUSTER\" )

You can disable both file name generation and $IFS splitting - thereby avoiding the need to quote the $expansion at all - and so only quote the quotes. This is probably overkill.

Last, there is another type of shell-quote that might be used. As I noted before the sh -c "$scriptlet" form of shell invocation is often used to provide a shell's script on the command-line. When $scriptlet gets complicated though - such as when quotes must contain other quotes - it can often be advantageous to use a here-document and sh -s instead - where the shell is specifically instructed to assign all following operands to the positional parameters as it would do in a -c case and yet to also take its script from stdin.

If your command must interpret quotes in this way then I would consider it better if it could do so in a file input. For example:

CLUSTER=cl1
command --stdin <<-SCRIPT
    cluster="$CLUSTER"
SCRIPT

If you do not quote the delimiter of a <<here-document then all of its contents are treated almost exactly like they were "soft-quoted - except that "double-quotes themselves are not treated specially. And so if we run the above with cat instead:

CLUSTER=cl1
cat <<-SCRIPT
        cluster="$CLUSTER"
SCRIPT

...it prints...

cluster="cl1"

Related Solutions

Bash – Why Does Bash Add Single Quotes to Unquoted Failed Pathname Expansions?

When instructed to echo commands as they are executed ("execution trace"), both bash and ksh add single quotes around any word with meta-characters (*, ?, ;, etc.) in it.

The meta-characters could have gotten into the word in a variety of ways. The word (or part of it) could have been quoted with single or double quotes, the characters could have been escaped with a \, or they remained as the result of a failed filename matching attempt. In all cases, the execution trace will contain single-quoted words, for example:

$ set -x
$ echo foo\;bar
+ echo 'foo;bar'

This is just an artifact of the way the shells implement the execution trace; it doesn't alter the way the arguments are ultimately passed to the command. The quotes are added, printed, and discarded. Here is the relevant part of the bash source code, print_cmd.c:

/* A function to print the words of a simple command when set -x is on. */
void
xtrace_print_word_list (list, xtflags)
...
{
  ...
  for (w = list; w; w = w->next)
    {
      t = w->word->word;
      ...
      else if (sh_contains_shell_metas (t))
        {
          x = sh_single_quote (t);
          fprintf (xtrace_fp, "%s%s", x, w->next ? " " : "");
          free (x);
        }

As to why the authors chose to do this, the code there doesn't say. But here's some similar code in variables.c, and it comes with a comment:

/* Print the value cell of VAR, a shell variable.  Do not print
   the name, nor leading/trailing newline.  If QUOTE is non-zero,
   and the value contains shell metacharacters, quote the value
   in such a way that it can be read back in. */
void
print_var_value (var, quote)
...
{
  ...
  else if (quote && sh_contains_shell_metas (value_cell (var)))
    {
      t = sh_single_quote (value_cell (var));
      printf ("%s", t);
      free (t);
    }

So possibly it's done so that it's easier to copy the command lines from the output of the execution trace and run them again.

Vim – How to escape filename containing single and double quotes mix

From :help filename-modifiers:

The file name modifiers can be used after "%", "#", "#n", "<cfile>", "<sfile>",
"<afile>" or "<abuf>".  ...

...
    :s?pat?sub?
            Substitute the first occurrence of "pat" with "sub".  This
            works like the |:s| command.  "pat" is a regular expression.
            Any character can be used for '?', but it must not occur in
            "pat" or "sub".
            After this, the previous modifiers can be used again.  For
            example ":p", to make a full path after the substitution.
    :gs?pat?sub?
            Substitute all occurrences of "path" with "sub".  Otherwise
            this works like ":s".

So rather than just handling double quotes or single quotes, let's just backslash escape everything unusual:

:!ls -l %:gs/[^0-9a-zA-Z_-]/\\&/

Works perfectly with the test filename you provided.

To use an absolute path, which you may want for rsync, you can add :p at the end:

:!ls -l %:gs/[^0-9a-zA-Z_-]/\\&/:p

Actually, it also works just fine if you backslash-escape literally every character, and it's shorter to type:

:!ls -l %:gs/./\\&/:p

So, in your rsync command, instead of %, use %:gs/./\\&/:p.

Best Answer

Related Solutions

Bash – Why Does Bash Add Single Quotes to Unquoted Failed Pathname Expansions?

Vim – How to escape filename containing single and double quotes mix

Related Question