Parameter expansion (variable expansion) and quotes within quotes

command lineshshellunix

While working in a (Unix) shell, sometimes I need to double-quote single-quotes or vice versa, i.e. to write something like "foo 'bar baz' moo" or 'foo "bar baz" moo'. This may be because

the resulting string will be parsed again (like with eval, with a shell run by watch, with a shell run by sshd on some remote server etc.) and these inner quotes are crucial to distinguish actual arguments during this second parsing;
or I just need the original argument to include quotes for whatever other reason.

Some ambiguity appears when instead of bar baz I have a variable, say $variable. This is what POSIX says:

2.2.2 Single-Quotes

Enclosing characters in single-quotes ('') shall preserve the literal value of each character within the single-quotes. A single-quote cannot occur within single-quotes.

2.2.3 Double-Quotes

Enclosing characters in double-quotes ("") shall preserve the literal value of all characters within the double-quotes, with the exception of the characters backquote, <dollar-sign>, and <backslash>, as follows:

$
The shall retain its special meaning introducing parameter expansion […], a form of command substitution […], and arithmetic expansion […].

[…]

It's not exactly obvious which rule applies for $variable inside single-quotes inside double-quotes (or inside double-quotes inside single-quotes), therefore I'm asking:

Which rule applies to:
- ```
variable=123
echo "'$variable'"
```
- ```
variable=123
echo '"$variable"'
```
^{(I'm aware this can be easily answered just by trying, it would be a reasonable research effort. I decided to put this effort into the community wiki answer below).}
Are there any exceptions, quirks, surprises?

^{(Please note there is a community wiki answer below. Instead of adding a separate answer with another exception or quirk, consider contributing to the community wiki one).}

Best Answer

Which rule applies?

Just by trying:

$ variable=123
$ echo "'$variable'"
'123'
$

$ variable=123
$ echo '"$variable"'
"$variable"
$

So the outer quotes matter. Common shells behave in the same way: sh, bash, dash, zsh, … It's because of this general rule:

If the current character is <backslash>, single-quote, or double-quote and it is not quoted, it shall affect quoting for subsequent characters up to the end of the quoted text.

^(source)

Only the outer quotes are not quoted, the rule applies to them. The inner quotes are quoted, they do not affect subsequent characters.

Are there any exceptions, quirks, surprises?

^{(Well, echo may introduce few surprises, see Why is printf better than echo? Still, this is a separate issue. For readability the answer uses echo with a safe value of $variable).}

The above code gives us either '123' or "$variable". What if we need "123" or '$variable'?

To get "123":

variable=123
echo "\"$variable\""
#     ^^         ^^   escaped inner quotes

variable=123
echo '"'"$variable"'"'
#    1 12         23 3   outer quotes (numbered pairs)
#     ^  ^^^^^^^^^  ^    quoted fragments

To get '$variable':

variable=123
echo "'"'$variable'"'"
#    1 12         23 3   outer quotes (numbered pairs)
#     ^  ^^^^^^^^^  ^    quoted fragments

The question notices that $ retains its special meaning in double-quotes. This is important:

The input characters within the quoted string that are also enclosed between $( and the matching ) shall not be affected by the double-quotes, but rather shall define that command whose output replaces the $(…) when the word is expanded.

^(source)

It means quoting within $(…) should be considered independently, despite the outer double-quotes. Indeed:
```
$ variable=123
$ echo "$(echo 'foo $variable')"
$ #            ^             ^   single-quotes matter!
foo $variable
$
```
(Note echo $(stuff) is bad code in general; it's here to illustrate the current issue).

In these two cases:
- echo "'$variable'" (tried way above)
- echo "$(echo 'foo $variable')" (discussed here)
there is $variable inside single-quotes inside double-quotes, but $(…) changes a lot. Similar situation with echo "`echo 'foo $variable'`" (although $(…) and backticks are not exactly equivalent).

Related Solutions

Windows – Setting and using variable within same command line in Windows cmd.exe

Note that cmd /C "set "EDITOR=vim" && echo %EDITOR%" would not work.
Nor would cmd /C "setlocal ENABLEDELAYEDEXPANSION && set "EDITOR=vim" && echo !EDITOR!"

What would is the /V option, to enable delayed environment variable expansion using ! as delimiter.

C:\> cmd /V /C "set EDITOR=vim&& echo !EDITOR!"
vim

As noted below by maoizm, it is cmd /V /C, not cmd /C /V (which would not work)

I can't think of any practical reason you'd ever actually want this within the context of a single command

Typically, you need this when you have to replace a value used multiple times in a long command line.
For instance, to deploy a file to Nexus (in multiple lines for readability):

cmd /v /c "set g=com.agroup&& set a=anArtifact&& set v=1.1.0&& \
           mvn deploy:deploy-file -Dfile=C:\path\!a!-!v!.jar \
           -Dpackaging=jar -DgroupId=!g! -DartifactId=!a! -Dversion=!v! \
           -DrepositoryId=nexus 
           -Durl=http://myserver/nexus/content/repositories/my-repo/"

Instead of having to replace group, artifact (used 2 times) and version in a long and complex command line, you can edit them at the beginning of said command. It is clearer/easier to manipulate and change the parameter values.

Linux – With Bash variables, what is the difference between $thevar and “$thevar”? (Specific odd behavior)

Different approach:

When you run git log --format="foo bar", those quotes aren't interpreted by git – they're removed by the shell (and protect the quoted text from splitting). This results in a single arg:

--format=foo bar

However, when unquoted variables are expanded, the results go through word-splitting, but not through unquoting. So if your variable contains --format="foo bar", it is expanded into these args:

--format="foo
bar"

This can be verified using:

```
printf '%s\n' $variable
```

...as well as any simple script which prints its received arguments.

#!/usr/bin/env perl
for $i (0..$#ARGV) {
    print ($i+1)." = ".$ARGV[$i]."\n";
}

#!/usr/bin/env python3
import sys
for i, arg in enumerate(sys.argv):
    print(i, "=", arg)

If you always have bash available, the preferred workaround is to use array variables:

myvar=( --format="foo bar" )

With this, the usual parsing is done during assignment, not during expansion. You use this syntax to expand the variable's contents, each element getting its own arg:

git log "${myvar[@]}"

Best Answer

Related Solutions

Windows – Setting and using variable within same command line in Windows cmd.exe

Linux – With Bash variables, what is the difference between $thevar and “$thevar”? (Specific odd behavior)

Related Question