Bash – How to Prevent Escaping of Special Characters in Loop as Argument for Another Command

bashshell

My input file (objects.lst) looks like this:

host.fqdn.com:/ 'host.fqdn.com [/]'
host.fqdn.com:/boot 'host.fqdn.com [/boot]'
host.fqdn.com:/tmp 'host.fqdn.com [/tmp]'
host.fqdn.com:/tmp '/tmp'
host.fqdn.com:/ '/usr/share'
host.fqdn.com:/var 'host.fqdn.com [/var]'

I want to loop over this file line by line and need every line as an argument for a 3rd party software.

So I tried something like:

#!/bin/bash
set -x
while read -r line; do
/opt/report -filesystem "$line" -detail
done < objects.lst

This will be interpreted as

 /opt/report -filesystem 'host.fqdn.com:/ '\''host.fqdn.com [/]'\''' -detail

But should be

/opt/report -filesystem host.fqdn.com:/ 'host.fqdn.com [/]'

I tried a lot of different things, like saving the argument in an array, echo -E into a variable, all kinds of different quotations and also via a for-loop, nothing worked the way I need it.

At the moment my workaround looks like this:

#!/bin/bash
set -x
while read -r line; do
echo "/opt/report -filesystem "${line}" -detail" | bash -
done < objects.lst

Is there another way to get this working without echoing/piping into bash – ?

I saw some examples with ${parameter@Q} (available with bash 4.4) but I'm on bash 4.2.

I obviously confused the set -x output, but my problem is still the same. I loop through the objects.lst and need the whole line exactly as it is in the objects.lst as an argument.

So if I understand the comments below correctly:

while read -r line; do
/opt/report -filesystem "$line" -detail
done < objects.lst

Should be the exact same thing as:

/opt/report -filesystem host.fqdn.com:/ 'host.fqdn.com [/]'

But it is not. Maybe the 3rd party software is the problem here (microfocus data protector / omnidb cli command) because in the loop, I get a "No Object Found" from omnidb/report.

But if I do a

while read -r line; do
echo "/opt/report -filesystem "$line" -detail"
done < objects.lst

And do a copy/paste from the output command, it works without a problem and prints the correct object information from the omnidb/report command.

Best Answer

With /opt/report -filesystem "$line" -detail, host.fqdn.com:/ 'host.fqdn.com [/]' is passed verbatim as one argument to the command and the set -x output is telling you just that by showing the value in single quotes which are the verbatim quotes in shells.

In

/opt/report -filesystem host.fqdn.com:/ 'host.fqdn.com [/]'

You're passing two arguments to the command: host.fqdn.com:/ and host.fqdn.com [/]. That's be cause the space character and single quote characters are both special in the shell syntax.

For the shell, as seen above '...' are the verbatim quotes which serves at passing text literally without any character inside being treated specially, and SPC (when not itself quoted) in the shell syntax is used to separate arguments (or more generally, tokens in the shell syntax)

If you wanted to pass host.fqdn.com:/ 'host.fqdn.com [/]' literally as one argument, as shown by the set -x output, you'd need something like 'host.fqdn.com:/ '\''host.fqdn.com [/]'\''', though using other types of quotes like "host.fqdn.com:/ 'host.fqdn.com [/]'" would also work.

$ printf '<%s>\n' 'host.fqdn.com:/ '\''host.fqdn.com [/]'\'''
<host.fqdn.com:/ 'host.fqdn.com [/]'>
$ printf '<%s>\n' "host.fqdn.com:/ 'host.fqdn.com [/]'"
<host.fqdn.com:/ 'host.fqdn.com [/]'>

If you do actually need that host.fqdn.com:/ 'host.fqdn.com [/]' to be interpreted as shell code as opposed to being one literal argument to pass to the command, you would need to invoke the shell interpreter on the content of $line like with your piping to bash (though your usage of double quotes was incorrect) or eval:

#!/bin/bash -
set -x
while IFS= read -r line; do
  eval "/opt/report -filesystem $line -detail"
done < objects.lst

That means however that if a line contains ;reboot; or $(reboot) for instance, that will invoke the reboot command.

If you only wanted to perform the shell tokenisation and quote removal without the other side effects like the running of those reboot commands above, you could use zsh instead which has operators for that:

#!/bin/zsh -
set -x
while IFS= read -r line; do
  /opt/report -filesystem "${(Q@)${(z)line}}" -detail
done < objects.lst

Where the z causes the tokenisation as zsh code, and Q removes one layer of quotes. For intance, on a line like:

foo 'bar baz' $(echo x y)

It would pass 3 arguments: foo, bar baz and $(echo x y).

Related Question