Bash add value to array with embedded variable and single quotes

arraybashquoting

I'm sure it's completely clear from the subject 🙂 Joking aside, I think part of the problem I'm having finding an answer is writing the search terms.

I have had good luck when issuing commands in a bash script by using an array for the arguments.

For example, I'm calling rsync and created an array like (spolier alert – superfluous variables will be referenced later):

#!/usr/bin/env bash
set -euo pipefail
rf=.rsync-filter
srcff="/home/roger/home$rf"

hm=/home/roger
syncto=/tmp/to-remote/home

rsarg=(-avv --prune-empty-dirs --stats)
rsarg+=(--dry-run)

rsync "${rsarg[@]}" "$hm" "$syncto/"  #works

exit

Where I'm having problems is in adding the $srcff variable to a filter rule and adding that to the rsarg array.

rsarg+=("--filter='merge $srcff'") # no dice
# or this
rsarg+=("--filter='merge ${srcff}'") # sad trombone

Upon running I get:

$ ./home-sync-simple
Unknown filter rule: `'merge /home/roger/home.rsync-filter''
rsync error: syntax or usage error (code 1) at exclude.c(904) [client=3.1.1]

I can get it to work by adding the merge rule to the rsync line, outside of the array and minus embedded variable:

rsync "${rsarg[@]}" --filter='merge /home/roger/home.rsync-filter' "$hm" "$syncto/"

Replacing the above filter with the variable outside the array, fails in the same manner:

rsync "${rsarg[@]}" "--filter='merge $srcff'" "$hm" "$syncto/"

I have tried many other variations, I'm assuming all possible excepting the one that actually works. Sometimes it results in "unexpected end of filter rule: merge" and other things. I've excluded those for brevity and because I think the versions above are closer to correct (he said, nervously). I am guessing I end up passing too few/many arguments.

Thanks in advance!

Best Answer

If this works:

rsync "${rsarg[@]}" --filter='merge /home/roger/home.rsync-filter' ...

Then I suppose you do not want to give the single-quotes to rsync. Here, the shell eats them before rsync sees them. Your other examples have quotes within quotes, so the inner ones stay and are passed for rsync to see.

So skip the single-quotes when building the array:

rsarg+=("--filter=merge $srcff")

Note that from the shell's point of view, the equal sign is nothing special, and there's no need to treat the part that comes after any differently than the part before. --foo=bar is the same as --foo="bar" or "--foo=bar", or even --fo"o=ba"r.

Related Solutions

Bash – Single Parenthesis in Variable Assignment

In your case parenthesis () are used as array definition, for example

a=(one two three)   # array definition
echo "${a}"         # print first element of array a
echo "${a[0]}"      # print first element of array a
echo "${a[1]}"      # print *second* element of array a
echo "${#a[@]}"     # print number of elements in array a

If you put single variable in array then you just have an array with single element.

To answer your other question whether parenthesis are also used for anything else: there are many situations in bash that in combination with other characters they can be used as:

command substitution: $()
process substitution: <() and >()
subshell: (command)
arithmetic evaluation: (())
function definition: fun () { echo x; }
pattern list in glob: ?(), *(), +(), @(), !() (only if extglob is enable)

Bash Arrays – Using Variables in Array Names

Some ideas:

A "parameter expansion" of a variable value (the ${...} part):
```
echo "arr$COUNTER[0] = ${arr$COUNTER[0]}"
```
will not work. You may get around by using eval (but I do not recommend it):
```
eval echo "arr$COUNTER[0] = \${arr$COUNTER[0]}"
```
That line could be written as this:
```
i="arr$COUNTER[0]"; echo "$i = ${!i}"
```
That is called indirection (the !) in Bash.
A similar issue happens with this line:
```
declare -a "arr$COUNTER=($field)"
```
Which should be split into two lines, and eval used:
```
declare -a "arr$COUNTER"
eval arr$COUNTER\=$ \$field $
```
Again, I do not recommend using eval (in this case).
As you are reading the whole file into the memory of the shell, we may as well use a simpler method to get all lines into an array:
```
readarray -t lines <"VALUES_FILE.txt"
```
That should be faster than calling awk for each line.

An script with all the above could be:

#!/bin/bash
valfile="VALUES_FILE.txt"

readarray -t lines <"$valfile"             ### read all lines in.

line_count="${#lines[@]}"
echo "Total number of lines $line_count"

for ((l=0;l<$line_count;l++)); do
    echo "Counter value is $l"             ### In which line are we?
    echo "Field = ${lines[l]}"             ### kepth only to help understanding.
    k="arr$l"                              ### Build the variable arr$COUNTER
    IFS=" " read -ra $k <<<"${lines[l]}"   ### Split into an array a line.
    eval max=\${#$k[@]}                    ### How many elements arr$COUNTER has?
    #echo "field $field and k=$k max=$max" ### Un-quote to "see" inside.
    for ((j=0;j<$max;j++)); do             ### for each element in the line.
        i="$k[$j]"; echo "$i = ${!i}"      ### echo it's value.
    done
done
echo "The End"
echo

However, still, AWK may be faster, if we could execute what you need in AWK.

A similar processing could be done in awk. Assuming the 6 values will be used as an IP (4 of them) and the other two are a number and an epoch time.

Just a very simple sample of an AWK script:

#!/bin/sh
valfile="VALUES_FILE.txt"
awk '
NF==6 { printf ( "IP: %s.%s.%s.%s\t",$1,$2,$3,$4)
        printf ( "number: %s\t",$5+2)
        printf ( "epoch: %s\t",$6)
        printf ( "\n" )
    }
' "$valfile"

Just make a new question with the details.

Best Answer

Related Solutions

Bash – Single Parenthesis in Variable Assignment

Bash Arrays – Using Variables in Array Names

Related Question