Bash – Iterate Through Sets of Command Arguments in Bash

bashshellshell-script

I have multiple "sets" of arguments for one command that I need to run in sequence (well, in parallel, technically). I also need to repeat the same logic after running each command.

#!/bin/bash

local pids=()

# Run commands in parallel.
my_command "$URL_ONE"   "$URL_ONE_TEXT"    "${TMP_DIR}/some_dir" &
pids+=("$!")

my_command "$URL_ONE"   "$URL_TWO_TEXT"    "${TMP_DIR}/some_similar_dir" &
pids+=("$!")

my_command "$URL_TWO"   "$URL_TWO_TEXT"    "${TMP_DIR}/third_dir" &
pids+=("$!")

my_command "$URL_THREE" "$URL_THREE_TEXT"  "${TMP_DIR}/fourth_dir" &
pids+=("$!")

# ...

# Wait for parallel commands to complete and exit if any fail.
for pid in "${pids[@]}"; do
    wait "$pid"
    if [[ $? -ne 0 ]]; then
        errecho "Failed."
        exit 1
    fi
done

Rather than repeating pids+=("$!") and other portions so frequently, what I'd like to do is define an array/set of arguments, loop through it, and execute the same logic for each set of arguments. For example:

#!/bin/bash

# This wouldn't actually work...
ARG_SETS=(
    ("$URL_ONE"   "$URL_ONE_TEXT"   "${TMP_DIR}/some_dir")
    ("$URL_ONE"   "$URL_TWO_TEXT"   "${TMP_DIR}/some_similar_dir")
    ("$URL_TWO"   "$URL_TWO_TEXT"   "${TMP_DIR}/third_dir")
    ("$URL_THREE" "$URL_THREE_TEXT" "${TMP_DIR}/fourth_dir")
)
for arg1 arg2 arg3 in "$ARG_SETS[@]"; do
    my_command "$arg1" "$arg2" "$arg3" &
    pids+=("$!")
done

But Bash does not support multidimensional arrays. Does anyone have any ideas for a good pattern to make this cleaner, or do something similar in design to my second example? Thanks!

Best Answer

This approach uses three arrays, one for each argument of the my_command:

pids=()

a=("$URL_ONE"               "$URL_ONE"                      "$URL_TWO"                  "$URL_THREE")
b=("$URL_ONE_TEXT"          "$URL_TWO_TEXT"                 "$URL_TWO_TEXT"         "$URL_THREE_TEXT")
c=("${TMP_DIR}/some_dir"    "${TMP_DIR}/some_similar_dir"   "${TMP_DIR}/third_dir"  "${TMP_DIR}/fourth_dir")

for i in ${!a[@]}
do
    my_command "${a[$i]}" "${b[$i]}" "${c[$i]}" &
    pids+=("$!")
done

# Wait for parallel commands to complete and exit if any fail.
for pid in "${pids[@]}"
do
    if ! wait "$pid"
    then
        errecho "Failed."
        exit 1
    fi
done

Alternate style

Depending on how many commands are to be run, you may want to consider the following alternative for defining the arrays:

a=();              b=();                   c=()
a+=("$URL_ONE");   b+=("$URL_ONE_TEXT");   c+=("${TMP_DIR}/some_dir")
a+=("$URL_ONE");   b+=("$URL_TWO_TEXT");   c+=("${TMP_DIR}/some_similar_dir")
a+=("$URL_TWO");   b+=("$URL_TWO_TEXT");   c+=("${TMP_DIR}/third_dir")
a+=("$URL_THREE"); b+=("$URL_THREE_TEXT"); c+=("${TMP_DIR}/fourth_dir")

Related Solutions

Bash – How to Loop Through Arguments in a Bash Script

There's a special syntax for this:

for i do
  printf '%s\n' "$i"
done

More generally, the list of parameters of the current script or function is available through the special variable $@.

for i in "$@"; do
  printf '%s\n' "$i"
done

Note that you need the double quotes around $@, otherwise the parameters undergo wildcard expansion and field splitting. "$@" is magic: despite the double quotes, it expands into as many fields as there are parameters.

print_arguments () {
  for i in "$@"; do printf '%s\n' "$i"; done
}
print_arguments 'hello world' '*' 'special  !\characters' '-n' # prints 4 lines
print_arguments ''                                             # prints one empty line
print_arguments                                                # prints nothing

Bash – Iterate through command line arguments in Bash

roaima's answer answers the question that you actually asked:

Q: What is the difference between these two code blocks? Why do they give different output?

A: The first loop is iterating over the command line arguments; the second one is iterating over the argument numbers (indices).

... although I presume that you would have figured that much out for yourself in another six to eight minutes — it's kind-of obvious. You probably want the second program to do something like

echo $$i                                        # This doesn't do what you want.

to display the argument that is indexed by the number that is stored in variable i (and referenced as $i). As noted in the comment, that doesn't do what you want. (It does do something; I encourage you to experiment and figure out what it does.) But this is close to something that does work:

eval echo \$$i                                  # Don't do this.

or, equivalently,

eval echo '$'"$i"                               # Don't do this.

These commands

get the value of i (one of the numbers 1, 2, 3, ...)
stick a $ in front of it, forming $1, $2, $3, etc.
use the eval command to say, "take this command line that I've just constructed, and evaluate it as if I had typed it.

So that would have the effect of executing

echo $1
echo $2
echo $3
   ︙

But, as the comments suggest, you should try to avoid this. eval can be dangerous if the input is anything other than plain words. Search this site; you'll find plenty of explanations of that.

But there is a fairly safe way to get the second program to do the same thing the first one does: change

echo $i

echo ${!i}

OK, first of all, ${i} is pretty much the same as $i. The ! gives you an effect similar to that of the eval command — ${!x} looks up the value of x (i.e., $x or ${x}) and uses that as the name of the variable to look up. So, if x=foo, then ${!x} is the same as $foo. The above code does the same thing with i, fetching the parameter whose name is the value of i.

By the way, you should always quote all references to shell variables (e.g., "$i", "$#", "$args", "${i}" and "${!i}") unless you have a good reason not to, and you’re sure you know what you’re doing.

Best Answer

Alternate style

Related Solutions

Bash – How to Loop Through Arguments in a Bash Script

Bash – Iterate through command line arguments in Bash

Related Question