Bash Shell Script Scripting Pipe – Piping for Loop Output Prevents Local Variable Modification

bashpipescriptingshellshell-script

I'm trying to write a simple bash function that takes, as its arguments, a number of files and/or directories. It should:

Fully qualify the filenames.
Sort them.
Remove duplicates.
Print all that actually exist.
Return the number of non-existent files.

I have a script that almost does what I want, but falls down on the sorting. The return value of the script as it stands is correct, but the output is not (unsorted and duplicates). If I uncomment the | sort -u statement as indicated, the output is correct but the return value is always 0.

N.B. Simpler solutions to solve the problem are welcome but the question is really about why is this occurring in the code I have. That is, why does adding the pipe seemingly stop the script incrementing the variable r?

Here's the script:

function uniqfile
{
    local r=0 

    for arg in "$@"
    do  
        readlink -e "$arg" || (( ++r ))

    done #| sort -u    ## remove that comment

    return $r
}

Best Answer

This is a well known bash pitfall, due to this feature:

Each command in a pipeline is executed as a separate process (i.e., in a subshell).

so that modified variables are local to the subshell, and not visible once back in the parent.

To avoid that, rephrase your code to avoid the pipeline, with a process substitution:

 for arg in "$@"
    do  
        readlink -e "$arg" || (( ++r ))

    done > >(sort -u)

Example

$ array[0]=0
$ array[1]=1
$ array[2]=2

You can then recall them individually:

$ echo ${array[0]}
0

$ echo ${array[1]}
1

$ echo ${array[2]}
2

Or as a list:

$ echo "${array[@]}"
0 1 2

You cannot just call the array as another variable:

$ echo $array
0

I believe this may have been what was misleading you into thinking it wasn't working correctly. Also setting an array element with another variable's value also works.

Example

$ newval=3
$ array[2]=$newval

$ echo "${array[@]}"
0 1 3

You could also use printf here, doing so will show you why you want to quote the results of ${array[@]}. But first let's add a string to the array:

$ array[0]="a string"

Without the quotes:

$ printf "%s\n" ${array[@]}
a
string
1
3

With quotes:

$ printf "%s\n" "${array[@]}"
a string
1
3

Passing things around

Well you have 2 tactics that you can make use of within Bash. You can simply set variables so that they're globally scoped. This is often the technique that most people employ.

The other method involves passing by value, values to functions within Bash. I'll show a basic function where I pass it 2 values.

Example

$ function hello() {
 printf "I got arg1: %d and arg2: %d\n" $1 $2
}

Now when I execute it with different arguments:

$ hello 1 2
I got arg1: 1 and arg2: 2

$ hello 3 4
I got arg1: 3 and arg2: 4

$ hello 3 blah
bash: printf: blah: invalid number
I got arg1: 3 and arg2: 0

You can read more about arrays and functions within Bash here to get a better understanding of the technologies.

References

Bash – How to resolve this “ambiguous redirect”

Put quotes around INSTALL_LOG_OUTPUT in your function, like this:

function install_log() {
    echo "$1" >> "$INSTALL_LOG_OUTPUT" 2>&1
}

After running your script again, you will probably get an error message indicating that INSTALL_LOG_OUTPUT is empty, with a message of this sort: bash: : No such file or directory.

Best Answer

Related Solutions

Bash – How to reassign new values to array elements

Example

Example

Passing things around

Example

References

Bash – How to resolve this “ambiguous redirect”

Related Question