Bash Globbing – How to Make Bash Glob a String Variable

bashquotingshell-scriptwildcards

System Info

OS: OS X

bash: GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin16)

Background

I want time machine to exclude a set of directories and files from all my git/nodejs project. My project directories are in ~/code/private/ and ~/code/public/ so I'm trying to use bash looping to do the tmutil.

Issue

Short Version

If I have a calculated string variable k, how do I make it glob in or right before a for-loop:

i='~/code/public/*'
j='*.launch'
k=$i/$j # $k='~/code/public/*/*.launch'

for i in $k # I need $k to glob here
do
    echo $i
done

In the long version below, you will see k=$i/$j. So I cannot hardcode the string in the for loop.

Long Version

#!/bin/bash
exclude='
*.launch
.classpath
.sass-cache
Thumbs.db
bower_components
build
connect.lock
coverage
dist
e2e/*.js
e2e/*.map
libpeerconnection.log
node_modules
npm-debug.log
testem.log
tmp
typings
'

dirs='
~/code/private/*
~/code/public/*
'

for i in $dirs
do
    for j in $exclude
    do
        k=$i/$j # It is correct up to this line

        for l in $k # I need it glob here
        do
            echo $l
        #   Command I want to execute
        #   tmutil addexclusion $l
        done
    done
done

Output

They are not globbed. Not what I want.

~/code/private/*/*.launch                                                                                   
~/code/private/*/.DS_Store                                                                                  
~/code/private/*/.classpath                                                                                 
~/code/private/*/.sass-cache                                                                                
~/code/private/*/.settings                                                                                  
~/code/private/*/Thumbs.db                                                                                  
~/code/private/*/bower_components                                                                           
~/code/private/*/build                                                                                      
~/code/private/*/connect.lock                                                                               
~/code/private/*/coverage                                                                                   
~/code/private/*/dist                                                                                       
~/code/private/*/e2e/*.js                                                                                   
~/code/private/*/e2e/*.map                                                                                  
~/code/private/*/libpeerconnection.log                                                                      
~/code/private/*/node_modules                                                                               
~/code/private/*/npm-debug.log                                                                              
~/code/private/*/testem.log                                                                                 
~/code/private/*/tmp                                                                                        
~/code/private/*/typings                                                                                    
~/code/public/*/*.launch                                                                                    
~/code/public/*/.DS_Store                                                                                   
~/code/public/*/.classpath                                                                                  
~/code/public/*/.sass-cache                                                                                 
~/code/public/*/.settings                                                                                   
~/code/public/*/Thumbs.db                                                                                   
~/code/public/*/bower_components                                                                            
~/code/public/*/build                                                                                       
~/code/public/*/connect.lock                                                                                
~/code/public/*/coverage                                                                                    
~/code/public/*/dist                                                                                        
~/code/public/*/e2e/*.js                                                                                    
~/code/public/*/e2e/*.map                                                                                   
~/code/public/*/libpeerconnection.log                                                                       
~/code/public/*/node_modules                                                                                
~/code/public/*/npm-debug.log                                                                               
~/code/public/*/testem.log                                                                                  
~/code/public/*/tmp                                                                                         
~/code/public/*/typings

Best Answer

You can force another round of evaluation with eval, but that's not actually necessary. (And eval starts having serious problems the moment your file names contain special characters like $.) The problem isn't with globbing, but with the tilde expansion.

Globbing happens after variable expansion, if the variable is unquoted, as here(*):

$ x="/tm*" ; echo $x
/tmp

So, in the same vein, this is similar to what you did, and works:

$ mkdir -p ~/public/foo/ ; touch ~/public/foo/x.launch
$ i="$HOME/public/*"; j="*.launch"; k="$i/$j"
$ echo $k
/home/foo/public/foo/x.launch

But with the tilde it doesn't:

$ i="~/public/*"; j="*.launch"; k="$i/$j"
$ echo $k
~/public/*/*.launch

This is clearly documented for Bash:

The order of expansions is: brace expansion; tilde expansion, parameter and variable expansion, ...

Tilde expansion happens before variable expansion so tildes inside variables are not expanded. The easy workaround is to use $HOME or the full path instead.

(* expanding globs from variables is usually not what you want)

Another thing:

When you loop over the patterns, as here:

exclude="foo *bar"
for j in $exclude ; do
    ...

note that as $exclude is unquoted, it's both split, and also globbed at this point. So if the current directory contains something matching the pattern, it's expanded to that:

$ i="$HOME/public/foo"
$ exclude="*.launch"
$ touch $i/real.launch
$ for j in $exclude ; do           # split and glob, no match
    echo "$i"/$j ; done
/home/foo/public/foo/real.launch

$ touch ./hello.launch
$ for j in $exclude ; do           # split and glob, matches in current dir!
    echo "$i"/$j ; done
/home/foo/public/foo/hello.launch  # not the expected result

To work around this, use an array variable instead of a splitted string:

$ exclude=("*.launch")
$ exclude+=("something else")
$ for j in "${exclude[@]}" ; do echo "$i"/$j ; done
/home/foo/public/foo/real.launch
/home/foo/public/foo/something else

As an added bonus, array entries can also contain whitespace without issues with splitting.

Something similar could be done with find -path, if you don't mind what directory level the targeted files should be. E.g. to find any path ending in /e2e/*.js:

$ dirs="$HOME/public $HOME/private"
$ pattern="*/e2e/*.js"
$ find $dirs -path "$pattern"
/home/foo/public/one/two/three/e2e/asdf.js

We have to use $HOME instead of ~ for the same reason as before, and $dirs needs to be unquoted on the find command line so it gets split, but $pattern should be quoted so it isn't accidentally expanded by the shell.

(I think you could play with -maxdepth on GNU find to limit how deep the search goes, if you care, but that's a bit of a different issue.)

Related Question