Bash – Iterating over glob patterns, not the files in them

bashforshell-scriptwildcards

I would like to perform the same aggregation operations on each of several bunches of files where each bunch is matched by one glob pattern. What I would not like to do is pipe each file name into the aggregation function separately.

My first attempt failed because the file names got globbed in the outer loop, flattening the whole collection into a flat list and not treating them as separate batches:

for fileglob in /path/to/bunch*1 /path/to/bunch*2 ... ; do
  stat "$fileglob" | awk [aggregation]
done

So I hid the * from the loop by escaping it, then unescaped it for the function:

for fileglob in /path/to/bunch*1 /path/to/bunch*2 ... ; do
  realglob=`echo "$fileglob" | sed 's/\\//g'`
  stat "$realglob" | awk [aggregation]
done

There has to be a better way. What is it?

GNU bash, version 3.2.51

Best Answer

This requires a careful use of quotes:

for fileglob in '/path/to/bunch*1' '/path/to/bunch*2' ... ; do
    stat $fileglob | awk [aggregation]
done

But that may fail on filenames with spaces (or newlines). Better to use this:

fileglobs=("/path/to/bunch*1" "/path/to/bunch*2")

for aglob in "${fileglobs[@]}" ; do
    set -- $aglob
    stat "$@" | awk [aggregation]
done

The glob gets correctly expanded and placed in the positional parameters with:

set -- $aglob

Then, each parameter is placed as an argument to stat in:

stat "$@"

And the output of stat goes (as one output) to awk.

Examples

First let's create some sample data.

$ mkdir .dotdir{1,2} regdir{1,2}
$ touch .dotfile{1,2} regfile{1..3}

So now we have the following:

$ tree -a
.
|-- .dotdir1
|-- .dotdir2
|-- .dotfile1
|-- .dotfile2
|-- regdir1
|-- regdir2
|-- regfile1
|-- regfile2
`-- regfile3

Now let's play some games. You can use the command echo to list out what a particular wildcard (*) would be for a given command like so:

$ echo *
regdir1 regdir2 regfile1 regfile2 regfile3


$ echo reg*
regdir1 regdir2 regfile1 regfile2 regfile3

$ echo .*
. .. .dotdir1 .dotdir2 .dotfile1 .dotfile2

$ echo .* *
. .. .dotdir1 .dotdir2 .dotfile1 .dotfile2 regdir1 regdir2 regfile1 regfile2 regfile3

$ echo .dotdir*
.dotdir1 .dotdir2

Changing the behavior?

You can use the command shopt -s dotglob to change the behavior of the * so that in addition to files like regfile1 it will also match .dotfile1.

excerpt from the bash man page

dotglob If set, bash includes filenames beginning with a `.' in the results 
        of pathname expansion.

Example:

$ shopt -s dotglob
$ echo *
.dotdir1 .dotdir2 .dotfile1 .dotfile2 regdir1 regdir2 regfile1 regfile2 regfile3

You can revert this behavior with this command:

$ shopt -u dotglob
$ echo *
regdir1 regdir2 regfile1 regfile2 regfile3

Your situation?

For you you're telling cp that you want to copy all the files that match the pattern *, and there aren't any files.

$ cp foo/.* .

Or you can do this if you want everything in the foo folder:

$ cp foo .

Or you can be explicit:

$ cp foot/.* foo/* .

A more compact form using brace expansion in bash:

$ cp foo/{.,}* .

At any time you can use the echo trick to see what your proposed file patterns (that's the fancy term for what the star is a part of).

$ echo {.,}*
. .. .dotdir1 .dotdir2 .dotfile1 .dotfile2 abc regdir1 regdir2 regfile1 regfile2 regfile3

Incidentally if you're going to copy a directory of files + other directories, you typically want to do this recursively, that's the -R switch to cp:

$ cp -R foo/. .

Shell – Iterating over files in Bash and obtaining the index and the counts

I would store the list of files in an array, so that you don't have to read the file system twice, increasing performance and reducing potential race conditions. Then use another variable as the index.

files=(*.jpg)
total=${#files[@]}
i=0
for f in "${files[@]}"; do
    i=$(( i + 1 ))
    echo index $i
    echo total $total
    echo "- Processing file: $f"
done

Explanation

files=(*.jpg): store the glob into the array $files
total=${#files[@]}: read the total into $total
i=0: initialise $i to 0.
i=$(( i + 1 )): add 1 to $i each loop

This presumes that the "first" loop is 1. Depending on your opinion, you might want to start at 0 instead.

Best Answer

Related Solutions

zsh – How to cp Hidden Files with Glob Patterns

Examples

Changing the behavior?

Your situation?

Shell – Iterating over files in Bash and obtaining the index and the counts

Explanation

Related Question