Shell – Why does `zip` in a for loop work when the file exists, but not when it doesn’t

quotingshellshell-scriptzipzsh

I have a directory that contains several sub-directories. There is a question about zipping the files that contains an answer that I ever-so-slightly modified for my needs. 

for i in */; do zip "zips/${i%/}.zip" "$i*.csv"; done

However, I run into a bizarre problem. For the first set of folders, where zips/<name>.zip does not exist, I get this error:

zip error: Nothing to do! (zips/2014-10.zip)
        zip warning: name not matched: 2014-11/*.csv

however when I just echo the zip statements:

for i in */; do echo zip "zips/${i%/}.zip" "$i*.csv"; done

Then run the echoed command (zip zips/2014-10.zip 2014-10/*.csv), it works fine and zips up the folder. Then the fun part about that is that subsequent runs of the original command will actually zip up folders that didn't work the first time!

To test this behavior yourself:

cd /tmp
mkdir -p 2016-01 2016-02 2016-03 zips
for i in 2*/; do touch "$i"/one.csv; done
for i in 2*/; do touch "$i"/two.csv; done
zip zips/2016-03.zip 2016-03/*.csv
for i in 2*/; do echo zip "zips/${i%/}.zip" "$i*.csv"; done
for i in 2*/; do zip "zips/${i%/}.zip" "$i*.csv"; done

You'll see that the echo prints these statements:

zip zips/2016-01.zip 2016-01/*.csv
zip zips/2016-02.zip 2016-02/*.csv
zip zips/2016-03.zip 2016-03/*.csv

However, the actual zip command will tell you:

        zip warning: name not matched: 2016-01/*.csv

zip error: Nothing to do! (zips/2016-01.zip)
        zip warning: name not matched: 2016-02/*.csv

zip error: Nothing to do! (zips/2016-02.zip)
updating: 2016-03/one.csv (stored 0%)
updating: 2016-03/two.csv (stored 0%)

So it's actually updating the zip file with the .csvs where the zip file exists, but not when the zip file is created. And if you copy one of the zip commands:

$ zip zips/2016-02.zip 2016-02/*.csv
adding: 2016-02/one.csv (stored 0%)
adding: 2016-02/two.csv (stored 0%)

Then re-run the zip-all-the-things:

for i in 2*/; do zip "zips/${i%/}.zip" "$i*.csv"; done

You'll see that it updates for 2016-02 and 2016-03. Here's my output of tree:

.
├── 2016-01
│   ├── one.csv
│   └── two.csv
├── 2016-02
│   ├── one.csv
│   └── two.csv
├── 2016-03
│   ├── one.csv
│   └── two.csv
└── zips
    ├── 2016-02.zip
    └── 2016-03.zip

Also, (un)surprisingly, this works just fine:

zsh -c "$(for i in 2*/; do echo zip "zips/${i%/}.zip" "$i*.csv"; done)"

What am I doing wrong here? (note, I am using zsh instead of bash, if that makes any difference)

Best Answer

Expansion by the shell

The quotes around "$i*.csv" make the difference. With the quotes, the shell expands that string to "2014-11/*.csv". That exact file doesn't exist, and zip reports an error. Without quotes, the * also expands (via filename expansion/"globbing"), and the resulting zip command is a complete list of matching files, each as a separate argument. You can get the second behaviour, inside the for loop, with:

for i in */ ; do zip "zips/${i%/}.zip" "$i"*.csv ; done

Expansion by zip

zip can also expand wildcards for itself, but not in all situations. From the zip manual:

The zip program can do the same matching on names that are in the zip archive being modified or, in the case of the -x (exclude) or -i (include) options, on the list of files to be operated on, by using backslashes or quotes to tell the shell not to do the name expansion.

The original command works on subsequent attempts, after you've successfully created an archive, because zip tries to match the wildcards against the contents of the existing archive. They exist there, and still exist on the filesystem, so they're reported with updating:.

To get zip to handle the wildcards when creating the archive, use the -r (recurse) option to recurse into the requested directory, and -i (include) to limit it to files matching the pattern:

 for i in */ ; do zip -r "zips/${i%/}.zip" "$i" -i '*.csv' ; done
Related Question