Shell Wildcards – Why Wildcard Character * Differs Between Zip and RM Commands

rmshellwildcardszip

I put together a script to do some file operations for me. I am using the wild card operator * to apply functions to all files of a type, but there is one thing I don't get. I can unzip all files in a folder like this

unzip "*".zip

However, to remove all zip files afterward, I need to do

rm *.zip

That is, it does not want the quotation marks. The unzip, on the other hand, does not work if I just give it the * (gives me a warning that "files were not matched").

Why is this different? To me, this seems like the exact same operation. Or am I using the wild card incorrectly?

Introductions to the wild card in Unix do not really go into this, and I could not locate anything in the rm or zip docs.

I am using the terminal on a Mac (Yosemite).

Best Answer

You've explained the situation very well. The final piece to the puzzle is that unzip can handle wildcards itself:

http://www.info-zip.org/mans/unzip.html

ARGUMENTS

file[.zip]

...

Wildcard expressions are similar to those supported in commonly used Unix shells (sh, ksh, csh) and may contain:

* matches a sequence of 0 or more characters

By quoting the * wildcard, you prevented your shell from expanding it, so that unzip sees the wildcard and deals with expanding it according to its own logic.

rm, by contrast, does not support wildcards on its own, so attempting to quote a wildcard will instruct rm to look for a literal asterisk in the filename instead.

The reason that unzip *.zip does not work is that unzip's syntax simply does not allow for multiple zip files; if there are multiple parameters, it expects the 2nd and subsequent ones to be files in the archive:

unzip [-Z] [-cflptTuvz[abjnoqsCDKLMUVWX$/:^]] file[.zip] [file(s) ...] [-x xfile(s) ...] [-d exdir]

Related Solutions

Unzipping certain archives works on Windows but not on Linux

I've done some digging in the source code (unzip60 from Ubuntu raring, though I suspect older versions don't differ much). The error in question is internally called TruncNTSD and defined in extract.c:295. Most uses of this message, as expected, are in win32/win32.c and indeed refer to NTFS security data, however there's only one place in the code where you should ever get this error outside of a win32 system (since you reported seeing it on ubuntu).

The place in question (extract.c:2118) is in a function called TestExtraField. As Wikipedia explains:

.ZIP file format includes the extra field facility within file headers, which can be used to store extra data not defined by existing .ZIP specifications, and allow compliant archivers not recognizing the fields to safely skip the fields.

That's indeed what NT does to store security information. Importantly, the function printing the error comments

/* we know the regular compressed file data tested out OK, or else we
 * wouldn't be here ==> print filename if any extra-field errors found
 */

So if you can unzip the files themselves fine, it looks like this error is safe to ignore. Looking further, the only place outside of win32 code that raises this error (assuming it's not a horrible bug in unzip) is test_compr_eb:extract.c:2227, which from a glance at the code looks like it occurs when a zipped file has associated extra fields that are marked as compressed, but the field data has a length of 0 bytes.

How this comes about I don't know - perhaps the program creating the zip files does this by accident, perhaps the extra fields are filtered out somewhere by security software. In any case, it looks harmless and probably has nothing to do with NT security at all. In conclusion, if your files unzip fine, it's completely safe to ignore.

Shell – Why does `zip` in a for loop work when the file exists, but not when it doesn’t

Expansion by the shell

The quotes around "$i*.csv" make the difference. With the quotes, the shell expands that string to "2014-11/*.csv". That exact file doesn't exist, and zip reports an error. Without quotes, the * also expands (via filename expansion/"globbing"), and the resulting zip command is a complete list of matching files, each as a separate argument. You can get the second behaviour, inside the for loop, with:

for i in */ ; do zip "zips/${i%/}.zip" "$i"*.csv ; done

Expansion by zip

zip can also expand wildcards for itself, but not in all situations. From the zip manual:

The zip program can do the same matching on names that are in the zip archive being modified or, in the case of the -x (exclude) or -i (include) options, on the list of files to be operated on, by using backslashes or quotes to tell the shell not to do the name expansion.

The original command works on subsequent attempts, after you've successfully created an archive, because zip tries to match the wildcards against the contents of the existing archive. They exist there, and still exist on the filesystem, so they're reported with updating:.

To get zip to handle the wildcards when creating the archive, use the -r (recurse) option to recurse into the requested directory, and -i (include) to limit it to files matching the pattern:

 for i in */ ; do zip -r "zips/${i%/}.zip" "$i" -i '*.csv' ; done