~ glob operator behavior with empty pattern

wildcardszsh

Per the documentation, ~ can be used as glob operator when EXTENDED_GLOB is set, allowing constructs like

print -rl glob*~globA*~glob*B

so anything that matches glob* but does not match globA* and glob*B.
All right:

touch file{1,2,3,4{,5}}
setopt extendedglob

now

print -rl file*~f*3~file4*

behaves as expected:

file1
file2

but when an empty pattern is used

print -rl file*~f*3~~file4*

prints

file1
file2
file3

It looks like only the last pattern is used to filter the results of the 1st glob (2nd and 3rd – which is empty – are ignored).

print -rl file*~f*3~file4*~~

produces

file1
file2
file4
file45

so now only the 2nd one is used…
What's weird is that if the pattern after the 1st tilde is empty it does not work at all e.g.

print -rl file*~~f*3~file4*

errors out with

zsh: no matches found: file*~~f*3~file4*

What am I missing here ?

this is with zsh 5.3.1 if it matters…

Best Answer

~~ doesn't use an empty pattern. Instead, it does one of two things:

At the end of the whole pattern, literally matches "~~".
In the middle, literally matches a single "~", then moves on to the next excluded pattern (i.e. a literal ~ followed by the ~ operator).

If we extend your file set with a few extras we can see that happen:

touch file3~ file3~~ file4~ file4~~

Now running your first odd command:

print -rl file*~f*3~~file4*
file1
file2
file3
file3~~

file3~ has been excluded, as have all three file4*s. file3~~ is still there, because it doesn't match f*3~.

The second one:

print -rl file*~f*3~file4*~~
file1
file2
file3~
file3~~
file4
file45
file4~

Only file4~~ has been excluded, because that's the only one matching file4*~~.

The final case now does have output:

print -rl file*~~f*3~file4*
file3~
file3~~

since those are the two files matching file*~ that don't match either f*3 or file4*.

Perhaps this could be a parsing bug, at least for the literal-then-operator case, but I can't see a use for an empty exclusion pattern so I'm not sure what else it should do.

Related Solutions

How to silently get an empty string from a glob pattern with no matches

Turn on the null_glob option for your pattern with the N glob qualifier.

list_of_files=(*(N))

If you're doing this on all the patterns in a script or function, turn on the null_glob option:

setopt null_glob

This answer has bash and ksh equivalents.

Do not use print or command substitution! That generates a string consisting of the file names with spaces between them, instead of a list of strings. (See What is word splitting? Why is it important in shell programming?)

zsh – How to cp Hidden Files with Glob Patterns

Disclaimer: This answer deals with Bash specifically but much of it applies to the question regarding glob patterns!

The star character (*) is a wildcard. There are a certain set of characters that it will take the place of and the first character being a dot (.) isn't one of them. This is a special case just because of how the Unix filesystems work, files that start with a dot are considered "hidden". That means that tools such as cp, ls, etc. will not "see" them unless explicitly told to do so.

Examples

First let's create some sample data.

$ mkdir .dotdir{1,2} regdir{1,2}
$ touch .dotfile{1,2} regfile{1..3}

So now we have the following:

$ tree -a
.
|-- .dotdir1
|-- .dotdir2
|-- .dotfile1
|-- .dotfile2
|-- regdir1
|-- regdir2
|-- regfile1
|-- regfile2
`-- regfile3

Now let's play some games. You can use the command echo to list out what a particular wildcard (*) would be for a given command like so:

$ echo *
regdir1 regdir2 regfile1 regfile2 regfile3


$ echo reg*
regdir1 regdir2 regfile1 regfile2 regfile3

$ echo .*
. .. .dotdir1 .dotdir2 .dotfile1 .dotfile2

$ echo .* *
. .. .dotdir1 .dotdir2 .dotfile1 .dotfile2 regdir1 regdir2 regfile1 regfile2 regfile3

$ echo .dotdir*
.dotdir1 .dotdir2

Changing the behavior?

You can use the command shopt -s dotglob to change the behavior of the * so that in addition to files like regfile1 it will also match .dotfile1.

excerpt from the bash man page

dotglob If set, bash includes filenames beginning with a `.' in the results 
        of pathname expansion.

Example:

$ shopt -s dotglob
$ echo *
.dotdir1 .dotdir2 .dotfile1 .dotfile2 regdir1 regdir2 regfile1 regfile2 regfile3

You can revert this behavior with this command:

$ shopt -u dotglob
$ echo *
regdir1 regdir2 regfile1 regfile2 regfile3

Your situation?

For you you're telling cp that you want to copy all the files that match the pattern *, and there aren't any files.

$ cp foo/.* .

Or you can do this if you want everything in the foo folder:

$ cp foo .

Or you can be explicit:

$ cp foot/.* foo/* .

A more compact form using brace expansion in bash:

$ cp foo/{.,}* .

At any time you can use the echo trick to see what your proposed file patterns (that's the fancy term for what the star is a part of).

$ echo {.,}*
. .. .dotdir1 .dotdir2 .dotfile1 .dotfile2 abc regdir1 regdir2 regfile1 regfile2 regfile3

Incidentally if you're going to copy a directory of files + other directories, you typically want to do this recursively, that's the -R switch to cp:

$ cp -R foo/. .