Bash Wildcards – How to Exclude One Pattern from Glob Match

I'd solve this problem by using GNU Make's filter-out and wildcard functions. The only part of your task that you can't do with them is filter out directories; that has to be done via the shell. Code below is untested and assumes (a) no whitespace or shell metacharacters in any filename, (b) TestData/*.dat and TestVectors/*.txt do not need to be checked for directories.

NORM_TOPLEVEL := $(shell for f in $(filter-out GNUMakefile,$(wildcard *)); \
                   do [ -d "$$f" ] || printf '%s\n' "$$f"; done)
NORM_TESTDIRS := $(wildcard TestData/*.dat) $(wildcard TestVectors/*.txt)

convert:
    chmod a-x $(NORM_TOPLEVEL) $(NORM_TESTDIRS)
    unix2dos --keepdate --quiet $(NORM_TOPLEVEL) $(NORM_TESTDIRS)
    dos2unix --keepdate --quiet GNUmakefile

.PHONY: convert

~~ doesn't use an empty pattern. Instead, it does one of two things:

• At the end of the whole pattern, literally matches "~~".
• In the middle, literally matches a single "~", then moves on to the next excluded pattern (i.e. a literal ~ followed by the ~ operator).

If we extend your file set with a few extras we can see that happen:

touch file3~ file3~~ file4~ file4~~

Now running your first odd command:

print -rl file*~f*3~~file4*
file1
file2
file3
file3~~

file3~ has been excluded, as have all three file4*s. file3~~ is still there, because it doesn't match f*3~.

The second one:

print -rl file*~f*3~file4*~~
file1
file2
file3~
file3~~
file4
file45
file4~

Only file4~~ has been excluded, because that's the only one matching file4*~~.

The final case now does have output:

print -rl file*~~f*3~file4*
file3~
file3~~

since those are the two files matching file*~ that don't match either f*3 or file4*.

Perhaps this could be a parsing bug, at least for the literal-then-operator case, but I can't see a use for an empty exclusion pattern so I'm not sure what else it should do.