ZSH wildcard expression limiting repetition support

You don't get a warning in bash, you get an error by ls (you'll find that ls exit status is non-0 which indicates it has failed).

In both zsh and bash, {...} is not a globbing operator, it's an expansion that occurs before globbing.

In:

ls -d -- *.{dot,svg,err}

(you forgot the -d and -- btw), the shell expands the {...} first:

ls -d -- *.dot *.svg *.err

and then does the glob. bash like most Bourne-like shells has that misfeature that non-matching globs are passed as-is. While on zsh, a non-matching glob is an error.

See how rm -f [ab].c in bash could delete the [ab].c file if there was no file called a.c nor b.c. In zsh, you'd get a no match error instead. See the failglob option in bash to get a similar behaviour.

ls -d -- *.{dot,svg,err}(N)

in zsh would enable the nullglob option on all 3 globs, so that if the globs don't match, they are removed, but that's probably not what you want, because if none of the globs match any file, the command will become:

ls -d --

Which will list . (the current directory) instead.

Best here is to use one glob that matches files with either of the 3 extensions:

ls -d -- *.(dot|svg|err)

That will give a sorted list of files to ls, ls will be run unless there's no file found matching that one pattern.

You also have the option to enable the sh/bash (bogus IMO) behaviour with emulate sh or with unsetopt nomatch. A slightly better approach is to enable the csh behaviour (which was also the behaviour of Unix shells before the Bourne shell was released):

setopt cshnullglob

With that option, the command is cancelled only if all the globs on the command line fail to match. If at least one matches, all the ones that don't match are removed so:

ls -d -- *.{dot,svg,err}

Will expand the dot, svg and err in turn, omitting the missing ones.

If you want to compare the effect on the order of the arguments of the different approaches, you need a command that (contrary to ls) doesn't sort them before displaying. With GNU ls, you can pass the -U option for that, or since ls does only print its arguments here, just use printf '%s\n' *... instead.

To match a literal ] and a literal - in a Bracket Expression you'll have to use them like this:

[^]/\^:-]

or, even better, since some tools require the backslash to be escaped:

[^]/\\^:-]

that is
The right-square-bracket ( ']' ) shall lose its special meaning and represent itself in a bracket expression if it occurs first in the list (after an initial '^', if any)
and
The hyphen-minus character shall be treated as itself if it occurs first (after an initial '^', if any) or last in the list
hence
If a bracket expression specifies both '-' and ']', the ']' shall be placed first (after the '^', if any) and the '-' last within the bracket expression.
The rules for bracket expressions are the same for ERE and BRE.