Returning Randomized Items from Glob Match in ZSH

randomwildcardszsh

With the glob qualifiers in zsh one can sort the results of a filename globbing pattern match in various ways. For example, the pattern *(om) would match all non-hidden names in the current directory, ordered by modification timestamp.

However, I have at times wanted to have a way of having a randomised ordering (for example, to get a random sampling of files). As far as I have seen, there is no qualifier that does this directly.

Question: How may I get a randomised list of pathnames back from a zsh filename globbing pattern?

Best Answer

Use a random sort key (glob qualifier oe)::

*(Noe\''REPLY=$RANDOM,$RANDOM'\')

Explanation:

oe is followed by a one-character delimiter, a chunk of code, and another delimiter. The chunk of code may not contain the delimiter. Special characters need to be escaped so that they are not parsed while parsing the glob qualifiers themselves.
I use ' as the delimiter character (with a backslash because it needs to be escaped), and I wrap the code with ' to protect special characters that may be present. This way I can write arbitrary code as long as it doesn't contain '.
This chunk of code is executed for each matching file name in turn.
REPLY is initially set to the file name and whatever the code sets REPLY to is used as a sort key).

To sample $n elements randomly, add the […] qualifier:

*(Noe\''REPLY=$RANDOM,$RANDOM'\'[1,$n])

Occasionally some elements will get the same sort key, so all permutations are not equally likely, with a slight preference for keeping whatever results from applying the sort function to a list in directory order¹, but the bias is small. I use $RANDOM,$RANDOM as the sort key rather than $RANDOM to reduce the bias: $RANDOM is a 15-bit number and the bias would be noticeable as the number of files approaches 2^15.

Note that $RANDOM is good enough for sampling if the slight bias isn't a concern. It isn't suitable for anything that involves security. If you want a secure random permutation, use GNU coreutils's shuf. (If your favorite OS lacks a native shuf and you don't want to install GNU coreutils for some reason, you can try ibara's reimplementation instead.)

securely_permuted=("${(0)$(printf '%s\0' *(N) | shuf -z))}")

or a simpler version that may run into a command line length limit:

securely_permuted=("${(0)$(shuf -z -- *(N)))}")

¹ _{Experimentally the sort is stable (e.g. *(omoe\''REPLY=1'\') is equivalent to *(om), but the order from just *(oe\''REPLY=1'\') doesn't match *(oN). In any case, it's a small bias in favor of some particular order.}

Related Solutions

Shell – returning strings corresponding to shell glob matching

POSIX shells (e.g., bash, dash, ksh, …) accept a ${var%suffix} construct that can be used to remove the trailing portion from the value of a variable. For instance, if path="sub/file.txt", then ${path%.txt} expands to sub/file.

There's a symmetric construct to remove a prefix: ${var#prefix}. The prefix can be a pattern. Doubling the # strips the longest matching prefix. For example, if path=sub/dir/file.txt, then ${path##*/} expands to file.txt (and ${path#*/} expands to dir/file.txt). The same goes for suffixes and %.

So, in your case you could write this (note that you can't combine the prefix stripping and the suffix stripping into a single expansion, at least not with only POSIX constructs):

for f in sub/*.txt; do
  base=${f%.txt}
  base=${base##*/}
  enscript -b "" -o "$base.ps" "$f"
  ps2pdf "$base.ps" "$base.pdf"
done

Alternatively, the GNU basename command accepts an optional second argument which is the file extension to remove. For instance, if $f is file.txt, then $(basename $f .txt) expands to file.

Note, however, that basename removes all path information except for the last component, so if you want to remove just the extension you have to put that back (see the dirname command).

Disable glob qualifiers in zsh

You can disable the PATTERN(QUALIFIERS) syntax by unsetting the bare_glob_qual option:

setopt no_bare_glob_qual

If the option extended_glob is set (and you should set it, the only reason not to set it is for backward compatibility with rare scripts that use unusual syntax), then there is another syntax for glob qualifiers: PATTERN(#qQUALIFIERS). So you can still use glob qualifiers, which are one of zsh's killer features, but you'll have to type a bit more.

Zsh lets you disable wildcard expansion (globbing) altogether, and this looks like a better choice for you. If a command is prefixed by noglob, then no globbing is performed on its arguments. For example, to be able to type URLs containing ? as arguments to wget, I have alias wget='noglob wget'. If you set alias ag='noglob ag', you can type ag mymethod(param).

If ag takes both a search pattern and file names as arguments, then disabling globbing is not good. If you're able to parse the arguments of ag, then you can perform wildcard expansion on them. I don't know the syntax of ag, so I'll give an example where I assume that ag only takes options that don't take arguments, and that its first non-option argument is a pattern and the rest are files.

ag () {
local i=1
  while [[ ${(P)i} = -* ]]; do ((++i)); done
  if ((i < $#)); then
    set -- "${(@)@[1,$i]}" $~@[$((i+1)),$#]
  fi
}
alias ag='noglob ag'

Best Answer

Related Solutions

Shell – returning strings corresponding to shell glob matching

Disable glob qualifiers in zsh

Related Question