Shell – bash/shell pathname expansion for mkdir, touch etc

shellwildcards

So doing something like this in bash and most other shells won't work to create multiple subdirectories or files within subdirectories… 

mkdir */test
touch */hello.txt

There are of course many ways of actually doing this, my preferred one is to use find when possible rather than using a for loop, for readability mainly.

But my question is, why do the above not work?
From what I understand it's because the full destination file/path does not exist but surely that's a good thing if I'm trying to mkdir or touch. I've always just moved on and never really questioned it.
But does anyone have a decent explanation for this that will help me understand once and for all?

Best Answer

The point that you may be missing – that many people have trouble with, especially if they have experience with other operating systems before they come to *nix – is that, in many other OSs, wildcards on the command line are normally passed to the command to process as it sees fit.  For example, in Windows Command Prompt,

rename *.jpeg *.jpg

Whereas, in *nix, in order to simplify the job of the programmer(s) of the individual commands (e.g., mv), wildcards (when not quoted or escaped) are handled by the shell, in a manner that’s independent of the interface and functionality of the command that the wildcards are arguments to.  Most programs that take filenames as command-line arguments expect those files to exist (yes, mkdir and touch are exceptions to that rule, as are mkfifo, mknod, and, to a partial extent, cp, ln, mv, and rename; and there are probably others), so it doesn’t really make sense for the shell to expand wildcards to names for files that don’t exist.  And for the shell (and, by that, I mean every shell – Bourne, bash, csh, fish, ksh, zsh, etc…) to handle the exceptions differently would probably be too much of a hassle.

That said, there are a couple of ways to get a result like what you want.  If you know what the wildcard is going to expand to, and it’s not long, you can generate it with brace expansion:

touch {red,orange,yellow,green,blue,indigo,violet}/rgb.txt

A more general solution:

sh -c 'for arg do mkdir -- "$arg"/test; done' -- *

Gilles helped me find another way to do it in bash. 

benbradley=(*)
mkdir "${benbradley[@]/%//test}"

Obviously benbradley is just an identifier here; you can use any name (e.g., any single letter).  It took me a couple of tries to get this right, so let me break it down:

  • identifier=value creates a (scalar) variable named identifier (if it doesn’t already exist) and assigns the value value to it.  For example, G=Man.  You can reference a scalar variable with $identifier; for example, $G, or, more safely, as ${identifier}.  For example, $Gage and $Gilla may be undefined, but ${G}age is Manage and ${G}illa is Manilla.
  • identifier=(value1 value2 … ) creates an array variable named identifier (if it doesn’t already exist) and assigns the listed values to it.  For example,
      spectrum=(red orange yellow green blue indigo violet)
    or 
      all_text_files=(*.txt)
    $name will reference the first element (and so is fairly useless).  You can reference an arbitrary element as ${name[subscript]}; for example, ${spectrum[0]} is red and ${spectrum[4]} is blue.  And you can use ${name[@]} and ${name[*]} to reference the entire array.

So, here it is in pieces:

      "   ${   benbradley[@]   /   %   /   /test   }   "
  • ${parameter/pattern/string} expands ${parameter} and replaces the longest match of pattern with string.
  • If pattern begins with #, it must match at the beginning of the expanded value of parameter.  If pattern begins with %, it must match at the end of the expanded value of parameter.  In other words,
      %(the-rest_of_the_pattern)
    acts like 
      (the-rest_of_the_regex)$
    (yes, that seems a little backwards).  So a pattern that is just a % is like a regular expression that is just a $ – it matches the end of the input string (search space).
  • And I’m using a string of /test.  So this replaces the end of the parameter with /test (i.e., it appends /test to the parameter).
  • Another thing about ${parameter/pattern/string} that’s non-intuitive is that it always ends with a }.  It need not, and cannot, end with a third /.  Therefore, a / in string cannot be interpreted as a delimiter, and therefore we can have a string of /test without needing to escape the /.
  • If parameter is an array variable subscripted with @ or *, the substitution operation is applied to each member of the array in turn.
  • When you reference an array as ${name[@]} (rather than ${name[*]}) and put the results in double quotes, you preserve the integrity of the elements of the array (i.e., you preserve spaces and other special characters in them) without combining the separate elements into one long word.
Related Question