Go n directories down

I'll go with Gilles and point you to Unison as suggested by hasen j. Unison was DropBox 20 years before DropBox. Rock solid code that a lot of people (myself included) use every day -- very worthwhile to learn. Still, join needs all the publicity it can get :)

This is only half an answer, but I have to get back to work :)

Basically, I wanted to demonstrate the little-known join utility which does just that: joins two tables on a some field.

First, set up a test case including file names with spaces:

for d in a b 'c c'; do mkdir -p "old/$d"; echo $RANDOM > "old/${d}/${d}.txt"; done
cp -r old new

(edit some directory and/or file names in new).

Now, we want to build a map: hash -> filename for each directory and then use join to match up files with the same hash. To generate the map, put the following in makemap.sh:

find "$1" -type f -exec md5 -r "{}" \; \
  | sed "s/\([a-z0-9]*\) ${1}\/\(.*\)/\1 \"\2\"/" \

makemap.sh spits out a file with lines of the form, 'hash "filename"', so we just join on the first column:

join <(./makemap.sh 'old') <(./makemap.sh 'new') >moves.txt

This generates moves.txt which looks like this:

49787681dd7fcc685372784915855431 "a/a.txt" "bar/a.txt"
bfdaa3e91029d31610739d552ede0c26 "c c/c c.txt" "c c/c c.txt"

The next step would be to actually do the moves, but my attempts got stuck on quoting... mv -i and mkdir -p should come handy.

This is not a job for find, since there is no recursion involved.

for x in */C/populate.sh; do
  mkdir -- "${x%%/*}$suffix"
  mkdir -- "${x%%/*}$suffix/C"
  cp -p -- "$x" "./${x%%/*}$suffix/C"
done

Remove the -p option to cp if you don't want to preserve the files' modification time.

To generate a random suffix, BSD/Linux mktemp is as portable as it gets.

suffix=$(mktemp -u tmp_XXXXX)

If you want something vaguely random-looking and POSIX-compliant, this gives a string that changes every second and varies from location to location; you can't really do better with only POSIX tools:

suffix=$({ hostname; pwd; date; } |
         cksum | uuencode -m /dev/stdin | awk 'NR==2 {print substr($0,3,5)}')

If you put this code in a makefile, remember to:

• double all $ signs;
• put all the code on one line, using ; instead to separate shell instructions (you can use backslash+newline+tab to put a line break in the makefile, but that sequence is removed to build the shell command);
• start the shell snippet with set -e, so that it aborts if there is any error.