Shell – Recursively compare directory contents by name, ignoring file extensions

diff()directoryfilenamesshell-script

I have a directory containing about 7,000 music files. I used lame to recursively re-encode all files in it to a separate directory, outputting all files with the same relative path and file name. The output files have a .mp3 extension, but some of the input files had different extensions (.wma, .aac, etc).

I can see that there is a file count difference of ~100 files missing in the output directory. What I want to do is run a compare of the two directories and obtain a list of the files that exist in the source, but not in the destination. This would be simple enough except I need to ignore differences in file extension.

I've tried using rsync with dry-run turned on but I couldn't figure out a way to ignore file extensions. I've also tried diff but was unable to find an option to only check by name but ignore file extensions. I started thinking I could just do a recursive ls on both directories, remove the file extensions, and then compare the outputs, but I really have no idea on where to start with modifying the ls output using sed or awk.

Best Answer

To see a listing, here are two variants, one that recurses into subdirectories and one that doesn't. All use syntax specific to bash, ksh and zsh.

comm -3 <(cd source && find -type f | sed 's/\.[^.]*$//' | sort) \
        <(cd dest && find -type f | sed 's/\.[^.]*$//' | sort)
comm -3 <(cd source && for x in *; do printf '%s\n' "${x%.*}"; done | sort) \
        <(cd dest && for x in *; do printf '%s\n' "${x%.*}"; done | sort)

Shorter, in zsh:

comm -3 <(cd source && print -lr **/*(:r)) <(cd dest && print -lr **/*(:r))
comm -3 <(print -lr source/*(:t:r)) <(print -lr dest/*(:t:r))

The comm command lists the lines that are common to two files (comm -12), that are only in the first file (comm -23) or that are only in the second file (comm -13). The numbers indicate what is subtracted from the output¹. The two input files must be sorted.

Here, the files are in fact the output of a command. The shell evaluates the <(…) construct by providing a “fake” file (a FIFO or a /dev/fd/ named file descriptor) as the argument to the command.

¹ _{So here the minus sayers are fully justified.}

If you want to perform actions on the files, you'll probably want to iterate over the source files.

cd source
for x in *; do
  set -- "…/dest/${x%.*}".*
  if [ $# -eq 1 ] && ! [ -e "$1" ]; then
    echo "$x has not been converted"
  elif [ $# -gt 1 ]; then
    echo "$x has been converted to more than one output file: " "$@"
  else
    echo "$x has been converted to $1"
  fi
done

Related Solutions

Compare an old file and new file, but ignore lines which only exist in new file

Use join to combine matching lines from the two files. Assuming the file names come after the checksums (as in md5sum output) and don't contain whitespace, this will print all file names that are present in both lists, together with the old checksum and the new checksum:

join -1 2 -2 2 <(sort -k 2 oldlist) <(sort -k 2 newlist)

To also see new files, pass the -a option to join. A bit of output postprocessing will remove the file names for which the checksum has not changed.

join -a 2 -1 2 -2 2 <(sort -k 2 oldlist) <(sort -k 2 newlist) |
awk '$2 != $3'

Bash – Manipulate file name piped from find command

As other answers have already identified, ${f%.*} is expanded by the shell before it runs the xargs command. You need this expansion to happen once for each file name, with the shell variable f set to the file name (passing -I f doesn't do that: xargs has no notion of shell variable, it looks for the string f in the command, so if you'd used e.g. xargs -I e echo … it would have executed commands like ./somedir/somefile.wmacho .mp3).

Keeping on this approach, tell xargs to invoke a shell that can perform the expansion. Better, tell find — xargs is a largely obsolete tool and is hard to use correctly, as modern versions of find have a construct that does the same job (and more) with fewer plumbing difficulties. Instead of find … -print0 | xargs -0 command …, run find … -exec command … {} +.

find . -name '*.wma' -type f -exec sh -c 'for f; do echo "${f%.*}.mp3"; done' _ {} +

The argument _ is $0 in the shell; the file names are passed as the positional arguments which for f; do … loops over. A simpler version of this command executes a separate shell for each file, which is equivalent but slightly slower:

find . -name '*.wma' -type f -exec sh -c 'echo "${0%.*}.mp3"' {} \;

You don't actually need to use find here, assuming you're running a reasonably recent shell (ksh93, bash ≥4.0, or zsh). In bash, put shopt -s globstar in your .bashrc to activate the **/ glob pattern to recurse in subdirectories (in ksh, that's set -o globstar). Then you can run

for f in **/*.wma; do
  echo "${f%.*}.mp3"
done

(If you have directories called *.wma, add [ -f "$f" ] || continue at the beginning of the loop.)

Best Answer

Related Solutions

Compare an old file and new file, but ignore lines which only exist in new file

Bash – Manipulate file name piped from find command

Related Question