Shell – How to execute many files in the same directory in a minimal, DRY, pretty way

TL,DR:

rsync -am --include='*.pdf' --include='*/' --exclude='*' ~/LaTeX/ ~/Output/

Rsync copies the source(s) to the destination. If you pass *.pdf as sources, the shell expands this to the list of files with the .pdf extension in the current directory. No recursive traversal happens because you didn't pass any directory as a source.

So you need to run rsync -a ~/LaTeX/ ~/Output/, but with a filter to tell rsync to copy .pdf files only. Rsync's filter rules can seem daunting when you read the manual, but you can construct many examples with just a few simple rules.

• Inclusions and exclusions:

  • Excluding files by name or by location is easy: --exclude=*~, --exclude=/some/relative/location (relative to the source argument, e.g. this excludes ~/LaTeX/some/relative/location).
  • If you only want to match a few files or locations, include them, include every directory leading to them (for example with --include=*/), then exclude the rest with --exclude='*'. This is because:
  • If you exclude a directory, this excludes everything below it. The excluded files won't be considered at all.
  • If you include a directory, this doesn't automatically include its contents. In recent versions, --include='directory/***' will do that.
  • For each file, the first matching rule applies (and anything never matched is included).

• Patterns:

  • If a pattern doesn't contain a /, it applies to the file name sans directory.
  • If a pattern ends with /, it applies to directories only.
  • If a pattern starts with /, it applies to the whole path from the directory that was passed as an argument to rsync.
  • * any substring of a single directory component (i.e. never matches /); ** matches any path substring.

• If a source argument ends with a /, its contents are copied (rsync -r a/ b creates b/foo for every a/foo). Otherwise the directory itself is copied (rsync -r a b creates b/a).

Thus here we need to include *.pdf, include directories containing them, and exclude everything else.

rsync -a --include='*.pdf' --include='*/' --exclude='*' ~/LaTeX/ ~/Output/

Note that this copies all directories, even the ones that contain no matching file or subdirectory containing one. This can be avoided with the --prune-empty-dirs option (it's not a universal solution since you then can't copy a directory even by matching it explicitly, but that's a rare requirement).

rsync -am --include='*.pdf' --include='*/' --exclude='*' ~/LaTeX/ ~/Output/

The script:

#!/bin/bash

# Get Prefixes

PREFIXES=$(ls | grep -Po '^(.*)(?!HT\d{4})-(.*)-(.*).tar.bz2$' | awk -F'-' '{print $1}' | uniq)

if [ -z "$1" ]; then
  echo need a number of keep files.
  exit 1
else
  NUMKEEP=$1
fi

for PREFIX in ${PREFIXES}; do

  ALL_FILES=$(ls -t ${PREFIX}*)

  if [ $(echo ${ALL_FILES} | wc -w) -lt $NUMKEEP ]; then
    echo Not enough files to be kept. Quit.
    continue
  fi

  KEEP=$(ls -t ${PREFIX}* | head -n${NUMKEEP})

  for file in $ALL_FILES ; do
    if [[ "$KEEP" =~ "$file" ]]; then
      echo keeping $file
    else
      echo RM $file
    fi
  done
done

Explanation:

• Calculate the prefixes:
  • Look for all files following the something-something-something.tar.bz2 regex, cutting of only the first part up to the first dash and make it unique.
  • the result is a normalized list of the PREFIXES
• Iterate through all PREFIXES:
• Calculate ALL_FILES with PREFIX
• Check if the amount of ALL_FILES is less than the number of files to be kept -> if true, we can stop here, nothing to remove
• Calculate the KEEP files which are the most recent NUMKEEP files
• Iterate through ALL_FILES and check if the given file is not in the KEEP file list. If so: remove it.

Example result when running it:

$ ./remove-old.sh 2
keeping bar-01-01.tar.bz2
keeping bar-01-02.tar.bz2
RM bar-01-03.tar.bz2
RM bar-01-04.tar.bz2
RM bar-01-05.tar.bz2
RM bar-01-06.tar.bz2
keeping foo-01-06.tar.bz2
keeping foo-01-05.tar.bz2
RM foo-01-04.tar.bz2
RM foo-01-03.tar.bz2
RM foo-01-02.tar.bz2

$ ./remove-old.sh 8
Not enough files to be kept. Quit.
Not enough files to be kept. Quit.