Remove Duplicate Files on OSX Using Find and Regular Expressions

findosxregular expression

On my Mac OS X 10.6.7, iTunes duplicated every single file in my music library. Now I have 3,840 files instead of 1,920. The problem is that iTunes didn't simply duplicate the whole folder, but each file in each folder, adding a 1 to the duplicate. It looks like this:

├── album1
│   ├── track1.mp3
│   ├── track1 1.mp3
│   ├── track2.mp3
│   └── track2 1.mp3
└── album2
    ├── track1.m4a
    └── track1 1.m4a

Now rather than clicking on 1,920 duplicates, I want to enter some smart line in the Terminal to get rid of the extra files:

find path/to/music/ -name "* 1.*" -delete

Though this will work in most cases, it will mess up things in the following situation:

└── album2
    ├── track 1.mp3
    └── track 1 1.mp3

Maybe I should exclude "* 1 1.*" and later on rename those files removing the extra 1?

How do I do this?

Is there a simple solution to this?

Best Answer

Something like this, haven't tested; it's in bash so you may have to convert some syntax:

IFS=$'\n'              # so that only newlines separate words, not spaces
set -f                 # disable globbing
FILES=$(find path/to/music/ -name "* 1.*")

for FILE in ${FILES}; do
    if [[ -f "${FILE% 1.*}" ]] ; do
        echo "Matched ${FILE}."
        # rm "${FILE}" # Uncomment me once you have confirmed it would do what you intend.
    fi
done

${FILE% 1.*} strips the last match of the 1.* syntax from the end, [[ -f ... ]] checks whether that file would exist; therefore, it would remove the files for which the file without the syntax at the end would exist. Please test before uncommenting rm, to be sure it is correct.

Related Solutions

Shell – Find, count and sort all audio files. ALAC (M4A) files

You should decompose your goal into several steps easier to solve. This will have two advantages:

It will be easier to solve,
The resulting code will be clearer and more reusable.

The scripts below basically follows these steps:

Generate raw statistic files. An easy way is to append the file size and the file name in a temporary file named after the original file extension. So, if you have the file /path/to/foo.mp3 which is 3000000 large, it will append 3000000 /path/to/foo.mp3 at the end of a temporary file named mp3.
Handle specific cases. Here it will process the temporary file m4a and create the two other files m4a_aac and m4a_alac based on the test you gave in the question.
Generate output. All the needed information being now available, it just has to:
- Count the number of line in each temporary file to determine the number of file of this type,
- Sum up each size to get the total size of files of this type.

Here is the script:

#!/bin/sh

# This script takes the searched directory as first parameter.
# For instance: ./this-script.sh ~/Music

: ${1:?"You must pass the search directory as first parameter."}
searchdir="$1"

# Create a temporary directory
statsdir=""
trap 'rm -rf $statsdir' EXIT
statsdir=$(mktemp -d "/tmp/tmp.XXXXXXXXXX") || exit 1

# Generate one listing file per extension
awkscript='/\.[[:alnum:]]+$/ {print $0 >statsdir"/"$(NF)}'
# For Linux: stat -c "%s %n"
# For Mac: stat -f "%z %N"
find "$searchdir" -type f -exec stat -f "%z %N" {} + | \
    awk -F '.' -v statsdir="$statsdir" "$awkscript"

# Distinguish between m4a/AAC and m4a/ALAC
if [ -f "$statsdir/m4a" ]; then
    input="$statsdir/m4a"
    while IFS= read -r line; do
        filename=${line#* }
        if avprobe "$filename" 2>&1 | grep -q 'Audio: alac'; then
            echo "$line" >> "$statsdir/m4a_alac"
        else
            echo "$line" >> "$statsdir/m4a_aac"
        fi
    done < "$input"
    rm "$statsdir/m4a"
fi

# Generate and display result
{
    printf "Type Count Size\n"
    for extension in $(ls "$statsdir"); do
        count=$(wc -l "$statsdir/$extension" | cut -d ' ' -f 1)
        totalsize=$(awk '{s+=$1} END {print s}' "$statsdir/$extension")
        printf "%s %d %d\n" "$extension" "$count" "$totalsize"
    done
} | column -t

Best Answer

Related Solutions

Shell – Find, count and sort all audio files. ALAC (M4A) files

Related Question