How to scan for duplicate files / images in one direction only

finderphotosscanning

I am looking for a solution to search for duplicate files / images on my mac but only in one direction: Folders A and B should be compared but I am only interested in files in folder A with duplicates in folder B. If there are any duplicates within B does not matter.

In detail:

On my Mac I have large collection of photos and images. Lets say they are all located in /User/JustMe/PhotoCollection/ or its sub folders.

Within these folders there same known duplicates which where created intentionally. For example I have one folder Summer Vacationand a sub folder which holds all vacation photos which should be used for a photo book. Thus I am not interested in finding these duplicates.

When I import photos from my camera it can happen that some of them have been imported before. Thus I would like to compare the import folder to the existing photos folder and find only duplicates

within the import folder (both files are located in the import folder)
or one version within the import folder and one within the existing photos folder

I am not interested in duplicates within the existing photos.

I tried some known duplicate scanners like Gemini, Cisdem, etc. They all allow to select multiple folders to compare but they don't seem to have an oneway option.

Can this be solved with macOS features or is there any tools which offers this option?

EDIT:

As requested here is an example:

/User/JustMe/PhotoCollection/
    SubfolderA/
       Image1.jpg
    SubfolderB/
       ImageA.jpg
       ImageB.jpg
       Selection/
           ImageA.jpg
    SubfolderC/
    ...

/User/JustMe/PhotoImport/
    NewImage1.jpg
    NewImage2.jpg
    NewImage2_Copy.jpg
    ImageA.jpg

Potential duplicates:

/User/JustMe/PhotoCollection/SubfolderB/ImageA.jpg    
/User/JustMe/PhotoCollection/SubfolderB/Selection/ImageA.jpg

/User/JustMe/PhotoImport/NewImage2.jpg
/User/JustMe/PhotoImport/NewImage2_Copy.jpg

/User/JustMe/PhotoImport/ImageA.jpg
/User/JustMe/PhotoCollection/SubfolderB/ImageA.jpg

The first duplicate is within the existing collection and is a known duplicates which should not be deleted / scanned for.

The second duplicate is within the import folder and should be deleted

The 3rd duplicate has one version in the import folder and one in the existing collection. The import version should be deleted.

All duplicate scanners I have tests do not only test the file name but also the capture date, other raw data etc. to make sure files are equal. Additionally they can scan for similar pictures which are not 100% identical but almost the same.

Best Answer

It looks like you're into photography and you haven't been keen for best practices for managing files. You have two options now.

1) You'd have to create (bash OR python script) which would look for MD5 checksum of the files and would delete only it finds duplicate with later date. Keep in mind, one careless step can delete the files you didn't want to delete.

2) (Suggested) There are several free and paid utilities that should be used to cleanup the duplicates. If it provides duplicates with date comparison. As you already mentioned (Gemini). I tested it few months ago and found it very useful.

There are several proven practices to manage the photo library. This is a good time to adapt one.

As you requested, here is a script which find duplicates in one folder.

#!/usr/bin/env python
# Syntax: duplicates.py DIRECTORY
import os, sys
top = sys.argv[1]
d = {}

for root, dirs, files in os.walk(top, topdown=False):
  for name in files:
    fn = os.path.join(root, name)
    basename, extension = os.path.splitext(name)

    basename = basename.lower() # ignore case

    if basename in d:
        print(d[basename])
        print(fn)
    else:
        d[basename] = fn

Save this file as duplicates.py and give it rights and then execute it on the folder.

./duplicates.py Images

It requires you to have an understanding for the osx terminal.

Related Solutions

MacOS – How to select all files within a directory with subfolders

I created an Automator application that does what you're looking for. You can download it here.

It's pretty simple, just drag-and-drop any folder onto the application, it will then find all images in the folder and any subfolders, and prompt you for a destination to copy them to.

If you'd prefer to activate it by a hotkey or the right-click menu, you can make it into a service by following these steps:

Open the application in Automator.
Choose File → Convert To… and select Service.
Change the first drop-down at the top to folders.
Save it and give it a name.
Optional: open the Services menu within Keyboard Preferences. You can assign a keyboard shortcut there.

Once you've done that, you should be able to activate the service by right clicking on any folder, or hitting your shortcut key with a folder selected.

Selecting files with one extensions that have the same name as files with a different extension

Open terminal and use cd to make the directory of *.nef files your current directory. Let's assume you have this structure:

mydir/jpegdir mydir/nefdir

So cd into mydir/nefdir

Then:

mkdir newdir
for file in *.nef; do 
    root=${$file%.nef}; 
    jpeg=${root}.jpg; 
    mv ../jpegdir/${jpeg} ./newdir/
done

example below. Note here that I put "echo" in front of the mv command. You want to omit that, but I often use it when I'm checking that a script is doing what I expect before executing it. You also don't say what you want to do with the files, but moving them to a new folder allows you to select them, and you can always move them all back. I also shortened my tmp variable names.

$ ls -R
total 0
drwxrwxr-x  4 tim  staff  136 Jun 13 08:02 .
drwx------  9 tim  staff  306 Jun 13 08:02 ..
drwxrwxr-x  8 tim  staff  272 Jun 13 08:03 jpegdir
drwxrwxr-x  5 tim  staff  170 Jun 13 08:03 nefdir

./jpegdir:
total 0
drwxrwxr-x  8 tim  staff  272 Jun 13 08:03 .
drwxrwxr-x  4 tim  staff  136 Jun 13 08:02 ..
-rw-rw-r--  1 tim  staff    0 Jun 13 08:03 foo1.jpeg
-rw-rw-r--  1 tim  staff    0 Jun 13 08:03 foo2.jpeg
-rw-rw-r--  1 tim  staff    0 Jun 13 08:03 foo3.jpeg
-rw-rw-r--  1 tim  staff    0 Jun 13 08:03 foo4.jpeg
-rw-rw-r--  1 tim  staff    0 Jun 13 08:03 foo5.jpeg
-rw-rw-r--  1 tim  staff    0 Jun 13 08:03 foo6.jpeg

./nefdir:
total 0
drwxrwxr-x  5 tim  staff  170 Jun 13 08:03 .
drwxrwxr-x  4 tim  staff  136 Jun 13 08:02 ..
-rw-rw-r--  1 tim  staff    0 Jun 13 08:03 foo2.nef
-rw-rw-r--  1 tim  staff    0 Jun 13 08:03 foo3.nef
-rw-rw-r--  1 tim  staff    0 Jun 13 08:03 foo5.nef

$ cd nefdir/
$ mkdir newdir/

$ for f in *.nef; do g=${f%.nef}; h=$g.jpeg; echo mv ../jpegdir/$h ./newdir; done
mv ../jpegdir/foo2.jpeg ./newdir
mv ../jpegdir/foo3.jpeg ./newdir
mv ../jpegdir/foo5.jpeg ./newdir

Best Answer

Related Solutions

MacOS – How to select all files within a directory with subfolders

Selecting files with one extensions that have the same name as files with a different extension

Related Question