Ubuntu – How to remove duplicate imported photos in Shotwell

duplicatephoto-managementshotwelltag

I've noticed that Shotwell has imported many images twice (e.g. from my camera SD card). Apparently the duplicate detection is buggy once a photo is imported, tagged and then re-imported.

I have "write meta data tags" enabled in the settings. If I import a photo test-images.jpg and add tags to it the photo will not be picked up by the duplicate detection upon another import of the same file.
The second time the file is imported it will be named test-images-1.jpg and placed in the library folder as per the active rules (not necessarily into the same folder).

test-images.jpg and test-images-1.jpg will have the same image data but due to the added tag/metadata the files are not the same anymore and won't be picked up by searching for duplicates (e.g. md5 hash).

My usage scenario that caused multiple duplicate is as follows:

I take pictures with my phone
I import the photos from my phone, add tags but leave the images on the phone as I want to keep them for sharing etc.
I add further tags to the imported photos
After some weeks I repeat the import step from the phone and old photos that I have already imported will be imported again (with '-1.jpg' or '-2.jpg' added)

How to clean up the duplicates?
Using a file name based search would be possible but I can't exclude that I have not imported a file ending with -1 to which was not imported as a duplicate.

How can I clean up my photo library? I tried to use the search function in Shotwell but with more than 1000 photos there must be a better, more reliable, less error prone an simpler way.

I'm not to worried about tags getting lost, typically the second import (the duplicate) has no tags applied.

Best Answer

Sort of spamish, but I found myself with the same problem a few monts ago, and I wrote a small utility that does just that:

https://github.com/jesjimher/imgdupes

It's a python script that scans a directory tree looking for duplicates. Its syntax is intentionally similar to fdupes, with the difference that imgdupes ignores all metadata and analyzes only the image data chunk of a JPEG file. This means that two different versions of the same image, with different tags, rotation flags, dates, etc., will be reported as duplicates even if physical files are different (and thus not detected as duplicates by fdupes/shotwell).

It was recently renamed to jpegdupes, and is now on Pypi repos, so scanning a tree for duplicated images might be done like this:

sudo pip install jpegdupes jpegdupes -d ~/Photos/ (or whatever your path is)

It would look for JPEGs which are actually the same picture (differing only in metadata), and would interactively show differences and ask for which version to keep.

Hope it helps.

Related Solutions

Ubuntu – Shotwell does not copy, but imports files without copying

When you import a folder with pictures into Shotwell it gives you two options:

enter image description here

In the first case, Copy Photos, Shotwell will store a local copy of the pictures in your computer. Thus, if you import from a removable drive, you will always have a copy of them available.

In the second case, Import in Place, shotwell will only create links to the pictures, thus if you unplug a removable drive with the pictures they will not be available.

If you want to preserve your structure of folders for your pictures I recommend that you copy your pictures manually to your hard drive (Pictures folder is a good option). After that you can Import in Place from shotwell (does not make sense to have 2 local copies of your pictures) and you will have them available. Shotwell will still classify them by date (default), but then you can 'Tag' your pictures if you want to arrange them in a different order (e.g. 'my birthday').

Ubuntu – How does Shotwell detect duplicates

I believe it is more advanced than simple names, I just tried. In fact it would seem that it doesn't base it on name at all.

So I just created the following:

TestDir
  │
  ├─ blue.png #Blue A4 page
  ├─ blue2.png #Blue A4 page
  ├─ green.png #Blue A4 page
  ├─ red.png #Yellow A4 page
  ├─ yellow.png #Yellow A4 page
  ├─ yellow2.png #Yellow A4 page
  └ TestDir2
       │
       ├─ yellow.png #Blue A4 page
       └─ yellow.png #Blue A4 page

Imported the folder TestDir (which imports from any subdirectories too). This was the notice:

6 duplicate photos were not imported:
/home/tim/Pictures/Test/red.png
/home/tim/Pictures/Test/yellow2.png
/home/tim/Pictures/Test/green.png
/home/tim/Pictures/Test/blue2.png
(and 2 more)

2 photos successfully imported.

The two it had imported were blue.png and yellow.png. This is because they were created first (it chooses the oldest if there are duplicates).

This was confirmed by the next test:

TestDir
  │
  ├─ blue.png #Blue A4 page
  ├─ blue2.png #Blue A4 page
  ├─ green.png #Blue A4 page (sorry OCD people)
  ├─ red.png #Yellow A4 page (sorry OCD people)
  ├─ yellow.png #Yellow A4 page
  ├─ yellow2.png #Yellow A4 page
  ├─ pink2.png #Pink A4 page 
  ├─ pink.png #Pink A4 page
  └ TestDir2
       │
       ├─ yellow.png #Blue A4 page
       └─ yellow.png #Blue A4 page

pink2.png and pink.png have been created. pink2.png was created first, then pink.png

8 duplicate photos were not imported:
/home/tim/Pictures/Test/red.png
/home/tim/Pictures/Test/yellow2.png
/home/tim/Pictures/Test/green.png
/home/tim/Pictures/Test/blue2.png
(and 4 more)

3 photos successfully imported.

The successful imported ones were blue.png,yellow.png and pink2.png.

Because of that I assume it uses a hashing algorithm.

It is accurate enough that changing just 1 pixel of colour from green to yellow on an A4 page caused it to not detect as a duplicate. Pretty accurate then!

In fact, I just found this post here:

Once this bug is fixed, Shotwell will use (exclusively) a full MD5 hash of the file to perform duplicate checking, which is a pretty darn accurate way to find duplicates.

In fact in the source code, at line 732 is this: ^{_{Kudos @ Jeremie Miserez}}

imported_full_md5_table.has_key(prepared_file.full_md5)) {

Sounds like it uses a MD5 hash!

_{Shapes for directory tree from here}

My pronouns are He / Him

Best Answer

Related Solutions

Ubuntu – Shotwell does not copy, but imports files without copying

Ubuntu – How does Shotwell detect duplicates

Related Question