Better duplicate detection

[saulwold]

Hi There,

Not sure if this is an immich-go idea or immich-app (or maybe both).

I know that I have a lot of duplicates in my 2Tb library with around 396000 images & videos. I have looked at the duplicate command code that immich-go uses, it seems to check filename (case sensitive) and the date. I know that in some cases I have filename.jpg and filename.JPG are the same image, but different filenames and also different EXIF metadata. At a file level the checksums are different, but at an image only checksum they match (strip'ed using exiftool -all= -o - -b filename.jpg | sha256sum).

From the immich-app standpoint I thought maybe adding an imageChecksum, but I have not dug too deeply into that code yet. I am not sure if it's possible to do something with immich-go to handle image level checksums for duplication checking.

Thoughts? Should I post this over in immich-app?

Thanks

[qodesmith]

This would be a fantastic addition to immich-go! I would have much more confidence in detecting duplicates knowing an additional comparison of image-only hashes was added. I can't write Go but I'm thinking about how I might achieve this in JavaScript with Bun.

[simulot]

I missed your post. Sorry.

The immich-go duplicate function search duplicated caused by the compression done by Google Photos. Your phone takes high resolution, high quality picture, but pictures are saved with the same name, but with higher compression.

Merging photos directly took from your phone with the ones coming from a takeout leads to duplicates: same name, but different size. Immich-go keeps the heaviest version.

The Immich app check hashes but not names, but only for its internal library for the current user. You can't upload two identical files.

Remains the external library. I haven't yet studied the possibility to track duplicates in this context. The immich app doesn't check the file hash.

I heard from the immich development team that they will add the possibility to find similar phots using some advanced technology. This addition would probably do a better job than anything else.