Unable to Delete Duplicate all under Utilities

[cafrays]

The bug

When trying to delete duplicate images, I'm getting and error after confirming.

Are you sure you want to bulk trash 109,969 duplicate assets? This will keep the largest asset of each group and trash all other duplicates.

I get the below error.

Not found or no asset.delete access
(Immich Server Error)

The OS that Immich Server is running on

Docker

Version of Immich Server

1.22

Version of Immich Mobile App

1.22

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      #- /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - 2283:2283
    depends_on:
      - redis
      - database
    restart: always
    healthcheck:
      disable: false

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always
    healthcheck:
      disable: false

  redis:
    container_name: immich_redis
    image: docker.io/redis:6.2-alpine@sha256:e3b17ba9479deec4b7d1eeec1548a253acc5374d68d3b27937fcfe4df8d18c7e
    healthcheck:
      test: redis-cli ping || exit 1
    restart: always

  database:
    container_name: immich_postgres
    image: tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    env_file:
      - .env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
    volumes:
      - pgdata:/var/lib/postgresql/data
    healthcheck:
      test: pg_isready --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM p1
      interval: 5m
      start_interval: 30s
      start_period: 5m
    command: ["postgres", "-c", "shared_preload_libraries=vectors.so", "-c", 'search_path="$$user", public, vectors', "-c", "logging_collector=on", "-c", "max_wal_size=2GB", "-c", "shared_buffers=512MB", "-c", "wal_compression=on"]
    restart: always

volumes:
  pgdata:

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=/pics/immich/library

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secrets for postgres and typesense. You should change these to random passwords
TYPESENSE_API_KEY=<redacted>
DB_PASSWORD=<redacted>

# The values below this line do not need to be changed
###################################################################################
DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

REDIS_HOSTNAME=immich_redis

Reproduction steps

1. Run Review duplicates utility
2. Confirm deletion
3. error

Relevant log output

Not found or no asset.delete access
(Immich Server Error)

Additional information

No response

[bo0tzz]

Can you post the server logs?

[cafrays]

Here are 100 lines

docker logs -n 100 immich_server
Error: ENOENT: no such file or directory, access 'upload/thumbs/64b6db82-42c3-49bc-ad75-5b2434cbe6e4/67/08/6708740b-d7a5-452b-9f72-9e57508b16db-thumbnail.webp'
    at async access (node:internal/fs/promises:605:10)
    at async sendFile (/usr/src/app/dist/utils/file.js:54:9)
    at async AssetMediaController.viewAsset (/usr/src/app/dist/controllers/asset-media.controller.js:58:9)
[Nest] 17  - 12/06/2024, 6:11:55 AM   ERROR [Api:LoggerRepository~brqcxs2q] Unable to send file: Error
Error: ENOENT: no such file or directory, access 'upload/thumbs/64b6db82-42c3-49bc-ad75-5b2434cbe6e4/d3/d7/d3d71fa1-3af1-452b-a9c3-98ce5836fc86-thumbnail.webp'
    at async access (node:internal/fs/promises:605:10)
    at async sendFile (/usr/src/app/dist/utils/file.js:54:9)
    at async AssetMediaController.viewAsset (/usr/src/app/dist/controllers/asset-media.controller.js:58:9)
[Nest] 17  - 12/06/2024, 6:11:55 AM   ERROR [Api:GlobalExceptionFilter~brqcxs2q] Unknown error: Error: ENOENT: no such file or directory, access 'upload/thumbs/64b6db82-42c3-49bc-ad75-5b2434cbe6e4/d3/d7/d3d71fa1-3af1-452b-a9c3-98ce5836fc86-thumbnail.webp'
Error: ENOENT: no such file or directory, access 'upload/thumbs/64b6db82-42c3-49bc-ad75-5b2434cbe6e4/d3/d7/d3d71fa1-3af1-452b-a9c3-98ce5836fc86-thumbnail.webp'
    at async access (node:internal/fs/promises:605:10)
    at async sendFile (/usr/src/app/dist/utils/file.js:54:9)
    at async AssetMediaController.viewAsset (/usr/src/app/dist/controllers/asset-media.controller.js:58:9)
[Nest] 17  - 12/06/2024, 6:21:57 AM   ERROR [Api:LoggerRepository~ifqh6jic] Unable to send file: Error
Error: ENOENT: no such file or directory, access 'upload/library/admin/2024/2024-12-03/04ed93ea-30c3-49c5-a45f-733fb615a6e5.mp4'
    at async access (node:internal/fs/promises:605:10)
    at async sendFile (/usr/src/app/dist/utils/file.js:54:9)
    at async AssetMediaController.playAssetVideo (/usr/src/app/dist/controllers/asset-media.controller.js:61:9)
[Nest] 17  - 12/06/2024, 6:21:57 AM   ERROR [Api:GlobalExceptionFilter~ifqh6jic] Unknown error: Error: ENOENT: no such file or directory, access 'upload/library/admin/2024/2024-12-03/04ed93ea-30c3-49c5-a45f-733fb615a6e5.mp4'
Error: ENOENT: no such file or directory, access 'upload/library/admin/2024/2024-12-03/04ed93ea-30c3-49c5-a45f-733fb615a6e5.mp4'
    at async access (node:internal/fs/promises:605:10)
    at async sendFile (/usr/src/app/dist/utils/file.js:54:9)
    at async AssetMediaController.playAssetVideo (/usr/src/app/dist/controllers/asset-media.controller.js:61:9)
[Nest] 17  - 12/06/2024, 6:24:50 AM   ERROR [Api:LoggerRepository~uzq2j5m9] Unable to send file: Error
Error: ENOENT: no such file or directory, access 'upload/thumbs/64b6db82-42c3-49bc-ad75-5b2434cbe6e4/d3/d7/d3d71fa1-3af1-452b-a9c3-98ce5836fc86-thumbnail.webp'
    at async access (node:internal/fs/promises:605:10)
    at async sendFile (/usr/src/app/dist/utils/file.js:54:9)
    at async AssetMediaController.viewAsset (/usr/src/app/dist/controllers/asset-media.controller.js:58:9)
[Nest] 17  - 12/06/2024, 6:24:50 AM   ERROR [Api:GlobalExceptionFilter~uzq2j5m9] Unknown error: Error: ENOENT: no such file or directory, access 'upload/thumbs/64b6db82-42c3-49bc-ad75-5b2434cbe6e4/d3/d7/d3d71fa1-3af1-452b-a9c3-98ce5836fc86-thumbnail.webp'
Error: ENOENT: no such file or directory, access 'upload/thumbs/64b6db82-42c3-49bc-ad75-5b2434cbe6e4/d3/d7/d3d71fa1-3af1-452b-a9c3-98ce5836fc86-thumbnail.webp'
    at async access (node:internal/fs/promises:605:10)
    at async sendFile (/usr/src/app/dist/utils/file.js:54:9)
    at async AssetMediaController.viewAsset (/usr/src/app/dist/controllers/asset-media.controller.js:58:9)
[Nest] 17  - 12/06/2024, 6:24:50 AM   ERROR [Api:LoggerRepository~yrh02k0j] Unable to send file: Error
Error: ENOENT: no such file or directory, access 'upload/thumbs/64b6db82-42c3-49bc-ad75-5b2434cbe6e4/67/08/6708740b-d7a5-452b-9f72-9e57508b16db-thumbnail.webp'
    at async access (node:internal/fs/promises:605:10)
    at async sendFile (/usr/src/app/dist/utils/file.js:54:9)
    at async AssetMediaController.viewAsset (/usr/src/app/dist/controllers/asset-media.controller.js:58:9)
[Nest] 17  - 12/06/2024, 6:24:50 AM   ERROR [Api:GlobalExceptionFilter~yrh02k0j] Unknown error: Error: ENOENT: no such file or directory, access 'upload/thumbs/64b6db82-42c3-49bc-ad75-5b2434cbe6e4/67/08/6708740b-d7a5-452b-9f72-9e57508b16db-thumbnail.webp'
Error: ENOENT: no such file or directory, access 'upload/thumbs/64b6db82-42c3-49bc-ad75-5b2434cbe6e4/67/08/6708740b-d7a5-452b-9f72-9e57508b16db-thumbnail.webp'
    at async access (node:internal/fs/promises:605:10)
    at async sendFile (/usr/src/app/dist/utils/file.js:54:9)
    at async AssetMediaController.viewAsset (/usr/src/app/dist/controllers/asset-media.controller.js:58:9)
[Nest] 17  - 12/06/2024, 6:25:32 AM     LOG [Api:EventRepository] Websocket Disconnect: T2K8zXGfUle0f4tCAAAX
[Nest] 17  - 12/06/2024, 6:25:36 AM     LOG [Api:EventRepository] Websocket Connect:    OhC2uw2quQMn3WzVAAAZ
[Nest] 17  - 12/06/2024, 6:37:08 AM   ERROR [Api:GlobalExceptionFilter] Unknown error: PayloadTooLargeError: request entity too large
PayloadTooLargeError: request entity too large
    at readStream (/usr/src/app/node_modules/raw-body/index.js:163:17)
    at getRawBody (/usr/src/app/node_modules/raw-body/index.js:116:12)
    at read (/usr/src/app/node_modules/body-parser/lib/read.js:79:3)
    at jsonParser (/usr/src/app/node_modules/body-parser/lib/types/json.js:138:5)
    at Layer.handle [as handle_request] (/usr/src/app/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/usr/src/app/node_modules/express/lib/router/index.js:328:13)
    at /usr/src/app/node_modules/express/lib/router/index.js:286:9
    at Function.process_params (/usr/src/app/node_modules/express/lib/router/index.js:346:12)
    at next (/usr/src/app/node_modules/express/lib/router/index.js:280:10)
    at cookieParser (/usr/src/app/node_modules/cookie-parser/index.js:71:5)
[Nest] 17  - 12/06/2024, 6:37:31 AM   ERROR [Api:LoggerRepository~w50r80im] Unable to send file: Error
Error: ENOENT: no such file or directory, access 'upload/thumbs/64b6db82-42c3-49bc-ad75-5b2434cbe6e4/d3/d7/d3d71fa1-3af1-452b-a9c3-98ce5836fc86-thumbnail.webp'
    at async access (node:internal/fs/promises:605:10)
    at async sendFile (/usr/src/app/dist/utils/file.js:54:9)
    at async AssetMediaController.viewAsset (/usr/src/app/dist/controllers/asset-media.controller.js:58:9)
[Nest] 17  - 12/06/2024, 6:37:31 AM   ERROR [Api:GlobalExceptionFilter~w50r80im] Unknown error: Error: ENOENT: no such file or directory, access 'upload/thumbs/64b6db82-42c3-49bc-ad75-5b2434cbe6e4/d3/d7/d3d71fa1-3af1-452b-a9c3-98ce5836fc86-thumbnail.webp'
Error: ENOENT: no such file or directory, access 'upload/thumbs/64b6db82-42c3-49bc-ad75-5b2434cbe6e4/d3/d7/d3d71fa1-3af1-452b-a9c3-98ce5836fc86-thumbnail.webp'
    at async access (node:internal/fs/promises:605:10)
    at async sendFile (/usr/src/app/dist/utils/file.js:54:9)
    at async AssetMediaController.viewAsset (/usr/src/app/dist/controllers/asset-media.controller.js:58:9)
[Nest] 17  - 12/06/2024, 6:37:31 AM   ERROR [Api:LoggerRepository~rgo7sghk] Unable to send file: Error
Error: ENOENT: no such file or directory, access 'upload/thumbs/64b6db82-42c3-49bc-ad75-5b2434cbe6e4/67/08/6708740b-d7a5-452b-9f72-9e57508b16db-thumbnail.webp'
    at async access (node:internal/fs/promises:605:10)
    at async sendFile (/usr/src/app/dist/utils/file.js:54:9)
    at async AssetMediaController.viewAsset (/usr/src/app/dist/controllers/asset-media.controller.js:58:9)
[Nest] 17  - 12/06/2024, 6:37:31 AM   ERROR [Api:GlobalExceptionFilter~rgo7sghk] Unknown error: Error: ENOENT: no such file or directory, access 'upload/thumbs/64b6db82-42c3-49bc-ad75-5b2434cbe6e4/67/08/6708740b-d7a5-452b-9f72-9e57508b16db-thumbnail.webp'
Error: ENOENT: no such file or directory, access 'upload/thumbs/64b6db82-42c3-49bc-ad75-5b2434cbe6e4/67/08/6708740b-d7a5-452b-9f72-9e57508b16db-thumbnail.webp'
    at async access (node:internal/fs/promises:605:10)
    at async sendFile (/usr/src/app/dist/utils/file.js:54:9)
    at async AssetMediaController.viewAsset (/usr/src/app/dist/controllers/asset-media.controller.js:58:9)
[Nest] 17  - 12/06/2024, 6:37:57 AM     LOG [Api:EventRepository] Websocket Disconnect: OhC2uw2quQMn3WzVAAAZ
[Nest] 17  - 12/06/2024, 3:38:00 PM     LOG [Api:EventRepository] Websocket Connect:    TpWAP-FdOqXePyNHAAAb
[Nest] 17  - 12/06/2024, 3:38:01 PM   ERROR [Api:LoggerRepository~pq1er9lc] Unable to send file: Error
Error: ENOENT: no such file or directory, access 'upload/thumbs/64b6db82-42c3-49bc-ad75-5b2434cbe6e4/d3/d7/d3d71fa1-3af1-452b-a9c3-98ce5836fc86-thumbnail.webp'
    at async access (node:internal/fs/promises:605:10)
    at async sendFile (/usr/src/app/dist/utils/file.js:54:9)
    at async AssetMediaController.viewAsset (/usr/src/app/dist/controllers/asset-media.controller.js:58:9)
[Nest] 17  - 12/06/2024, 3:38:01 PM   ERROR [Api:GlobalExceptionFilter~pq1er9lc] Unknown error: Error: ENOENT: no such file or directory, access 'upload/thumbs/64b6db82-42c3-49bc-ad75-5b2434cbe6e4/d3/d7/d3d71fa1-3af1-452b-a9c3-98ce5836fc86-thumbnail.webp'
Error: ENOENT: no such file or directory, access 'upload/thumbs/64b6db82-42c3-49bc-ad75-5b2434cbe6e4/d3/d7/d3d71fa1-3af1-452b-a9c3-98ce5836fc86-thumbnail.webp'
    at async access (node:internal/fs/promises:605:10)
    at async sendFile (/usr/src/app/dist/utils/file.js:54:9)
    at async AssetMediaController.viewAsset (/usr/src/app/dist/controllers/asset-media.controller.js:58:9)
[Nest] 17  - 12/06/2024, 3:38:01 PM   ERROR [Api:LoggerRepository~sycmpryk] Unable to send file: Error
Error: ENOENT: no such file or directory, access 'upload/thumbs/64b6db82-42c3-49bc-ad75-5b2434cbe6e4/67/08/6708740b-d7a5-452b-9f72-9e57508b16db-thumbnail.webp'
    at async access (node:internal/fs/promises:605:10)
    at async sendFile (/usr/src/app/dist/utils/file.js:54:9)
    at async AssetMediaController.viewAsset (/usr/src/app/dist/controllers/asset-media.controller.js:58:9)
[Nest] 17  - 12/06/2024, 3:38:01 PM   ERROR [Api:GlobalExceptionFilter~sycmpryk] Unknown error: Error: ENOENT: no such file or directory, access 'upload/thumbs/64b6db82-42c3-49bc-ad75-5b2434cbe6e4/67/08/6708740b-d7a5-452b-9f72-9e57508b16db-thumbnail.webp'
Error: ENOENT: no such file or directory, access 'upload/thumbs/64b6db82-42c3-49bc-ad75-5b2434cbe6e4/67/08/6708740b-d7a5-452b-9f72-9e57508b16db-thumbnail.webp'
    at async access (node:internal/fs/promises:605:10)
    at async sendFile (/usr/src/app/dist/utils/file.js:54:9)
    at async AssetMediaController.viewAsset (/usr/src/app/dist/controllers/asset-media.controller.js:58:9)

[bo0tzz]

A ton of files are missing, suggesting some sort of storage issue.

[SWB90]

I have the same issue.

Immich running on Truenas Scale: ElectricEel-24.10.0.2
Immich server: v1.122.2

Google photos export via takeout and imported the majority of the files via iOS app + immich-go

"Are you sure you want to bulk trash 74,611 duplicate assets?" returns:

Not found or no asset.delete access (Immich Server Error)

I can manually process the duplicates with no issues.
Permissions on the storage volume are fine and immich has full access.

[mblack4d]

I was typing just now to add to this as I was running into the same error. It might be related to the # of assets. I manually removed enough to get my /duplicates/ below 70k duplicate assets and accidentally hit Shift+Space while manually removing. _{Shift+C was helpful versus finding the trash button every time.}
Anyways - I decided to go for it once and while typing up a comment - I wanted to verify something and now all of my Duplicate Assets are gone!

[umrath]

Same issue here.

[umrath]

I'm not sure why this has been converted into a discussion.
To me it's clearly a bug.

[bo0tzz]

It's a Q&A discussion because from the OP's log it was apparent that there was a problem with their storage.

[umrath]

Ah, I understand.

Sadly, there is something to this problem, because I have the very same isse.
I have about 22k duplicates in 73k files.
I can delete them one by one, without any problem.
I have done that with ~200 duplicates just hammering Shift-c - but that's not really a reasonable solution, hence I would very much like to use the bulk delete option.

Even if there is a problem with the storage, I would expect the option to delete whatever it can and then just leave the remaining duplicates as is.
I'm not aware of any storage issues in my system, though.

[umrath]

My makeshift solution to the problem:

(I placed items to put pressure on Shift and c on the keyboard to essentially auto delete all duplicates.)

[umrath]

After juggling with the setup above and all quirks following that, e.g., the Immich-tab in de browser blew up in memory usage to unseen heights, reaching 80 GB and more, which astonishingly didn't cause the MacBook to crash but essentially halted the tab itself and subsequent adjustments to regularly reload the tab (which in itself caused the issue that it locked my account at some point during the night and required manual intervention to set the password anew), I managed to lower the amount of duplicates to ~14000 by morning.

At that point I was finally able to bulk delete the rest.

Conclusion: There definitely is a bug in the bulk delete functionality for duplicates, which is probably related/triggered by a certain amount of duplicates or something similar.