Worker exited

[Thloluvin]

The bug

The assests are not loaded when a specific album is opened. After about a minute I receive a 502 Error.
A review of the logs shows that the worker has exited.

Going back to the overview and open again this album the same happens.
There are approx. 500 assets in this album. Other albums with more assets can be opened without any problems, but a delay.
However, it sometimes takes up to 30 seconds for the assets to be displayed, which is new.
In the past, I was able to open albums without any major delays.
This problem and the delay only occur with the web version.

With the app, all albums are shown immediately without any delay and without any error message.

The OS that Immich Server is running on

Ubuntu 22.10

Version of Immich Server

v1.119.1

Version of Immich Mobile App

1.119.0 build.164

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.transcoding.yml
    #   service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    volumes:
      # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
      - /path/4_MA-Verein:/usr/src/app/ma:ro
      - /path/5_Musical:/usr/src/app/musical:ro
      - /path/6_KulturhausWartberg:/usr/src/app/khw:ro
      - /path/7_Geburtstage:/usr/src/app/bdays:ro
      - /path/8_Hochzeiten&JGA:/usr/src/app/hochzeit:ro
      - /path/9_Konzerte&Festivals:/usr/src/app/konzerte:ro
      - /path/10_Lanpartys:/usr/src/app/lan:ro
      - /path/11_Lost_Places:/usr/src/app/lostplaces:ro
      - /path/12_Partys:/usr/src/app/partys:ro
      - /path/13_Radfahren:/usr/src/app/radfahren:ro
      - /path/14_Sylvester:/usr/src/app/sylvester:ro
      - /path/15_Urlaube:/usr/src/app/urlaub:ro
      - /path/16_Wandern:/usr/src/app/wandern:ro

    env_file:
      - stack.env
    ports:
      - 2283:2283
    depends_on:
      - redis
      - database
    restart: always
    healthcheck:
      disable: false
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.immich.entrypoints=websecure"
      - "traefik.http.routers.immich.rule=Host(`immich.domain.eu`)"
      - "traefik.http.routers.immich.tls=true"
      - "traefik.http.routers.immich.tls.certresolver=http_resolver"
      - "traefik.http.routers.immich.service=immich"
      - "traefik.http.services.immich.loadbalancer.server.port=2283"
      - "traefik.docker.network=proxy"
      - "traefik.http.routers.immich.middlewares=default@file"
    networks:
      - immich
      - proxy

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - stack.env
    restart: always
    healthcheck:
      disable: false
    networks:
      - immich

  redis:
    container_name: immich_redis
    image: docker.io/redis:6.2-alpine@sha256:2ba50e1ac3a0ea17b736ce9db2b0a9f6f8b85d4c27d5f5accc6a416d8f42c6d5
    healthcheck:
      test: redis-cli ping || exit 1
    restart: always
    networks:
      - immich

  database:
    container_name: immich_postgres
    image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
    volumes:
      # Do not edit the next line. If you want to change the database storage location on your system, edit the value of DB_DATA_LOCATION in the .env file
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    healthcheck:
      test: pg_isready --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1
      interval: 5m
      start_interval: 30s
      start_period: 5m
    command: ["postgres", "-c", "shared_preload_libraries=vectors.so", "-c", 'search_path="$$user", public, vectors', "-c", "logging_collector=on", "-c", "max_wal_size=2GB", "-c", "shared_buffers=512MB", "-c", "wal_compression=on"]
    restart: always
    networks:
      - immich

volumes:
  model-cache:
networks:
  proxy:
    external: true
  immich:

Your .env content

IMMICH_VERSION=release
DB_DATA_LOCATION=path/database
DB_DATABASE_NAME=DB_Name
DB_USERNAME=DB_User
DB_PASSWORD=DB_Pass
UPLOAD_LOCATION=/path/upload
LOG_LEVEL=warn

Reproduction steps

1. open specific album
2. wait about one minute
3. 502 error on the screen
   ...

Relevant log output

Output log with firefox:

2024-11-03T19:28:17.730494673Z <--- Last few GCs --->

2024-11-03T19:28:17.730500384Z 

2024-11-03T19:28:17.730504952Z [17:0x5c7b6320000]    93571 ms: Mark-Compact 4051.8 (4138.3) -> 4038.8 (4141.3) MB, 852.07 / 0.00 ms  (average mu = 0.104, current mu = 0.024) allocation failure; scavenge might not succeed

2024-11-03T19:28:17.730509801Z [17:0x5c7b6320000]    95170 ms: Mark-Compact 4054.7 (4141.3) -> 4041.8 (4144.3) MB, 1575.17 / 0.00 ms  (average mu = 0.048, current mu = 0.015) allocation failure; scavenge might not succeed

2024-11-03T19:28:17.730514510Z 

2024-11-03T19:28:17.730518738Z 

2024-11-03T19:28:17.730522856Z <--- JS stacktrace --->

2024-11-03T19:28:17.730527374Z 

2024-11-03T19:28:17.730531732Z FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

2024-11-03T19:28:17.730552380Z ----- Native stack trace -----

2024-11-03T19:28:17.730556869Z 

2024-11-03T19:28:17.730870732Z  1: 0xb8ced1 node::OOMErrorHandler(char const*, v8::OOMDetails const&) [immich-api]

2024-11-03T19:28:17.731269684Z  2: 0xf06460 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich-api]

2024-11-03T19:28:17.731643539Z  3: 0xf06747 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich-api]

2024-11-03T19:28:17.732018737Z  4: 0x11182e5  [immich-api]

2024-11-03T19:28:17.732378386Z  5: 0x1130168 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich-api]

2024-11-03T19:28:17.732742723Z  6: 0x1106281 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich-api]

2024-11-03T19:28:17.733120496Z  7: 0x1107415 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich-api]

2024-11-03T19:28:17.733486176Z  8: 0x10e4a66 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [immich-api]

2024-11-03T19:28:17.733913630Z  9: 0x1540896 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [immich-api]

2024-11-03T19:28:17.733918439Z 10: 0x7efc859d9ef6 

2024-11-03T19:28:18.359773238Z api worker exited with code null


Output Log with Edge:

2024-11-03T19:47:34.312827054Z <--- Last few GCs --->

2024-11-03T19:47:34.312831823Z 

2024-11-03T19:47:34.312835399Z [17:0x3dd9c320000]  1150129 ms: Mark-Compact 4049.8 (4136.3) -> 4036.9 (4139.3) MB, 860.70 / 0.00 ms  (average mu = 0.106, current mu = 0.026) allocation failure; scavenge might not succeed

2024-11-03T19:47:34.312839637Z [17:0x3dd9c320000]  1150998 ms: Mark-Compact 4052.7 (4139.3) -> 4040.0 (4142.6) MB, 847.20 / 0.00 ms  (average mu = 0.068, current mu = 0.026) allocation failure; scavenge might not succeed

2024-11-03T19:47:34.312843394Z 

2024-11-03T19:47:34.312846630Z 

2024-11-03T19:47:34.312849866Z <--- JS stacktrace --->

2024-11-03T19:47:34.312853182Z 

2024-11-03T19:47:34.312856338Z FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

2024-11-03T19:47:34.312859705Z ----- Native stack trace -----

2024-11-03T19:47:34.312862941Z 

2024-11-03T19:47:34.313170082Z  1: 0xb8ced1 node::OOMErrorHandler(char const*, v8::OOMDetails const&) [immich-api]

2024-11-03T19:47:34.313534139Z  2: 0xf06460 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich-api]

2024-11-03T19:47:34.313868842Z  3: 0xf06747 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich-api]

2024-11-03T19:47:34.314234101Z  4: 0x11182e5  [immich-api]

2024-11-03T19:47:34.314605112Z  5: 0x1118874 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [immich-api]

2024-11-03T19:47:34.315017408Z  6: 0x112f764 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [immich-api]

2024-11-03T19:47:34.315491270Z  7: 0x112ff7c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich-api]

2024-11-03T19:47:34.315791278Z  8: 0x1106281 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich-api]

2024-11-03T19:47:34.316216569Z  9: 0x1107415 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich-api]

2024-11-03T19:47:34.316612475Z 10: 0x10e4a66 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [immich-api]

2024-11-03T19:47:34.316992833Z 11: 0x1540896 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [immich-api]

2024-11-03T19:47:34.316999455Z 12: 0x7ff93e099ef6 

2024-11-03T19:47:34.950860300Z api worker exited with code null

Additional information

No response

[mmomjian]

What is the system specs of your server and what resources have been allocated to the Immich containers?

[Thloluvin]

Board:

CPU:

My Docker stats:

=> 128GB Physical Memory

No limits set for the containers
No Graphic Card used (CPU only)

[Thloluvin]

Updated to latest Version and tried again today.
Now this problem seems to be gone.
All Pictures are loaded immediately again when open an album.