Permission denied on internal db

[Slyke]

Helm version:

version.BuildInfo{Version:"v3.15.3", GitCommit:"3bb50bbbdd9c946ba9989fbe4fb4104766302a64", GitTreeState:"clean", GoVersion:"go1.22.5"}

Harbor version:

v2.11.0

Logs:

init DB, DB version:15
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with this locale configuration:
  provider:    libc
  LC_COLLATE:  en_US.UTF-8
  LC_CTYPE:    en_US.UTF-8
  LC_MESSAGES: C
initdb: error: could not access directory "/var/lib/postgresql/data/pgdata/pg15": Permission denied
  LC_MONETARY: C
  LC_NUMERIC:  C
  LC_TIME:     C
The default text search configuration will be set to "english".

Data page checksums are disabled.

Pod description:

Name:         harbor-test-database-0
Namespace:    XXXXXXXXXXXXXXXX
Priority:     0
Node:         XXXXXXXXXXXXXXXXXXXXXX
Start Time:   Mon, 12 Aug 2024 20:45:07 +0000
Labels:       app=harbor
              app.kubernetes.io/component=database
              app.kubernetes.io/instance=harbor-test
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=harbor
              app.kubernetes.io/part-of=harbor
              app.kubernetes.io/version=2.11.0
              chart=harbor
              component=database
              controller-revision-hash=harbor-test-database-5d9b7cf448
              heritage=Helm
              release=harbor-test
              statefulset.kubernetes.io/pod-name=harbor-test-database-0
Annotations:  checksum/secret: a2b0779dc9dfd5560e4dd773aa858315c10c217e309c425bd6afa29964c21ff6
              container.seccomp.security.alpha.kubernetes.io/data-permissions-ensurer: runtime/default
              container.seccomp.security.alpha.kubernetes.io/database: runtime/default
Status:       Running
IP:           10.244.0.248
IPs:
  IP:           10.244.0.248
Controlled By:  StatefulSet/harbor-test-database
Init Containers:
  data-permissions-ensurer:
    Container ID:  containerd://8c0e5fd88285b8601c3c2f586201da2567bb22d86ac8a41f2277f99b5a8b12a1
    Image:         goharbor/harbor-db:v2.11.0
    Image ID:      docker.io/goharbor/harbor-db@sha256:103820016e01b0f6a2c1ee88692d3774ff4dd345c85b58e7c928e3a2b958bda3
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
    Args:
      -c
      chmod -R 700 /var/lib/postgresql/data/pgdata || true
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Mon, 12 Aug 2024 20:45:08 +0000
      Finished:     Mon, 12 Aug 2024 20:45:08 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/lib/postgresql/data from database-data (rw)
Containers:
  database:
    Container ID:   containerd://893adb6931833f1d36f9c20b5be1fcb0346df85baafee0c2667f6dd3e37d4a5d
    Image:          goharbor/harbor-db:v2.11.0
    Image ID:       docker.io/goharbor/harbor-db@sha256:103820016e01b0f6a2c1ee88692d3774ff4dd345c85b58e7c928e3a2b958bda3
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Mon, 12 Aug 2024 20:46:01 +0000
      Finished:     Mon, 12 Aug 2024 20:46:01 +0000
    Ready:          False
    Restart Count:  3
    Liveness:       exec [/docker-healthcheck.sh] delay=300s timeout=1s period=10s #success=1 #failure=3
    Readiness:      exec [/docker-healthcheck.sh] delay=1s timeout=1s period=10s #success=1 #failure=3
    Environment Variables from:
      harbor-test-database  Secret  Optional: false
    Environment:
      PGDATA:  /var/lib/postgresql/data/pgdata
    Mounts:
      /dev/shm from shm-volume (rw)
      /var/lib/postgresql/data from database-data (rw)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  shm-volume:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  512Mi
  database-data:
    Type:        PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:   harbor-db-pvc
    ReadOnly:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                 From               Message
  ----     ------     ----                ----               -------
  Normal   Scheduled  76s                 default-scheduler  Successfully assigned cicd/harbor-test-database-0 to k-w-003
  Normal   Pulled     75s                 kubelet            Container image "goharbor/harbor-db:v2.11.0" already present on machine
  Normal   Created    75s                 kubelet            Created container data-permissions-ensurer
  Normal   Started    75s                 kubelet            Started container data-permissions-ensurer
  Normal   Pulled     23s (x4 over 75s)   kubelet            Container image "goharbor/harbor-db:v2.11.0" already present on machine
  Normal   Created    23s (x4 over 75s)   kubelet            Created container database
  Normal   Started    22s (x4 over 75s)   kubelet            Started container database
  Warning  BackOff    16s (x10 over 73s)  kubelet            Back-off restarting failed container

[crlsgms]

I'm also getting an error updating from 1.14.3 to 1.15.0 (or straigh to 1.15.1) database, core, jobservice, redis and registry in crashloopbackoff

database

database upgrade DB from 14 to 15
database popen failure: Cannot allocate memory
database initdb: error: program "postgres" is needed by initdb but was not found in the same directory as "/usr/pgsql/15/bin/initdb"

core

goroutine 1 gp=0xc0000061c0 m=0 mp=0x4418680 [running]:
runtime.systemstack_switch()
    /usr/local/go/src/runtime/asm_amd64.s:474 +0x8 fp=0xc00013e750 sp=0xc00013e740 pc=0x474a08
runtime.main()
    /usr/local/go/src/runtime/proc.go:171 +0x67 fp=0xc00013e7e0 sp=0xc00013e750 pc=0x442467
runtime.goexit({})
    /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00013e7e8 sp=0xc00013e7e0 pc=0x476a21

rax    0x0
rbx    0x1
rcx    0x7f9186eb2dbc
rdx    0x6
rdi    0x1
rsi    0x1
rbp    0x7f9186e25740
rsp    0x7fffe32d0950
r8     0x0
r9     0x73
r10    0x8
r11    0x246
r12    0x6
r13    0x0
r14    0x4417840
r15    0x1ffffffffffffff
rip    0x7f9186eb2dbc
rflags 0x246
cs     0x33
fs     0x0
gs     0x0

jobserver also like the core pod, runtime errors on same asm_amd64

redis

1:C 30 Aug 2024 03:52:13.902 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 30 Aug 2024 03:52:13.902 * Redis version=7.2.4, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 30 Aug 2024 03:52:13.902 * Configuration loaded
1:M 30 Aug 2024 03:52:13.903 * monotonic clock: POSIX clock_gettime
1:M 30 Aug 2024 03:52:13.903 # Failed to write PID file: Permission denied
                _._
           _.-``__ ''-._
      _.-``    `.  `_.  ''-._           Redis 7.2.4 (00000000/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._
 (    '      ,       .-`  | `,    )     Running in standalone mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
 |    `-._   `._    /     _.-'    |     PID: 1
  `-._    `-._  `-./  _.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    |           https://redis.io
  `-._    `-._`-.__.-'_.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    |
  `-._    `-._`-.__.-'_.-'    _.-'
      `-._    `-.__.-'    _.-'
          `-._        _.-'
              `-.__.-'

1:M 30 Aug 2024 03:52:13.903 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
1:M 30 Aug 2024 03:52:13.903 # Fatal: Can't initialize Background Jobs. Error message: Operation not permitted

registry even start for a while, then goes on the same error as core and jobservice

registry 10.20.91.16 - - [30/Aug/2024:03:55:52 +0000] "GET / HTTP/1.1" 200 0 "" "kube-probe/1.26"
registry 10.20.91.16 - - [30/Aug/2024:03:56:02 +0000] "GET / HTTP/1.1" 200 0 "" "kube-probe/1.26"
registry 10.20.91.16 - - [30/Aug/2024:03:56:02 +0000] "GET / HTTP/1.1" 200 0 "" "kube-probe/1.26"
registry 10.20.91.16 - - [30/Aug/2024:03:56:12 +0000] "GET / HTTP/1.1" 200 0 "" "kube-probe/1.26"
registryctl goroutine 1 gp=0xc0000061c0 m=0 mp=0x1f3ebc0 [running]:
registryctl runtime.systemstack_switch()
registryctl     /usr/local/go/src/runtime/asm_amd64.s:474 +0x8 fp=0xc00011a750 sp=0xc00011a740 pc=0x472888
registryctl runtime.main()
registryctl     /usr/local/go/src/runtime/proc.go:171 +0x67 fp=0xc00011a7e0 sp=0xc00011a750 pc=0x441427
registryctl runtime.goexit({})
registryctl     /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00011a7e8 sp=0xc00011a7e0 pc=0x4748a1
registryctl
registryctl rax    0x0
registryctl rbx    0x1
registryctl rcx    0x7ff5d1eb6dbc
registryctl rdx    0x6
registryctl rdi    0x1
registryctl rsi    0x1
registryctl rbp    0x7ff5d1e29740
registryctl rsp    0x7ffc7da59ac0
registryctl r8     0x0
registryctl r9     0x73
registryctl r10    0x8
registryctl r11    0x246
registryctl r12    0x6
registryctl r13    0x0
registryctl r14    0x1f3e180
registryctl r15    0x1ffffffffffffff
registryctl rip    0x7ff5d1eb6dbc
registryctl rflags 0x246
registryctl cs     0x33
registryctl fs     0x0
registryctl gs     0x0

anyhow its now impossible to upgrade from version 1.14.3 to whatever new 1.15 version (I'm using rancher btw)

[MinerYang]

How do you upgrade your helm harbor? I cannot reproduce this issue at my side. May need more contexts here. e.g. any configurations apply to your database settings?