Liveness/Readiness Probe fails for Helm chart installation after 0.2.2

[michael-datareply]

We wanted to upgrade from helm chart version 0.2.1 and image version 0.17.0
Unfortunately for all versions thereafter both the Liveness and the Readiness probes consistently fail.
Here is the error for versions 0.2.7 and 0.20.0:

Created container akhq
Started container akhq
Readiness probe failed: Get "http://100.0.38.182:28081/health": dial tcp 100.0.38.182:28081: connect: connection refused
Liveness probe failed: dial tcp 100.0.38.182:28081: connect: connection refused
Readiness probe failed: Get "http://100.0.38.182:28081/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Readiness probe failed: HTTP probe failed with statuscode: 409

[tchiotludo]

please provide you yaml configuration please

[michael-datareply]

Since we use helmfile with multiple levels of values files I guess it's more helpful to see the final resources that get deployed based on all those values:

# Source: akhq/templates/secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: akhq-secrets
  labels:
    app.kubernetes.io/name: akhq
    helm.sh/chart: akhq-0.2.7
    app.kubernetes.io/instance: akhq
    app.kubernetes.io/managed-by: Helm
type: Opaque
data:
  application-secrets.yml: "fAogIGFraH..."
---
# Source: akhq/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: akhq
  labels:
    app.kubernetes.io/name: akhq
    helm.sh/chart: akhq-0.2.7
    app.kubernetes.io/instance: akhq
    app.kubernetes.io/managed-by: Helm
data:
  application.yml: |
    akhq:
      server:
        access-log:
          enabled: false
          name: org.akhq.log.access
---
# Source: akhq/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: akhq
  labels:
    app.kubernetes.io/name: akhq
    helm.sh/chart: akhq-0.2.7
    app.kubernetes.io/instance: akhq
    app.kubernetes.io/managed-by: Helm
  annotations:
spec:
  type: ClusterIP
  ports:
    - port: 80
      targetPort: http
      protocol: TCP
      name: http
  selector:
    app.kubernetes.io/name: akhq
    app.kubernetes.io/instance: akhq
---
# Source: akhq/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: akhq
  labels:
    app.kubernetes.io/name: akhq
    helm.sh/chart: akhq-0.2.7
    app.kubernetes.io/instance: akhq
    app.kubernetes.io/managed-by: Helm
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: akhq
      app.kubernetes.io/instance: akhq
  template:
    metadata:
      annotations:
        checksum/config: df50cc39d01281d3769c9867818b2216bc075ed31cad8748a87e4dddd38f0318
        checksum/secrets: cd8eb17fe34977a75348b705cc147f729c6ce33cbd18d6f0947fbf96a9f8ad6e
      labels:
        app.kubernetes.io/name: akhq
        app.kubernetes.io/instance: akhq
    spec:
      imagePullSecrets:
                - name: artifactory-secret-v2
      serviceAccountName: default
      initContainers:
        - name: create-keystore
          command:
          - /bin/sh
          - -c
          - |
            keytool -import -keystore /tmp/truststore.jks -alias CARoot -file /app/ca.crt -noprompt -storepass dataaccess
          image: openjdk:11-slim
          volumeMounts:
          - mountPath: /app/ca.crt
            name: cluster-ca
            subPath: ca.crt
          - mountPath: /tmp
            name: share
      containers:
        - name: akhq
          image: "tchiotludo/akhq:0.20.0"
          imagePullPolicy: Always
          env:
            - name: JAAS
              valueFrom:
                secretKeyRef:
                  key: sasl.jaas.config
                  name: strimzi-user-akhq
            - name: MICRONAUT_ENVIRONMENTS
              value: secrets
            - name: MICRONAUT_CONFIG_FILES
              value: /app/application.yml,/app/application-secrets.yml
          volumeMounts:
            - mountPath: /share
              name: share
            - name: config
              mountPath: /app/application.yml
              subPath: application.yml
            - name: secrets
              mountPath: /app/application-secrets.yml
              subPath: application-secrets.yml
          ports:
            - name: http
              containerPort: 8080
              protocol: TCP
            - name: management
              containerPort: 28081
              protocol: TCP
          livenessProbe:
            tcpSocket:
              port: management
          readinessProbe:
            httpGet:
              path: /health
              port: management
          resources:
            {}
      volumes:
      - name: cluster-ca
        secret:
          secretName: strimzi-cluster-ca-cert
      - emptyDir: {}
        name: share
      - name: config
        configMap:
          name: akhq
      - name: secrets
        secret:
          secretName: akhq-secrets
          ```

[tchiotludo]

REally strange, please go inside the pods and try the management port with a simple curl.

[michael-datareply]

Hi, so I finally figured it out. The health check was only a symptom. Turns out I missed that the way the application-secrets.yml file is created was changed slightly. By just using the old configuration I had a"|" on line 1 of the file and thus the application didn't start. Curious that this didn't show up in the logs though (it did show up when CURLing the /health endpoint from within the pod).
Feel free to close this, thank you for your assistance!

[Karabiy]

As running into same issue, just want to mention that would be useful to get additional debugging information on which issue root cause it can be, as it does feel like app start itself with the

O  main       i.m.runtime.Micronaut      Startup completed in 2255ms. Server Running: http://akhq-5fddcbf5sd-xjlv2:8080

Though, it doesn't, and no more log appear and it becomes really hard to dig further

[tchiotludo]

You can add a lot of logs adding this to the configuration files:

logger:
  levels:
    org.akhq: TRACE
    io.micronaut: TRACE

What is the version of helm charts & version of akhq ?

[usernameisnull]

any progress, I still encounter this problem

[usernameisnull]

Temporarily changing the timeoutSeconds of livenessProbe to 20 can solve the issue