Continuous 502 with Kibana via nginx ingress controller

[hustluh]

I am having an issue using the latest images of ECK's Elastic + Kibana via Helm Chart, installed with following the quick start guides.

My Kibana pod is in a running state, as is my Elastic pod. The services are all running as well.

There are no errors in the logs to share.

This is what I am seeing:

kubectl get all -n elastic-stack
NAME                                                  READY   STATUS    RESTARTS   AGE
pod/elasticsearch-es-default-0                        1/1     Running   0          6d2h
pod/es-kb-quickstart-eck-kibana-kb-7c48f78499-fmsvz   1/1     Running   0          6d2h

NAME                                          TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
service/elasticsearch-es-default              ClusterIP   None           <none>        9200/TCP   6d2h
service/elasticsearch-es-http                 ClusterIP   10.43.55.145   <none>        9200/TCP   6d2h
service/elasticsearch-es-internal-http        ClusterIP   10.43.99.193   <none>        9200/TCP   6d2h
service/elasticsearch-es-transport            ClusterIP   None           <none>        9300/TCP   6d2h
service/es-kb-quickstart-eck-kibana-kb-http   ClusterIP   10.43.21.65    <none>        5601/TCP   6d2h

NAME                                             READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/es-kb-quickstart-eck-kibana-kb   1/1     1            1           6d2h

NAME                                                        DESIRED   CURRENT   READY   AGE
replicaset.apps/es-kb-quickstart-eck-kibana-kb-7c48f78499   1         1         1       6d2h

NAME                                        READY   AGE
statefulset.apps/elasticsearch-es-default   1/1     6d2h

kubectl get all -n elastic-system
NAME                     READY   STATUS    RESTARTS         AGE
pod/elastic-operator-0   1/1     Running   23 (2d12h ago)   13d

NAME                               TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
service/elastic-operator-webhook   ClusterIP   10.43.75.75   <none>        443/TCP   13d

NAME                                READY   AGE
statefulset.apps/elastic-operator   1/1     13d

kubectl get ing -n elastic-stack -o yaml
apiVersion: v1
items:
- apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    annotations:
      field.cattle.io/publicEndpoints: '[{"addresses":["10.1.0.66","10.1.0.71","10.1.0.95","10.1.2.216"],"port":443,"protocol":"HTTPS","serviceName":"elastic-stack:es-kb-quickstart-eck-kibana-kb-http","ingressName":"elastic-stack:kibana-ingress","hostname":"redacted.example.com","path":"/kibana/","allNodes":false}]'
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"networking.k8s.io/v1","kind":"Ingress","metadata":{"annotations":{"nginx.ingress.kubernetes.io/add-base-url":"true","nginx.ingress.kubernetes.io/ssl-redirect":"false"},"name":"kibana-ingress","namespace":"elastic-stack"},"spec":{"ingressClassName":"nginx","rules":[{"host":"redacted.example.com","http":{"paths":[{"backend":{"service":{"name":"es-kb-quickstart-eck-kibana-kb-http","port":{"number":5601}}},"path":"/kibana/","pathType":"Prefix"}]}}]}}
      nginx.ingress.kubernetes.io/add-base-url: "true"
      nginx.ingress.kubernetes.io/ssl-redirect: "false"
    creationTimestamp: "2024-12-10T17:14:02Z"
    generation: 2
    name: kibana-ingress
    namespace: elastic-stack
    resourceVersion: "210737293"
    uid: 6db13847-c89c-410b-ae3f-199ba2aa6734
  spec:
    ingressClassName: nginx
    rules:
    - host: redacted.example.com
      http:
        paths:
        - backend:
            service:
              name: es-kb-quickstart-eck-kibana-kb-http

Additionally, here are logs from the nginx-ingress controller when trying to access the URL:

2025/01/13 18:55:45 [error] 1637#1637: *576687 upstream prematurely closed connection while reading response header from upstream, client: 10.10.0.104, server: redacted.example.com, request: "GET /kibana/ HTTP/2.0", upstream: "http://10.42.3.30:5601/kibana/", host: "redacted.example.com"
2025-01-13T11:55:45.451896265-07:00 2025/01/13 18:55:45 [error] 1637#1637: *576687 upstream prematurely closed connection while reading response header from upstream, client: 10.10.0.104, server: redacted.example.com, request: "GET /kibana/ HTTP/2.0", upstream: "http://10.42.3.30:5601/kibana/", host: "redacted.example.com"
2025-01-13T11:55:45.454045501-07:00 2025/01/13 18:55:45 [error] 1637#1637: *576687 upstream prematurely closed connection while reading response header from upstream, client: 10.10.0.104, server: redacted.example.com, request: "GET /kibana/ HTTP/2.0", upstream: "http://10.42.3.30:5601/kibana/", host: "redacted.example.com"
2025-01-13T11:55:45.454268893-07:00 10.10.0.104 - - [13/Jan/2025:18:55:45 +0000] "GET /kibana/ HTTP/2.0" 502 552 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" 35 0.008 [elastic-stack-mr-eskb-eck-kibana-kb-http-5601] [] 10.42.3.30:5601, 10.42.3.30:5601, 10.42.3.30:5601 0, 0, 0 0.004, 0.002, 0.003 502, 502, 502 5a8074050332d776d773e1c5759981a6

[Harish27012002]

u need to port the kibana service to node port and u can acces it

[hustluh]

u need to port the kibana service to node port and u can acces it

While this would help me access it directly, this does not help solve the issue.