Breaking update v2.16.0

[spanozzo]

Updating to version 2.16.0 breaks our deployment.

We currently run Elasticsearch and Kibana version 8.17.0, and seems like the new Kibana default security context is not compatible with xpack.security.audit.enabled: true.

This is what we see from Kibana logs:

[2024-12-27T13:38:55.453+00:00][WARN ][environment] Detected an uncaughtException: Error: EROFS: read-only file system, open '/usr/share/kibana/logs/audit.log'
[Error: EROFS: read-only file system, open '/usr/share/kibana/logs/audit.log'] {
errno: -30,
code: 'EROFS',
syscall: 'open',
path: '/usr/share/kibana/logs/audit.log'
}

After setting the xpack.security.audit.enabled: false the problem is gone and the update is possible, but that's definitely not what we want.

I haven't tested, but maybe set xpack.security.audit.appender.type: console would solve the problem, even if I don't know what could be the consequences.

Do you have any other idea on how to manage it?

[pebrc]

A possible workaround that we are currently considering to add through the operator directly in #8380 by default is an additional emptyDir volume in the logs mount path. For example like so:

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana
spec:
  config:
    xpack.security.audit.enabled: true
  version: 8.16.0
  count: 1
  elasticsearchRef:
    name: elasticsearch
  podTemplate:
    spec:
      containers:
      - name: kibana
        volumeMounts:
          - name: kibana-logs
            mountPath: /usr/share/kibana/logs
      volumes:
        - name: kibana-logs
          emptyDir: {}

There is a trade-off with this approach in that emptyDir volumes are not suitable for large amounts of data as you are effectively writing into the kubelets filesystem. Depending on the logging volume your are seeing through these audit logs you might want to chose a different volume type or as you said use the console appender. In case you want to keep writing to files it would be recommened to ingest the logs e.g. via Elastic Agent and use for example the rolling file appender to rotate the logs and remove old ones.

[geusops]

Hello team,

Testing the suggested manifest I suspect the podTemplate might be missing.

Here's the snippet we've tested in lab:

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: eck-lab
spec:
  config:
    xpack.security.audit.enabled: true
  version: 8.17.0
  count: 1
  elasticsearchRef:
    name: "eck-lab"
  http:
    service:
      spec:
        type: LoadBalancer
  podTemplate:
    spec:
      containers:
      - name: kibana
        volumeMounts:
        - name: kibana-logs
          mountPath: /usr/share/kibana/logs
      volumes:
      - name: kibana-logs
        emptyDir: {}
    metadata:
      labels:
        scrape: kb

[pebrc]

Good catch. My mistake. Will update the example.