Splunk Operator: undefined update triggered splunk-operator to recreate cluster-manager pod

[yaroslav-nakonechnikov]

Please select the type of request

Bug

Tell us more

Describe the request
Undefined update triggered splunk-operator to recreate cluster-manager pod.
Today we faced uncontrolled restart of whole splunk cluster, and after reading the logs we saw that:

{ [-]
   ClusterManager: { [+]
   }
   controller: clustermanager
   controllerGroup: enterprise.splunk.com
   controllerKind: ClusterManager
   kind: ClusterManager
   level: INFO
   msg: isClusterManagerReadyForUpgrade  kind is set to
   name: prod
   namespace: splunk-operator
   reconcileID: ab0300f0-2ac5-4a3c-8d05-a0020fc00f4b
   time: 2025-02-22T22:51:27.879374921Z
}

{ [-]
   ClusterManager: { [+]
   }
   controller: clustermanager
   controllerGroup: enterprise.splunk.com
   controllerKind: ClusterManager
   level: INFO
   msg: UpdateStatefulSetPods  Recycling Pod for updates
   name: splunk-prod-cluster-manager
   namespace: splunk-operator
   podName: splunk-prod-cluster-manager-0
   podRevision: splunk-prod-cluster-manager-59847545
   reconcileID: ab0300f0-2ac5-4a3c-8d05-a0020fc00f4b
   statefulSetRevision: splunk-prod-cluster-manager-5fd96cb98d
   time: 2025-02-22T22:51:27.879538871Z
}

{ [-]
   annotations: { [+]
   }
   apiVersion: audit.k8s.io/v1
   auditID: 06fbfd50-a04a-44e5-8f27-a7869a82b833
   kind: Event
   level: RequestResponse
   objectRef: { [-]
     apiVersion: v1
     name: splunk-prod-cluster-manager-0
     namespace: splunk-operator
     resource: pods
   }
   requestObject: { [+]
   }
   requestReceivedTimestamp: 2025-02-22T22:51:27.885329Z
   requestURI: /api/v1/namespaces/splunk-operator/pods/splunk-prod-cluster-manager-0
   responseObject: { [+]
   }
   responseStatus: { [+]
   }
   sourceIPs: [ [+]
   ]
   stage: ResponseComplete
   stageTimestamp: 2025-02-22T22:51:27.929949Z
   user: { [-]
     extra: { [+]
     }
     groups: [ [+]
     ]
     uid: 2348e17c-ef36-4204-8f38-c3f23bc2f26c
     username: system:serviceaccount:splunk-operator:splunk-operator-controller-manager
   }
   userAgent: manager/v0.0.0 (linux/amd64) kubernetes/$Format
   verb: delete
}

and after that CM was recreated.
but we didn't do any changes this night and we don't want that splunk-operator will kill pods by its own without clear message in the log.

Expected behavior
Pods shouldn't be restarted.

Splunk setup on K8S
AWS EKS 1.32
splunk-operator 7.0 with Splunk 9.2.4