Clarify what the wildfly-operator does

[rachmatowicz]

In looking at the documentation for this operator, I see a lot of good description about how to build the operator and how to configure it and use it, but little information on what it actually does for a deployed application. From my point of view, in terms of what it actually does under the covers, its like a black box and I just have to hope its doing what it should be doing.

There are significant differences between deploying a stateless application on wildfly and a stateful application on wildfly with respect to guaranteeing consistency in the face of deployments, scaling up and scaling down and rolling updates, which will all come for free if what this operator does is based on Deployments or StatefulSets. Also, the way in which Wildfly applications are exposed can have significant implications for the correctness of application behaviour.

My understanding of an operator is that it allows you to extend Kubernetes by defining a desired state and a controller which will enforce that desired state. I suppose what I am asking here is, when you use the Wildfly operator, what is that defined state that is guaranteed?

So what I would like to know is:

• what workload resources does it deploy to control a Wildfly-based application and how are those resources related
• what guarantees does it enforce between those workload resources over and above the default Kubernetes workload resources semantics

For example, through reading docs, I know it uses StatefulSets in some way, presumably to provide persistent identity and persistent storage access as Pods are created and re-incarnated. I also know that in scaledown, there are some guarantees regarding clean termination of in-flight transactions.

Every one of these native and non-native Kubernetes behaviours should be identified and described so that when I use the Wildfly operator, I know exactly what I am getting and whether the guarantees it provides are sufficient for my application requirements.

[yersan]

In looking at the documentation for this operator, I see a lot of good description about how to build the operator and how to configure it and use it, but little information on what it actually does for a deployed application. From my point of view, in terms of what it actually does under the covers, its like a black box and I just have to hope its doing what it should be doing.

What the Operator does and its capabilities are described in the user guide documentation: https://github.com/wildfly/wildfly-operator/blob/main/doc/user-guide.adoc. This documentation is a high-level documentation to understand what the Operator can do for you, it doesn't describe in detail how it does what it does and I think it has never been its main goal. You can use the Operator and inspect the resources created, which could help you understand how it works.

There are significant differences between deploying a stateless application on wildfly and a stateful application on wildfly with respect to guaranteeing consistency in the face of deployments, scaling up and scaling down and rolling updates, which will all come for free if what this operator does is based on Deployments or StatefulSets. Also, the way in which Wildfly applications are exposed can have significant implications for the correctness of application behavior.

This Operator is based solely on StatatefulSet and its use case is applicable to stateful applications that require a clean termination of in-flight transactions when the pods are scaled down. That's it. If your application doesn't require such capabilities, you could use Helm Charts or plain StatefulSets instead of an Operator.

My understanding of an operator is that it allows you to extend Kubernetes by defining a desired state and a controller which will enforce that desired state. I suppose what I am asking here is, when you use the Wildfly operator, what is that defined state that is guaranteed?

Maybe one way to see what is this state is to check the spec section of your Custom Resource. You specify there the configuration you want and the Operator will keep your instances satisfying the state you have defined there. E.g, three replicas with X environment variables, with Y persistent storage ...

In addition to such a state, the Operator does what the human will do when you need to scale down an application and you want to ensure that any inflight transactions have a clean termination.

So what I would like to know is:
what workload resources does it deploy to control a Wildfly-based application and how are those resources related

It deploys a service to expose your application, a headless service to have a stable point for inbound EJB calls, a service to expose the management port, a route to expose your application, a ServiceMonitor for metrics, and a stateful set to control the replicas of your application. Those are standard Openshift resources, once you have deployed an application, you could inspect them to see how they are related together.

what guarantees does it enforce between those workload resources over and above the default Kubernetes workload resources semantics
For example, through reading docs, I know it uses StatefulSets in some way, presumably to provide persistent identity and persistent storage access as Pods are created and re-incarnated. I also know that in scaledown, there are some guarantees regarding clean termination of in-flight transactions.

The Operator extends the OpensShift API with a new resource that you can use to deploy your applications ensuring they will have a clean termination of in-flight transactions when they are scaled down. It creates for your a set of standard resources based on your Custom Resource Spec configuration, and the created resources are the standard OpenShift resources (Services, StatefulSet ...).

Every one of these native and non-native Kubernetes behaviours should be identified and described so that when I use the Wildfly operator, I know exactly what I am getting and whether the guarantees it provides are sufficient for my application requirements.

Well, the user guide could be improved to describe exactly the resources it deploys but that adds a maintenance cost to keep them up to date, in addition, there could be resources that are deployed upon your CR configuration which adds more difficulties for describing them. So far we have just a high-level description of what it does.

[rachmatowicz]

Hi Yeray, unfortunately, the termination of in-flight transactions is not the only issue that arises when deploying a stateful Wildfly application to OpenShift. There are a number of them, such as broken rolling updates due to marshalling incompatabilities between different Wildfly/Infinispan versions (Paul is working on this) and if an application is not exposed correctly to the world outside the cluster, stickiness of sessions to stateful beans will also break. Those are two, there may be more.
If you look at the Kubernetes documentation, they describe in detail what the guarantees are for each abstraction (e.g replica sets). This request is just asking for the same level of detail for the Operator.
But i'll leave it to your judgement to decide what is best.

[rachmatowicz]

Just for your information, have a look at what other operators do with respect to describing their behavior:
https://github.com/lightbend/akka-cluster-operator/blob/master/README.md

[mchoma]

@rachmatowicz thank you for raising this. This is interesting discussion indeed.

such as broken rolling updates due to marshalling incompatabilities between different Wildfly/Infinispan versions (Paul is working on this)

Do you mean Use Case when user wants to upgrade Wildfly with zero downtime? And problem is in temporary state when cluster is formed of different server versions?

and if an application is not exposed correctly to the world outside the cluster, stickiness of sessions to stateful beans will also break

Can you elaborate more what mean correctly or not correctly exposure? Does Service + Route expose stateful bean correctly?

[rachmatowicz]

Hi @mchoma
I think these issues are being clarified in the currently active discussion with Paul, Yeray and yourself, but in answer to the first question,yes, and in answer to the second question, any load balancer/ingress controller involved in routing requests from external clients to the deployment must support Cookie-based routing of session-oriented requests, and as Paul noted, not all do.