Scaling down is failing with WildFly 18 S2I

[jmesnil]

Steps to reproduce

• use the branch https://github.com/jmesnil/wildfly-operator/tree/105_wildfly_18_S2I that uses S2I for WildFly 18.
• run a local operator: make run-local-operator
• deploy an app built from WildFly 18 S2I with 2 replicas:

apiVersion: wildfly.org/v1alpha1
kind: WildFlyServer
metadata:
  name: quickstart
spec:
  applicationImage: "quay.io/wildfly-quickstarts/wildfly-operator-quickstart:18.0"
  replicas: 2

• wait for this resource to be deployed and set its replicas to 1

The Operator will start recovery but error appears and the pod quickstart-1 is not terminated:

{"level":"info","ts":1570609202.4663382,"logger":"wildflyserver_controller","msg":"Reconciling WildFlyServer","Request.Namespace":"default","Request.Name":"quickstart"}
{"level":"info","ts":1570609202.466368,"logger":"wildlfyserver_resources","msg":"Getting resource","WildFlyServer.Namespace":"default","WildFlyServer.Name":"quickstart","Resource.Name":"quickstart"}
{"level":"info","ts":1570609202.466398,"logger":"wildlfyserver_resources","msg":"Got resource","WildFlyServer.Namespace":"default","WildFlyServer.Name":"quickstart","Resource.Name":"quickstart"}
{"level":"info","ts":1570609202.466454,"logger":"wildlfyserver_resources","msg":"Updating Resource","WildFlyServer.Namespace":"default","WildFlyServer.Name":"quickstart","Resource.Name":"quickstart-1","Resource.Type":"*v1.Pod"}
{"level":"info","ts":1570609202.484062,"logger":"wildlfyserver_resources","msg":"Updated resource","WildFlyServer.Namespace":"default","WildFlyServer.Name":"quickstart","Resource.Name":"quickstart-1","Resource.Type":"*v1.Pod"}
{"level":"info","ts":1570609202.48409,"logger":"wildflyserver_controller","msg":"Label for pod succesfully updated","Request.Namespace":"default","Request.Name":"quickstart","Pod Name":"quickstart-1","Label name":"wildfly.org/operated-by-loadbalancer","Label value":"disabled"}
{"level":"info","ts":1570609202.489409,"logger":"wildflyserver_controller","msg":"Reconciling WildFlyServer","Request.Namespace":"default","Request.Name":"quickstart"}
{"level":"info","ts":1570609202.489507,"logger":"wildlfyserver_resources","msg":"Getting resource","WildFlyServer.Namespace":"default","WildFlyServer.Name":"quickstart","Resource.Name":"quickstart"}
{"level":"info","ts":1570609202.4895518,"logger":"wildlfyserver_resources","msg":"Got resource","WildFlyServer.Namespace":"default","WildFlyServer.Name":"quickstart","Resource.Name":"quickstart"}
{"level":"info","ts":1570609202.489682,"logger":"wildlfyserver_resources","msg":"Updating status of WildFlyServer","WildFlyServer.Namespace":"default","WildFlyServer.Name":"quickstart"}
{"level":"info","ts":1570609202.49952,"logger":"wildlfyserver_resources","msg":"Updated status of WildFlyServer","WildFlyServer.Namespace":"default","WildFlyServer.Name":"quickstart"}
{"level":"info","ts":1570609202.499551,"logger":"wildflyserver_controller","msg":"Scaling down statefulset by verification if pods are clean by recovery","StatefulSet.Namespace":"default","StatefulSet.Name":"quickstart"}
{"level":"info","ts":1570609202.499586,"logger":"wildflyserver_controller","msg":"Statefulset was not scaled to the desired replica size 1 (current StatefulSet size: 2). Transaction recovery scaledown process has not cleaned all pods. Please, check status of the WildflyServer quickstart","StatefulSet.Namespace":"default","StatefulSet.Name":"quickstart"}
{"level":"info","ts":1570609202.499646,"logger":"wildlfyserver_resources","msg":"Resource generations do not match","WildFlyServer.Name":"quickstart","WildFlyServer.Namespace":"default","Resource.Name":"quickstart","Resource.Type":"*v1.StatefulSet","WildFlyServer.Generation":2,"Resource.Generation":"1"}
{"level":"info","ts":1570609202.4996989,"logger":"wildlfyserver_resources","msg":"Updating Resource","WildFlyServer.Namespace":"default","WildFlyServer.Name":"quickstart","Resource.Name":"quickstart","Resource.Type":"*v1.StatefulSet"}
{"level":"info","ts":1570609202.572308,"logger":"wildlfyserver_resources","msg":"Updated resource","WildFlyServer.Namespace":"default","WildFlyServer.Name":"quickstart","Resource.Name":"quickstart","Resource.Type":"*v1.StatefulSet"}
{"level":"info","ts":1570609202.572331,"logger":"wildflyserver_controller","msg":"Updating StatefulSet to be up to date with the WildFlyServer Spec","StatefulSet.Namespace":"default","StatefulSet.Name":"quickstart"}
{"level":"info","ts":1570609202.572361,"logger":"wildflyserver_controller","msg":"Reconciling WildFlyServer","Request.Namespace":"default","Request.Name":"quickstart"}
{"level":"info","ts":1570609202.572383,"logger":"wildlfyserver_resources","msg":"Getting resource","WildFlyServer.Namespace":"default","WildFlyServer.Name":"quickstart","Resource.Name":"quickstart"}
{"level":"info","ts":1570609202.5723991,"logger":"wildlfyserver_resources","msg":"Got resource","WildFlyServer.Namespace":"default","WildFlyServer.Name":"quickstart","Resource.Name":"quickstart"}
{"level":"info","ts":1570609202.5724769,"logger":"wildflyserver_controller","msg":"Transaction recovery scaledown processing","Request.Namespace":"default","Request.Name":"quickstart","Pod Name":"quickstart-1","IP Address":"172.17.0.6","Pod State":"SCALING_DOWN_RECOVERY_INVESTIGATION","Pod Phase":"Running"}
{"level":"info","ts":1570609202.573067,"logger":"wildflyserver_controller","msg":"Setting up back-off period and orphan detection properties for scaledown transaction reocovery","Request.Namespace":"default","Request.Name":"quickstart","Pod Name":"quickstart-1"}
{"level":"info","ts":1570609216.269278,"logger":"wildflyserver_controller","msg":"Marking pod as being setup for transaction recovery. Adding annotation recovery-properties-setup","Request.Namespace":"default","Request.Name":"quickstart","Pod Name":"quickstart-1"}
{"level":"info","ts":1570609216.26931,"logger":"wildlfyserver_resources","msg":"Updating Resource","WildFlyServer.Namespace":"default","WildFlyServer.Name":"quickstart","Resource.Name":"quickstart-1","Resource.Type":"*v1.Pod"}
{"level":"info","ts":1570609216.347253,"logger":"wildlfyserver_resources","msg":"Updated resource","WildFlyServer.Namespace":"default","WildFlyServer.Name":"quickstart","Resource.Name":"quickstart-1","Resource.Type":"*v1.Pod"}
{"level":"info","ts":1570609216.347289,"logger":"wildflyserver_controller","msg":"Restarting application server to apply the env properies","Request.Namespace":"default","Request.Name":"quickstart","Pod Name":"quickstart-1"}
{"level":"info","ts":1570609234.566953,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 1/10","Request.Namespace":"default","Request.Name":"quickstart","Pod Name":"quickstart-1"}
{"level":"info","ts":1570609240.969735,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 2/10","Request.Namespace":"default","Request.Name":"quickstart","Pod Name":"quickstart-1"}
{"level":"info","ts":1570609257.2155662,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 3/10","Request.Namespace":"default","Request.Name":"quickstart","Pod Name":"quickstart-1"}
{"level":"info","ts":1570609268.479636,"logger":"wildflyserver_controller","msg":"Verification the recovery listener is setup to run transaction recovery at quickstart-1","Request.Namespace":"default","Request.Name":"quickstart"}
{"level":"info","ts":1570609278.723425,"logger":"wildflyserver_controller","msg":"Query to find the transaction recovery port to force scan at pod quickstart-1","Request.Namespace":"default","Request.Name":"quickstart"}

{"level":"info","ts":1570609298.224145,"logger":"wildlfyserver_resources","msg":"Updating Resource","WildFlyServer.Namespace":"default","WildFlyServer.Name":"quickstart","Resource.Name":"quickstart-1","Resource.Type":"*v1.Pod"}
{"level":"info","ts":1570609298.2792509,"logger":"wildlfyserver_resources","msg":"Updated resource","WildFlyServer.Namespace":"default","WildFlyServer.Name":"quickstart","Resource.Name":"quickstart-1","Resource.Type":"*v1.Pod"}
{"level":"info","ts":1570609298.279275,"logger":"wildflyserver_controller","msg":"Executing recovery scan at quickstart-1","Request.Namespace":"default","Request.Name":"quickstart","Pod IP":"172.17.0.6","Recovery port":4712}
{"level":"error","ts":1570609328.2970738,"logger":"wildflyserver_controller","msg":"Failures during scaling down recovery processing","Request.Namespace":"default","Request.Name":"quickstart","Desired replica size":1,"Number of pods to be removed":1,"error":"Found 1 errors:\n [[Failed to run transaction recovery scan for scaling down pod quickstart-1. Please, verify the pod log file. Error: Cannot process TCP connection to 172.17.0.6:4712, error: dial tcp 172.17.0.6:4712: i/o timeout]],","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/Users/jmesnil/Developer/wildfly-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/wildfly/wildfly-operator/pkg/controller/wildflyserver.(*ReconcileWildFlyServer).Reconcile\n\t/Users/jmesnil/Developer/wildfly-operator/pkg/controller/wildflyserver/wildflyserver_controller.go:183\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/Users/jmesnil/Developer/wildfly-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/Users/jmesnil/Developer/wildfly-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/Users/jmesnil/Developer/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/Users/jmesnil/Developer/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/Users/jmesnil/Developer/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1570609332.941532,"logger":"wildflyserver_controller","msg":"Scaling down statefulset by verification if pods are clean by recovery","StatefulSet.Namespace":"default","StatefulSet.Name":"quickstart"}
{"level":"info","ts":1570609332.9416032,"logger":"wildflyserver_controller","msg":"Statefulset was not scaled to the desired replica size 1 (current StatefulSet size: 2). Transaction recovery scaledown process has not cleaned all pods. Please, check status of the WildflyServer quickstart","StatefulSet.Namespace":"default","StatefulSet.Name":"quickstart"}
{"level":"info","ts":1570609332.942085,"logger":"wildflyserver_controller","msg":"Reconciling WildFlyServer","Request.Namespace":"default","Request.Name":"quickstart"}
{"level":"info","ts":1570609332.942118,"logger":"wildlfyserver_resources","msg":"Getting resource","WildFlyServer.Namespace":"default","WildFlyServer.Name":"quickstart","Resource.Name":"quickstart"}
{"level":"info","ts":1570609332.942139,"logger":"wildlfyserver_resources","msg":"Got resource","WildFlyServer.Namespace":"default","WildFlyServer.Name":"quickstart","Resource.Name":"quickstart"}
{"level":"info","ts":1570609332.942218,"logger":"wildflyserver_controller","msg":"Transaction recovery scaledown processing","Request.Namespace":"default","Request.Name":"quickstart","Pod Name":"quickstart-1","IP Address":"172.17.0.6","Pod State":"SCALING_DOWN_RECOVERY_INVESTIGATION","Pod Phase":"Running"}
{"level":"info","ts":1570609332.94224,"logger":"wildflyserver_controller","msg":"Recovery properties at pod were already defined. Skipping server restart.","Request.Namespace":"default","Request.Name":"quickstart","Pod Name":"quickstart-1"}
{"level":"info","ts":1570609333.0835469,"logger":"wildflyserver_controller","msg":"Verification the recovery listener is setup to run transaction recovery at quickstart-1","Request.Namespace":"default","Request.Name":"quickstart"}
{"level":"info","ts":1570609346.274996,"logger":"wildflyserver_controller","msg":"Query to find the transaction recovery port to force scan at pod quickstart-1","Request.Namespace":"default","Request.Name":"quickstart"}
{"level":"info","ts":1570609383.975874,"logger":"wildlfyserver_resources","msg":"Updating Resource","WildFlyServer.Namespace":"default","WildFlyServer.Name":"quickstart","Resource.Name":"quickstart-1","Resource.Type":"*v1.Pod"}
{"level":"info","ts":1570609383.995701,"logger":"wildlfyserver_resources","msg":"Updated resource","WildFlyServer.Namespace":"default","WildFlyServer.Name":"quickstart","Resource.Name":"quickstart-1","Resource.Type":"*v1.Pod"}
{"level":"info","ts":1570609383.99572,"logger":"wildflyserver_controller","msg":"Executing recovery scan at quickstart-1","Request.Namespace":"default","Request.Name":"quickstart","Pod IP":"172.17.0.6","Recovery port":4712}
{"level":"error","ts":1570609414.026708,"logger":"wildflyserver_controller","msg":"Failures during scaling down recovery processing","Request.Namespace":"default","Request.Name":"quickstart","Desired replica size":1,"Number of pods to be removed":1,"error":"Found 1 errors:\n [[Failed to run transaction recovery scan for scaling down pod quickstart-1. Please, verify the pod log file. Error: Cannot process TCP connection to 172.17.0.6:4712, error: dial tcp 172.17.0.6:4712: i/o timeout]],","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/Users/jmesnil/Developer/wildfly-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/wildfly/wildfly-operator/pkg/controller/wildflyserver.(*ReconcileWildFlyServer).Reconcile\n\t/Users/jmesnil/Developer/wildfly-operator/pkg/controller/wildflyserver/wildflyserver_controller.go:183\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/Users/jmesnil/Developer/wildfly-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/Users/jmesnil/Developer/wildfly-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/Users/jmesnil/Developer/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/Users/jmesnil/Developer/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/Users/jmesnil/Developer/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1570609414.0272572,"logger":"wildflyserver_controller","msg":"Scaling down statefulset by verification if pods are clean by recovery","StatefulSet.Namespace":"default","StatefulSet.Name":"quickstart"}
{"level":"info","ts":1570609414.0280929,"logger":"wildflyserver_controller","msg":"Statefulset was not scaled to the desired replica size 1 (current StatefulSet size: 2). Transaction recovery scaledown process has not cleaned all pods. Please, check status of the WildflyServer quickstart","StatefulSet.Namespace":"default","StatefulSet.Name":"quickstart"}
{"level":"info","ts":1570609414.0281641,"logger":"wildflyserver_controller","msg":"Reconciling WildFlyServer","Request.Namespace":"default","Request.Name":"quickstart"}
{"level":"info","ts":1570609414.028191,"logger":"wildlfyserver_resources","msg":"Getting resource","WildFlyServer.Namespace":"default","WildFlyServer.Name":"quickstart","Resource.Name":"quickstart"}
{"level":"info","ts":1570609414.028212,"logger":"wildlfyserver_resources","msg":"Got resource","WildFlyServer.Namespace":"default","WildFlyServer.Name":"quickstart","Resource.Name":"quickstart"}
{"level":"info","ts":1570609414.028299,"logger":"wildflyserver_controller","msg":"Transaction recovery scaledown processing","Request.Namespace":"default","Request.Name":"quickstart","Pod Name":"quickstart-1","IP Address":"172.17.0.6","Pod State":"SCALING_DOWN_RECOVERY_INVESTIGATION","Pod Phase":"Running"}
{"level":"info","ts":1570609414.028814,"logger":"wildflyserver_controller","msg":"Recovery properties at pod were already defined. Skipping server restart.","Request.Namespace":"default","Request.Name":"quickstart","Pod Name":"quickstart-1"}
{"level":"info","ts":1570609414.310617,"logger":"wildflyserver_controller","msg":"Verification the recovery listener is setup to run transaction recovery at quickstart-1","Request.Namespace":"default","Request.Name":"quickstart"}
{"level":"info","ts":1570609480.981805,"logger":"wildflyserver_controller","msg":"Query to find the transaction recovery port to force scan at pod quickstart-1","Request.Namespace":"default","Request.Name":"quickstart"}
{"level":"info","ts":1570609516.8324108,"logger":"wildlfyserver_resources","msg":"Updating Resource","WildFlyServer.Namespace":"default","WildFlyServer.Name":"quickstart","Resource.Name":"quickstart-1","Resource.Type":"*v1.Pod"}
{"level":"info","ts":1570609516.858887,"logger":"wildlfyserver_resources","msg":"Updated resource","WildFlyServer.Namespace":"default","WildFlyServer.Name":"quickstart","Resource.Name":"quickstart-1","Resource.Type":"*v1.Pod"}
{"level":"info","ts":1570609516.858921,"logger":"wildflyserver_controller","msg":"Executing recovery scan at quickstart-1","Request.Namespace":"default","Request.Name":"quickstart","Pod IP":"172.17.0.6","Recovery port":4712}
{"level":"error","ts":1570609546.8717709,"logger":"wildflyserver_controller","msg":"Failures during scaling down recovery processing","Request.Namespace":"default","Request.Name":"quickstart","Desired replica size":1,"Number of pods to be removed":1,"error":"Found 1 errors:\n [[Failed to run transaction recovery scan for scaling down pod quickstart-1. Please, verify the pod log file. Error: Cannot process TCP connection to 172.17.0.6:4712, error: dial tcp 172.17.0.6:4712: i/o timeout]],","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/Users/jmesnil/Developer/wildfly-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/wildfly/wildfly-operator/pkg/controller/wildflyserver.(*ReconcileWildFlyServer).Reconcile\n\t/Users/jmesnil/Developer/wildfly-operator/pkg/controller/wildflyserver/wildflyserver_controller.go:183\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/Users/jmesnil/Developer/wildfly-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/Users/jmesnil/Developer/wildfly-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/Users/jmesnil/Developer/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/Users/jmesnil/Developer/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/Users/jmesnil/Developer/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1570609546.872455,"logger":"wildflyserver_controller","msg":"Scaling down statefulset by verification if pods are clean by recovery","StatefulSet.Namespace":"default","StatefulSet.Name":"quickstart"}
{"level":"info","ts":1570609546.872515,"logger":"wildflyserver_controller","msg":"Statefulset was not scaled to the desired replica size 1 (current StatefulSet size: 2). Transaction recovery scaledown process has not cleaned all pods. Please, check status of the WildflyServer quickstart","StatefulSet.Namespace":"default","StatefulSet.Name":"quickstart"}
{"level":"info","ts":1570609546.873015,"logger":"wildflyserver_controller","msg":"Reconciling WildFlyServer","Request.Namespace":"default","Request.Name":"quickstart"}
{"level":"info","ts":1570609546.873062,"logger":"wildlfyserver_resources","msg":"Getting resource","WildFlyServer.Namespace":"default","WildFlyServer.Name":"quickstart","Resource.Name":"quickstart"}
{"level":"info","ts":1570609546.873087,"logger":"wildlfyserver_resources","msg":"Got resource","WildFlyServer.Namespace":"default","WildFlyServer.Name":"quickstart","Resource.Name":"quickstart"}
{"level":"info","ts":1570609546.8731961,"logger":"wildflyserver_controller","msg":"Transaction recovery scaledown processing","Request.Namespace":"default","Request.Name":"quickstart","Pod Name":"quickstart-1","IP Address":"172.17.0.6","Pod State":"SCALING_DOWN_RECOVERY_INVESTIGATION","Pod Phase":"Running"}
{"level":"info","ts":1570609546.8732271,"logger":"wildflyserver_controller","msg":"Recovery properties at pod were already defined. Skipping server restart.","Request.Namespace":"default","Request.Name":"quickstart","Pod Name":"quickstart-1"}
{"level":"info","ts":1570609546.978324,"logger":"wildflyserver_controller","msg":"Executing recovery scan at quickstart-1","Request.Namespace":"default","Request.Name":"quickstart","Pod IP":"172.17.0.6","Recovery port":4712}
{"level":"info","ts":1570609576.988582,"logger":"wildflyserver_controller","msg":"Cannot update scaledown pod %v while resetting the annotation map to %v","Request.Namespace":"default","Request.Name":"quickstart","quickstart-1":{"recovery-properties-setup":"true","wildfly.org/wildfly-server-generation":"2"}}
{"level":"error","ts":1570609576.988641,"logger":"wildflyserver_controller","msg":"Failures during scaling down recovery processing","Request.Namespace":"default","Request.Name":"quickstart","Desired replica size":1,"Number of pods to be removed":1,"error":"Found 1 errors:\n [[Failed to run transaction recovery scan for scaling down pod quickstart-1. Please, verify the pod log file. Error: Cannot process TCP connection to 172.17.0.6:4712, error: dial tcp 172.17.0.6:4712: i/o timeout]],","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/Users/jmesnil/Developer/wildfly-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/wildfly/wildfly-operator/pkg/controller/wildflyserver.(*ReconcileWildFlyServer).Reconcile\n\t/Users/jmesnil/Developer/wildfly-operator/pkg/controller/wildflyserver/wildflyserver_controller.go:183\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/Users/jmesnil/Developer/wildfly-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/Users/jmesnil/Developer/wildfly-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/Users/jmesnil/Developer/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/Users/jmesnil/Developer/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/Users/jmesnil/Developer/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1570609576.989574,"logger":"wildflyserver_controller","msg":"Scaling down statefulset by verification if pods are clean by recovery","StatefulSet.Namespace":"default","StatefulSet.Name":"quickstart"}
{"level":"info","ts":1570609576.989593,"logger":"wildflyserver_controller","msg":"Statefulset was not scaled to the desired replica size 1 (current StatefulSet size: 2). Transaction recovery scaledown process has not cleaned all pods. Please, check status of the WildflyServer quickstart","StatefulSet.Namespace":"default","StatefulSet.Name":"quickstart"}
{"level":"info","ts":1570609576.989634,"logger":"wildflyserver_controller","msg":"Reconciling WildFlyServer","Request.Namespace":"default","Request.Name":"quickstart"}
{"level":"info","ts":1570609576.989663,"logger":"wildlfyserver_resources","msg":"Getting resource","WildFlyServer.Namespace":"default","WildFlyServer.Name":"quickstart","Resource.Name":"quickstart"}
{"level":"info","ts":1570609576.990089,"logger":"wildlfyserver_resources","msg":"Got resource","WildFlyServer.Namespace":"default","WildFlyServer.Name":"quickstart","Resource.Name":"quickstart"}
{"level":"info","ts":1570609576.9901729,"logger":"wildflyserver_controller","msg":"Transaction recovery scaledown processing","Request.Namespace":"default","Request.Name":"quickstart","Pod Name":"quickstart-1","IP Address":"172.17.0.6","Pod State":"SCALING_DOWN_RECOVERY_INVESTIGATION","Pod Phase":"Running"}
{"level":"info","ts":1570609576.990216,"logger":"wildflyserver_controller","msg":"Recovery properties at pod were already defined. Skipping server restart.","Request.Namespace":"default","Request.Name":"quickstart","Pod Name":"quickstart-1"}
{"level":"info","ts":1570609577.0214589,"logger":"wildflyserver_controller","msg":"Verification the recovery listener is setup to run transaction recovery at quickstart-1","Request.Namespace":"default","Request.Name":"quickstart"}
{"level":"info","ts":1570609583.940879,"logger":"wildflyserver_controller","msg":"Query to find the transaction recovery port to force scan at pod quickstart-1","Request.Namespace":"default","Request.Name":"quickstart"}
{"level":"info","ts":1570609600.952806,"logger":"wildlfyserver_resources","msg":"Updating Resource","WildFlyServer.Namespace":"default","WildFlyServer.Name":"quickstart","Resource.Name":"quickstart-1","Resource.Type":"*v1.Pod"}
{"level":"info","ts":1570609600.986976,"logger":"wildlfyserver_resources","msg":"Updated resource","WildFlyServer.Namespace":"default","WildFlyServer.Name":"quickstart","Resource.Name":"quickstart-1","Resource.Type":"*v1.Pod"}
{"level":"info","ts":1570609600.987,"logger":"wildflyserver_controller","msg":"Executing recovery scan at quickstart-1","Request.Namespace":"default","Request.Name":"quickstart","Pod IP":"172.17.0.6","Recovery port":4712}
{"level":"error","ts":1570609631.057188,"logger":"wildflyserver_controller","msg":"Failures during scaling down recovery processing","Request.Namespace":"default","Request.Name":"quickstart","Desired replica size":1,"Number of pods to be removed":1,"error":"Found 1 errors:\n [[Failed to run transaction recovery scan for scaling down pod quickstart-1. Please, verify the pod log file. Error: Cannot process TCP connection to 172.17.0.6:4712, error: dial tcp 172.17.0.6:4712: i/o timeout]],","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/Users/jmesnil/Developer/wildfly-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/wildfly/wildfly-operator/pkg/controller/wildflyserver.(*ReconcileWildFlyServer).Reconcile\n\t/Users/jmesnil/Developer/wildfly-operator/pkg/controller/wildflyserver/wildflyserver_controller.go:183\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/Users/jmesnil/Developer/wildfly-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/Users/jmesnil/Developer/wildfly-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/Users/jmesnil/Developer/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/Users/jmesnil/Developer/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/Users/jmesnil/Developer/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1570609631.0576699,"logger":"wildflyserver_controller","msg":"Scaling down statefulset by verification if pods are clean by recovery","StatefulSet.Namespace":"default","StatefulSet.Name":"quickstart"}
{"level":"info","ts":1570609631.058065,"logger":"wildflyserver_controller","msg":"Statefulset was not scaled to the desired replica size 1 (current StatefulSet size: 2). Transaction recovery scaledown process has not cleaned all pods. Please, check status of the WildflyServer quickstart","StatefulSet.Namespace":"default","StatefulSet.Name":"quickstart"}
{"level":"info","ts":1570609631.058101,"logger":"wildflyserver_controller","msg":"Reconciling WildFlyServer","Request.Namespace":"default","Request.Name":"quickstart"}
{"level":"info","ts":1570609631.058122,"logger":"wildlfyserver_resources","msg":"Getting resource","WildFlyServer.Namespace":"default","WildFlyServer.Name":"quickstart","Resource.Name":"quickstart"}
{"level":"info","ts":1570609631.0581372,"logger":"wildlfyserver_resources","msg":"Got resource","WildFlyServer.Namespace":"default","WildFlyServer.Name":"quickstart","Resource.Name":"quickstart"}
{"level":"info","ts":1570609631.058197,"logger":"wildflyserver_controller","msg":"Transaction recovery scaledown processing","Request.Namespace":"default","Request.Name":"quickstart","Pod Name":"quickstart-1","IP Address":"172.17.0.6","Pod State":"SCALING_DOWN_RECOVERY_INVESTIGATION","Pod Phase":"Running"}
{"level":"info","ts":1570609631.058563,"logger":"wildflyserver_controller","msg":"Recovery properties at pod were already defined. Skipping server restart.","Request.Namespace":"default","Request.Name":"quickstart","Pod Name":"quickstart-1"}
{"level":"info","ts":1570609631.1286862,"logger":"wildflyserver_controller","msg":"Verification the recovery listener is setup to run transaction recovery at quickstart-1","Request.Namespace":"default","Request.Name":"quickstart"}
{"level":"info","ts":1570609660.518332,"logger":"wildflyserver_controller","msg":"Query to find the transaction recovery port to force scan at pod quickstart-1","Request.Namespace":"default","Request.Name":"quickstart"}
{"level":"info","ts":1570609674.827886,"logger":"wildlfyserver_resources","msg":"Updating Resource","WildFlyServer.Namespace":"default","WildFlyServer.Name":"quickstart","Resource.Name":"quickstart-1","Resource.Type":"*v1.Pod"}
{"level":"info","ts":1570609674.89456,"logger":"wildlfyserver_resources","msg":"Updated resource","WildFlyServer.Namespace":"default","WildFlyServer.Name":"quickstart","Resource.Name":"quickstart-1","Resource.Type":"*v1.Pod"}
{"level":"info","ts":1570609674.8945868,"logger":"wildflyserver_controller","msg":"Executing recovery scan at quickstart-1","Request.Namespace":"default","Request.Name":"quickstart","Pod IP":"172.17.0.6","Recovery port":4712}
{"level":"error","ts":1570609704.906108,"logger":"wildflyserver_controller","msg":"Failures during scaling down recovery processing","Request.Namespace":"default","Request.Name":"quickstart","Desired replica size":1,"Number of pods to be removed":1,"error":"Found 1 errors:\n [[Failed to run transaction recovery scan for scaling down pod quickstart-1. Please, verify the pod log file. Error: Cannot process TCP connection to 172.17.0.6:4712, error: dial tcp 172.17.0.6:4712: i/o timeout]],","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/Users/jmesnil/Developer/wildfly-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/wildfly/wildfly-operator/pkg/controller/wildflyserver.(*ReconcileWildFlyServer).Reconcile\n\t/Users/jmesnil/Developer/wildfly-operator/pkg/controller/wildflyserver/wildflyserver_controller.go:183\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/Users/jmesnil/Developer/wildfly-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/Users/jmesnil/Developer/wildfly-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/Users/jmesnil/Developer/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/Users/jmesnil/Developer/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/Users/jmesnil/Developer/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1570609704.906705,"logger":"wildflyserver_controller","msg":"Scaling down statefulset by verification if pods are clean by recovery","StatefulSet.Namespace":"default","StatefulSet.Name":"quickstart"}
{"level":"info","ts":1570609704.907205,"logger":"wildflyserver_controller","msg":"Statefulset was not scaled to the desired replica size 1 (current StatefulSet size: 2). Transaction recovery scaledown process has not cleaned all pods. Please, check status of the WildflyServer quickstart","StatefulSet.Namespace":"default","StatefulSet.Name":"quickstart"}
{"level":"info","ts":1570609704.9072561,"logger":"wildflyserver_controller","msg":"Reconciling WildFlyServer","Request.Namespace":"default","Request.Name":"quickstart"}
{"level":"info","ts":1570609704.9072928,"logger":"wildlfyserver_resources","msg":"Getting resource","WildFlyServer.Namespace":"default","WildFlyServer.Name":"quickstart","Resource.Name":"quickstart"}
{"level":"info","ts":1570609704.907316,"logger":"wildlfyserver_resources","msg":"Got resource","WildFlyServer.Namespace":"default","WildFlyServer.Name":"quickstart","Resource.Name":"quickstart"}
{"level":"info","ts":1570609704.907434,"logger":"wildflyserver_controller","msg":"Transaction recovery scaledown processing","Request.Namespace":"default","Request.Name":"quickstart","Pod Name":"quickstart-1","IP Address":"172.17.0.6","Pod State":"SCALING_DOWN_RECOVERY_INVESTIGATION","Pod Phase":"Running"}
{"level":"info","ts":1570609704.907469,"logger":"wildflyserver_controller","msg":"Recovery properties at pod were already defined. Skipping server restart.","Request.Namespace":"default","Request.Name":"quickstart","Pod Name":"quickstart-1"}
{"level":"info","ts":1570609704.97966,"logger":"wildflyserver_controller","msg":"Executing recovery scan at quickstart-1","Request.Namespace":"default","Request.Name":"quickstart","Pod IP":"172.17.0.6","Recovery port":4712}
{"level":"info","ts":1570609734.990844,"logger":"wildflyserver_controller","msg":"Cannot update scaledown pod %v while resetting the annotation map to %v","Request.Namespace":"default","Request.Name":"quickstart","quickstart-1":{"recovery-properties-setup":"true","wildfly.org/wildfly-server-generation":"2"}}
{"level":"error","ts":1570609734.990901,"logger":"wildflyserver_controller","msg":"Failures during scaling down recovery processing","Request.Namespace":"default","Request.Name":"quickstart","Desired replica size":1,"Number of pods to be removed":1,"error":"Found 1 errors:\n [[Failed to run transaction recovery scan for scaling down pod quickstart-1. Please, verify the pod log file. Error: Cannot process TCP connection to 172.17.0.6:4712, error: dial tcp 172.17.0.6:4712: i/o timeout]],","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/Users/jmesnil/Developer/wildfly-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/wildfly/wildfly-operator/pkg/controller/wildflyserver.(*ReconcileWildFlyServer).Reconcile\n\t/Users/jmesnil/Developer/wildfly-operator/pkg/controller/wildflyserver/wildflyserver_controller.go:183\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/Users/jmesnil/Developer/wildfly-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/Users/jmesnil/Developer/wildfly-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/Users/jmesnil/Developer/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/Users/jmesnil/Developer/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/Users/jmesnil/Developer/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1570609734.991883,"logger":"wildflyserver_controller","msg":"Scaling down statefulset by verification if pods are clean by recovery","StatefulSet.Namespace":"default","StatefulSet.Name":"quickstart"}
{"level":"info","ts":1570609734.991981,"logger":"wildflyserver_controller","msg":"Statefulset was not scaled to the desired replica size 1 (current StatefulSet size: 2). Transaction recovery scaledown process has not cleaned all pods. Please, check status of the WildflyServer quickstart","StatefulSet.Namespace":"default","StatefulSet.Name":"quickstart"}
{"level":"info","ts":1570609734.992033,"logger":"wildflyserver_controller","msg":"Reconciling WildFlyServer","Request.Namespace":"default","Request.Name":"quickstart"}
{"level":"info","ts":1570609734.992064,"logger":"wildlfyserver_resources","msg":"Getting resource","WildFlyServer.Namespace":"default","WildFlyServer.Name":"quickstart","Resource.Name":"quickstart"}
{"level":"info","ts":1570609734.992089,"logger":"wildlfyserver_resources","msg":"Got resource","WildFlyServer.Namespace":"default","WildFlyServer.Name":"quickstart","Resource.Name":"quickstart"}
{"level":"info","ts":1570609734.99261,"logger":"wildflyserver_controller","msg":"Transaction recovery scaledown processing","Request.Namespace":"default","Request.Name":"quickstart","Pod Name":"quickstart-1","IP Address":"172.17.0.6","Pod State":"SCALING_DOWN_RECOVERY_INVESTIGATION","Pod Phase":"Running"}
{"level":"info","ts":1570609734.992639,"logger":"wildflyserver_controller","msg":"Recovery properties at pod were already defined. Skipping server restart.","Request.Namespace":"default","Request.Name":"quickstart","Pod Name":"quickstart-1"}
{"level":"info","ts":1570609735.028022,"logger":"wildflyserver_controller","msg":"Verification the recovery listener is setup to run transaction recovery at quickstart-1","Request.Namespace":"default","Request.Name":"quickstart"}
{"level":"info","ts":1570609747.660465,"logger":"wildflyserver_controller","msg":"Query to find the transaction recovery port to force scan at pod quickstart-1","Request.Namespace":"default","Request.Name":"quickstart"}
{"level":"info","ts":1570609767.905767,"logger":"wildlfyserver_resources","msg":"Updating Resource","WildFlyServer.Namespace":"default","WildFlyServer.Name":"quickstart","Resource.Name":"quickstart-1","Resource.Type":"*v1.Pod"}
{"level":"info","ts":1570609767.952047,"logger":"wildlfyserver_resources","msg":"Updated resource","WildFlyServer.Namespace":"default","WildFlyServer.Name":"quickstart","Resource.Name":"quickstart-1","Resource.Type":"*v1.Pod"}
{"level":"info","ts":1570609767.952076,"logger":"wildflyserver_controller","msg":"Executing recovery scan at quickstart-1","Request.Namespace":"default","Request.Name":"quickstart","Pod IP":"172.17.0.6","Recovery port":4712}
{"level":"error","ts":1570609798.004528,"logger":"wildflyserver_controller","msg":"Failures during scaling down recovery processing","Request.Namespace":"default","Request.Name":"quickstart","Desired replica size":1,"Number of pods to be removed":1,"error":"Found 1 errors:\n [[Failed to run transaction recovery scan for scaling down pod quickstart-1. Please, verify the pod log file. Error: Cannot process TCP connection to 172.17.0.6:4712, error: dial tcp 172.17.0.6:4712: i/o timeout]],","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/Users/jmesnil/Developer/wildfly-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/wildfly/wildfly-operator/pkg/controller/wildflyserver.(*ReconcileWildFlyServer).Reconcile\n\t/Users/jmesnil/Developer/wildfly-operator/pkg/controller/wildflyserver/wildflyserver_controller.go:183\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/Users/jmesnil/Developer/wildfly-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/Users/jmesnil/Developer/wildfly-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/Users/jmesnil/Developer/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/Users/jmesnil/Developer/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/Users/jmesnil/Developer/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1570609876.28108,"logger":"wildflyserver_controller","msg":"Scaling down statefulset by verification if pods are clean by recovery","StatefulSet.Namespace":"default","StatefulSet.Name":"quickstart"}
{"level":"info","ts":1570609876.281184,"logger":"wildflyserver_controller","msg":"Statefulset was not scaled to the desired replica size 1 (current StatefulSet size: 2). Transaction recovery scaledown process has not cleaned all pods. Please, check status of the WildflyServer quickstart","StatefulSet.Namespace":"default","StatefulSet.Name":"quickstart"}
{"level":"info","ts":1570609876.282038,"logger":"wildflyserver_controller","msg":"Reconciling WildFlyServer","Request.Namespace":"default","Request.Name":"quickstart"}
{"level":"info","ts":1570609876.282075,"logger":"wildlfyserver_resources","msg":"Getting resource","WildFlyServer.Namespace":"default","WildFlyServer.Name":"quickstart","Resource.Name":"quickstart"}
{"level":"info","ts":1570609876.282134,"logger":"wildlfyserver_resources","msg":"Got resource","WildFlyServer.Namespace":"default","WildFlyServer.Name":"quickstart","Resource.Name":"quickstart"}
{"level":"info","ts":1570609876.282273,"logger":"wildflyserver_controller","msg":"Transaction recovery scaledown processing","Request.Namespace":"default","Request.Name":"quickstart","Pod Name":"quickstart-1","IP Address":"172.17.0.6","Pod State":"SCALING_DOWN_RECOVERY_INVESTIGATION","Pod Phase":"Running"}
{"level":"info","ts":1570609876.2823071,"logger":"wildflyserver_controller","msg":"Recovery properties at pod were already defined. Skipping server restart.","Request.Namespace":"default","Request.Name":"quickstart","Pod Name":"quickstart-1"}

{"level":"info","ts":1570609876.660885,"logger":"wildflyserver_controller","msg":"Verification the recovery listener is setup to run transaction recovery at quickstart-1","Request.Namespace":"default","Request.Name":"quickstart"}
{"level":"info","ts":1570609937.733844,"logger":"wildflyserver_controller","msg":"Query to find the transaction recovery port to force scan at pod quickstart-1","Request.Namespace":"default","Request.Name":"quickstart"}

[jmesnil]

@ochaloup fiy, this can be reproduced and I have not the same issue with WildFly 17 S2I.

[ochaloup]

@jmesnil I'm trying to reproduce what you observe. I didn't follow your exact setup as I use the codeready and master branch of the operator. But still I was not able to reproduce the issue.

From the log I can see that the failures happens on socket dial on the podIP:4712 to execute recovery scan. The call fails after 30s which is timeout time to dial to that ip address : port. It's just strange that you don't hit the same for the WFLY17.

I will continue in investigation the next day where I'll try to run the exact branch and the minikube.

[ochaloup]

@jmesnil after some struggle I reproduced the issue and the trouble is that the operator runs on a different network from where the pods run. The operator needs to connect directly to the pod and call to the socket. This is not possible as the minikube runs in the virtual machine and the operator runs on the localhost. If the network is setup for the localhost may connect to the IPs (or DNSes) to the virtual machine the socket works.
When the operator and the pods run in the same space - Kubernetes defines the network as flat from what I know - then the dial to the socket works.

This issue should be closed.

Unfortunately currently it's not possible to run operaror locally and process scaledown.
What will help is when the https://issues.jboss.org/browse/JBEAP-17611 is done. That will mean that there is only CLI calls which goes over the kubernetes API server and that's the reason (I assume) to be accesible over the network from localhost to the virtualized environment.

[jmesnil]

why is this issue not happening with WildFly 17 S2I?

[ochaloup]

@jmesnil because the WFLY17 S2I does not define the recovery-listener. When the listener is not defined then no recovery is launched. The transactions are left unfinished and scale down proceed. See #75 (comment)

[jmesnil]

@ochaloup This is another good reason to provide a proper management operator for recovery scan....

[ochaloup]

I do agree and I plan to work on the issue JBEAP-17611 soon ;-)

[jmesnil]

ok so that means that I'll comment the scale down test for WildFly 18 S2I until it is possible for the operator to issue a recovery scan in WildFly using a management operation (targeting WildFly 19 then)

[ochaloup]

@jmesnil I don't think it's a good idea. The e2e test should be still working. What does not work is running the operator locally on localhost and the rest as part of minikube. If the operator and the pods are at the same network (as it's usual openshift/kubernetes deployment) then all works fine.

I would really be happy if we can have the scale down test enabled.