K8SPXC-1159: Wrong cluster status when invalid user password applied

[inelpandzic]

user password applied.

CHANGE DESCRIPTION

Problem:
When a user applies an invalid password (rejected by MySQL), we return the error and set the cluster to Error state.
But in the next reconcile loop, we don't return the error from r.reconcileUsers and move on with the reconciliation, we then set the status to Ready. This results in a flipping cluster status, like this:

cluster1   cluster1-haproxy.default   error     3                3         3h
cluster1   cluster1-haproxy.default   ready    3                3         3h
cluster1   cluster1-haproxy.default   error     3                3         3h
cluster1   cluster1-haproxy.default   ready    3                3         3h
cluster1   cluster1-haproxy.default   error     3                3         3h

The reason why we end up in the ready state again is when handling users we check if the cluster is ready and only then proceed and since now it is in the Error state, we just return nil and don't do anything.
But now again in the next reconciliation, the cluster is Ready, we proceed with updating the user pass that is still invalid, we return an error and the cluster ends up again in the Error state. That's how we end up flipping like this.

Solution:
Now when checking if the cluster is ready or not before handling users, we also check if the invalid password was applied, if it is, we proceed with handling the user to try updating it again. We check the error message in the status to see if the error is an invalid password.

CHECKLIST

Jira

• Is the Jira ticket created and referenced properly?
• Does the Jira ticket have the proper statuses for documentation (Needs Doc) and QA (Needs QA)?
• Does the Jira ticket link to the proper milestone (Fix Version field)?

Tests

• Is an E2E test/test case added for the new feature/change?
• Are unit tests added where appropriate?
• Are OpenShift compare files changed for E2E tests (compare/*-oc.yml)?

Config/Logging/Testability

• Are all needed new/changed options added to default YAML files?
• Are the manifests (crd/bundle) regenerated if needed?
• Did we add proper logging messages for operator actions?
• Did we ensure compatibility with the previous version or cluster upgrade process?
• Does the change support oldest and newest supported PXC version?
• Does the change support oldest and newest supported Kubernetes version?

[JNKPercona]

Test name	Status
affinity-8-0	passed
auto-tuning-8-0	passed
cross-site-8-0	failure
demand-backup-cloud-8-0	passed
demand-backup-encrypted-with-tls-8-0	passed
demand-backup-8-0	passed
haproxy-5-7	passed
haproxy-8-0	passed
init-deploy-5-7	passed
init-deploy-8-0	passed
limits-8-0	passed
monitoring-2-0-8-0	failure
one-pod-5-7	passed
one-pod-8-0	passed
pitr-8-0	passed
pitr-gap-errors-8-0	passed
proxy-protocol-8-0	passed
proxysql-sidecar-res-limits-8-0	passed
recreate-8-0	passed
restore-to-encrypted-cluster-8-0	passed
scaling-proxysql-8-0	passed
scaling-8-0	passed
scheduled-backup-5-7	passed
scheduled-backup-8-0	passed
security-context-8-0	passed
smart-update1-8-0	passed
smart-update2-8-0	passed
storage-8-0	passed
tls-issue-cert-manager-ref-8-0	passed
tls-issue-cert-manager-8-0	passed
tls-issue-self-8-0	passed
upgrade-consistency-8-0	passed
upgrade-haproxy-8-0	passed
upgrade-proxysql-8-0	passed
users-5-7	passed
users-8-0	passed
validation-hook-8-0	passed
We run 37 out of 37

commit: ae11ec9
image: perconalab/percona-xtradb-cluster-operator:PR-1543-ae11ec9a