fernet.InvalidToken after redeploy awx on existing postgres pvc

[shaikabdulm]

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that the AWX Operator is open source software provided for free and that I might not receive a timely response.

Bug Summary

I have awx deployed on cloud instance running on k8s with persistent storage for postgresql database. As part of security practices, redeployed instance with same image and ipaddress.
Able to use the existing persistent storage claim to deploy.
Issue is when I try to launch job, getting error

2024-10-24 17:21:46,264 INFO [-] awx.main.wsrelay Starting producer for metrics
2024-10-24 17:22:29,849 WARNING [-] awx.main.tasks.system Heartbeat skew - interval=62.4549, expected=60
min_value in DecimalField should be Decimal type.
2024-10-24 17:22:30,020 INFO [-] awx.main.tasks.system Host awx-task-6f859bcf77-pfnhc Automatically Deprovisioned.
2024-10-24 17:31:24,789 INFO [8cbc4e8fcbff45aa9e1c50bc935d1754] awx.analytics.job_lifecycle job-9 waiting {"type": "job", "task_id": 9, "state": "waiting", "work_unit_id": nu
ll, "task_name": "job_test_debug"}
2024-10-24 17:31:26,029 INFO [8cbc4e8fcbff45aa9e1c50bc935d1754] awx.analytics.job_lifecycle job-9 pre run {"type": "job", "task_id": 9, "state": "pre_run", "work_unit_id": nu
ll, "task_name": "job_test_debug"}
2024-10-24 17:31:26,293 INFO [8cbc4e8fcbff45aa9e1c50bc935d1754] awx.analytics.job_lifecycle job-9 preparing playbook {"type": "job", "task_id": 9, "state": "preparing_playboo
k", "work_unit_id": null, "task_name": "job_test_debug"}
2024-10-24 17:31:26,462 ERROR [8cbc4e8fcbff45aa9e1c50bc935d1754] awx.main.utils.encryption Failed to decrypt Credential(pk=3).password; if you've recently restored from a da
tabase backup or are running in a clustered environment, check that your SECRET_KEY value is correct
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/cryptography/fernet.py", line 134, in _verify_signature
h.verify(data[-32:])
cryptography.exceptions.InvalidSignature: Signature did not match digest.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/utils/encryption.py", line 159, in decrypt_field
return smart_str(decrypt_value(key, value))
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/utils/encryption.py", line 136, in decrypt_value
value = f.decrypt(encrypted)
^^^^^^^^^^^^^^^^^^^^
File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/cryptography/fernet.py", line 91, in decrypt
return self._decrypt_data(data, timestamp, time_info)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/cryptography/fernet.py", line 152, in _decrypt_data
self._verify_signature(data)
File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/cryptography/fernet.py", line 136, in _verify_signature
raise InvalidToken
cryptography.fernet.InvalidToken
2024-10-24 17:31:26,516 ERROR [8cbc4e8fcbff45aa9e1c50bc935d1754] awx.main.tasks.jobs job 9 (running) Exception occurred while running task
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/cryptography/fernet.py", line 134, in _verify_signature
h.verify(data[-32:])
cryptography.exceptions.InvalidSignature: Signature did not match digest.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/tasks/jobs.py", line 517, in run
passwords = self.build_passwords(self.instance, kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/tasks/jobs.py", line 855, in build_passwords
value = runtime_passwords.get(field, cred.get_input('password' if field == 'ssh_password' else field, default=''))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/models/credential/init.py", line 283, in get_input
return decrypt_field(self, field_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/utils/encryption.py", line 159, in decrypt_field
return smart_str(decrypt_value(key, value))

File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/utils/encryption.py", line 136, in decrypt_value
value = f.decrypt(encrypted)
        ^^^^^^^^^^^^^^^^^^^^

File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/cryptography/fernet.py", line 91, in decrypt
return self._decrypt_data(data, timestamp, time_info)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/cryptography/fernet.py", line 152, in _decrypt_data
self._verify_signature(data)
File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/cryptography/fernet.py", line 136, in _verify_signature
raise InvalidToken
cryptography.fernet.InvalidToken
2024-10-24 17:31:26,522 INFO [8cbc4e8fcbff45aa9e1c50bc935d1754] awx.analytics.job_lifecycle job-9 post run {"type": "job", "task_id": 9, "state": "post_run", "work_unit_id": null, "task_name": "job_test_debug"}
2024-10-24 17:31:26,895 INFO [8cbc4e8fcbff45aa9e1c50bc935d1754] awx.analytics.job_lifecycle job-9 finalize run {"type": "job", "task_id": 9, "state": "finalize_run", "work_unit_id": null, "task_name": "job_test_debug"}

My deployment details:
OS: Debian12
kubernetes:
kubectl version
Client Version: v1.29.5
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.5

AWX operator deployment:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:

Find the latest tag here: https://github.com/ansible/awx-operator/releases

• github.com/ansible/awx-operator/config/default?ref=2.18.0
  #- awx-task-pv.yaml
  #- awx-task-pvc.yaml
  #- awx-postgres-pv.yaml
  #- awx-postgres-pvc.yaml
  #- awx-deployment.yaml

Set the image tags to match the git version from above

images:

• name: quay.io/ansible/awx-operator
  newTag: 2.18.0

Specify a custom namespace in which to install AWX

namespace: awx

AWX Operator version

2.18

AWX version

24.5.0

Kubernetes platform

kubernetes

Kubernetes/Platform version

kubectl version Client Version: v1.29.5 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.29.5

Modifications

yes

Steps to reproduce

apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
name: awx
namespace: awx
spec:

Awx secret for redploying

#secret_key: "my-secret-key"
secret_key_secret: "IKM8WW2FxTfkAmck167vNJFzbOI76mQA"
old_postgres_configuration_secret: "database-secret-key"

Custom nodeport 30786 for awx service

service_type: nodeport
nodeport_port: 30786

Persistent volumes

postgres_storage_class: "manual"
projects_persistence: true
projects_existing_claim: "awx-web-pvc"

Expected results

Ansible job need to be executed without error

Actual results

File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/cryptography/fernet.py", line 152, in _decrypt_data
self._verify_signature(data)
File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/cryptography/fernet.py", line 136, in _verify_signature
raise InvalidToken
cryptography.fernet.InvalidToken

Additional information

No response

Operator Logs

No response

[shaikabdulm]

Verified the secret key is same after re-deploying awx

kubectl -n awx exec -i -t awx-task-6f859bcf77-6bnv5 -c awx-task -- bash
bash-5.1# awx-manage shell_plus --quiet
Python 3.11.7 (main, Jan 22 2024, 00:00:00) [GCC 11.4.1 20231218 (Red Hat 11.4.1-3)] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)

settings.SECRET_KEY
b'IKM8WW2FxTfkAmck167vNJFzbOI76mQA'
quit()
bash-5.1# cat /etc/tower/SECRET_KEY
IKM8WW2FxTfkAmck167vNJFzbOI76mQA
bash-5.1#

Created secrets using same manifest files before re-deployment

cat awx-postgres-configuration.yaml
apiVersion: v1
kind: Secret
type: Opaque
data:

the data is abbreviated in this example

password: "my-secret-database"
database: "YXd4"
host: "YXd4LXBvc3RncmVzLTE1"
port: "NTQzMg=="
type: "bWFuYWdlZA=="
username: "YXd4"
metadata:
name: awx-postgres-configuration
namespace: awx
labels:
app.kubernetes.io/component: awx
app.kubernetes.io/managed-by: awx-operator
app.kubernetes.io/operator-version: 2.18.0
app.kubernetes.io/part-of: awx
name: awx-postgres-configuration
namespace: awx