K8s Secret Inline Mutation annotation sometimes is not respected

[kajov]

Describe the bug:
Vault Secrets webhook is sometimes not respecting k8s annotation for vault-env

annotation provided for the Secret object to have inline mutated:
http://vault.vault.svc.cluster.local:8200

Error thrown:

time="2023-05-31T18:22:28Z" level=error msg="failed to request new Vault token" app=vault-secrets-webhook err="Put \"https://vault:8200/v1/auth/kubernetes/login\": http: server gave HTTP response to HTTPS client"
time="2023-05-31T18:22:33Z" level=error msg="failed to request new Vault token" app=vault-secrets-webhook err="Put \"https://vault:8200/v1/auth/kubernetes/login\": http: server gave HTTP response to HTTPS client"
time="2023-05-31T18:22:33Z" level=error msg="Admission review request failed" app=vault-secrets-webhook dry-run=false error="failed to create vault client: timeout [10s] during waiting for Vault token" kind=v1/Secret name=keycloak-postgresql ns=iam op=update path=/secrets request-id=06d88778-eaa5-4731-aeb9-879a30b8abc6 webhook-id=vault-secrets-secret webhook-kind=mutating wh-version=v1beta1
ti

Expected behaviour:
Inline mutation should put the value of the Secret in the k8s Secret Object

Steps to reproduce the bug:
Continuously trigger the keycloak configurator job or any kind of secret creation via helm chart webhook then randomly would not respect the annotations vault-addr and default to the original https://vault:8200

Additional Information:

Vault is running with tls_disabled: true
Vault server is responding with HTTP
We are exposing vault via Ambassador external DNS

Environment details:

• Kubernetes version: v1.23.17-eks-a59e1f0
• Cloud-provider/provisioner: EKS
• bank-vaults version (e.g. 0.4.17): Official Hashicorp Vault v1.13.0, vault-secrets-webhook - v1.19.0
• Install method (e.g. helm or static manifests): Helm
• Logs from the misbehaving component (and any other relevant logs): Relevant block is in the first lines
• Resource definition (possibly in YAML format) that caused the issue, without sensitive data:
  Can be provided if necessary, for the time being, do not have clearance to do so

/kind bug

[kajov]

Additional note:

Hashicopr Vault official helm chart deploys a couple of Vault services.
Service 1: vault with port open on 8200 and nodePort/LB ready to bind to it
Service 2: vault-internal bound to port 8200 no external port, usually this service from Hashicorp Vault documentation is used when Vault is being configured to run in HA mode.

Individual check on the Ambassador debug admin console, indicates an error when annotation is added to Service 1 that it can be routed to Service 2, disabling Service 2 is optional, but HA mode will not work if it is disabled.

Tested out with removing Service 2 did not cause the problem to be resolved, which eliminates the possibility of being a wrong service routing on the Kubernetes/Ambassador part.