VDS: Log and record Vault request failures

[benashz]

The VDS controller failed to report whenever a secret sync action had failed. With this PR all secret failures are recorded to the event recorder and logged.

Sample log output:

2023-12-08T13:26:25Z    ERROR   syncSecret      Vault request failed    {"controller": "vaultdynamicsecret", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "VaultDynamicSecret", "VaultDynamicSecret": {"name":"create-static-create-static-creds-0","namespace":"vds-a1xdxdsw4y-k8s-ns"}, "namespace": "vds-a1xdxdsw4y-k8s-ns", "name": "create-static-create-static-creds-0", "reconcileID": "24d2a885-2123-4295-a2f3-98333ff59f10", "path": "vds-a1xdxdsw4y-db/static-creds/dev-postgres-static", "method": "GET", "error": "Error making API request.\n\nURL: GET http://vault.vault.svc.cluster.local:8200/v1/vds-a1xdxdsw4y-db/static-creds/dev-postgres-static\nCode: 403. Errors:\n\n* 1 error occurred:\n\t* permission denied\n\n"}
github.com/hashicorp/vault-secrets-operator/controllers.(*VaultDynamicSecretReconciler).syncSecret
        /workspace/controllers/vaultdynamicsecret_controller.go:304
github.com/hashicorp/vault-secrets-operator/controllers.(*VaultDynamicSecretReconciler).Reconcile
        /workspace/controllers/vaultdynamicsecret_controller.go:210
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227

Sample event:

6m34s       Warning   SecretSyncError   vaultdynamicsecret/create-static-create-static-creds-0   Failed to sync the secret, horizon=6.60288929s, err=Error making API request....

Fixes #507

[adrianmoisey]

This fix looks like exactly what I was looking for, thanks!

[kschoche]

Lgtm