Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Aws secretsmanager additions #6381

Open
wants to merge 32 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 14 commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
655471b
Add SecretKey to AWS SecretsManager TriggerAuthentication spec
nrichardson-akasa Aug 2, 2024
8025fd3
Update CHANGELOG
nrichardson-akasa Aug 2, 2024
fd88dd7
Fix linting issue
nrichardson-akasa Aug 2, 2024
124e48a
Update E2E tests
nrichardson-akasa Aug 5, 2024
b9951d1
Update E2E tests
nrichardson-akasa Aug 5, 2024
b1421f3
Merge branch 'main' into aws-secretsmanager-additions
nrichardson-akasa Aug 5, 2024
dabf94c
Update E2E tests
nrichardson-akasa Aug 5, 2024
23175d8
Merge branch 'main' into aws-secretsmanager-additions
nrichardson-akasa Aug 6, 2024
f1717da
fixed mixedxe tab/space issue
mpechner-akasa Nov 29, 2024
f113aac
reverted back to exportloopref
mpechner-akasa Nov 29, 2024
2a7211b
Merge branch 'main' into aws-secretsmanager-additions
mpechner-akasa Nov 29, 2024
7f81639
Merge branch 'main' into aws-secretsmanager-additions
mpechner-akasa Dec 2, 2024
b3272b9
Merge branch 'main' into aws-secretsmanager-additions
mpechner-akasa Dec 2, 2024
a6b4067
Merge branch 'main' into aws-secretsmanager-additions
mpechner-akasa Dec 6, 2024
1f34e8f
remoace tab with spaces
mpechner-akasa Dec 16, 2024
decfda0
Merge branch 'main' into aws-secretsmanager-additions
mpechner-akasa Dec 17, 2024
3e92f5c
Needed the hlper added in here
mpechner-akasa Dec 17, 2024
45e365c
Merge branch 'aws-secretsmanager-additions' of github.com:nrichardson…
mpechner-akasa Dec 17, 2024
e6ec7c4
only changed awssecreetmanager_test.go to see if my approach is corr…
mpechner-akasa Dec 22, 2024
a0a839a
nd keeps removing
mpechner-akasa Dec 22, 2024
f294e18
Not sure what the correct way to import the local helper is.
mpechner-akasa Dec 22, 2024
fe3d095
Trying blank import.
mpechner-akasa Dec 23, 2024
1b1c8c7
fix path "github.com/kedacore/keda/v2/tests/helper"
mpechner-akasa Dec 23, 2024
07fa347
added call to GetRandomNumber() just to keep goimport from removeing …
mpechner-akasa Dec 23, 2024
6791609
really am a noob.
mpechner-akasa Dec 23, 2024
98e4d32
another noob error.
mpechner-akasa Dec 23, 2024
f8f8bac
really am gonna make every dumb mistake.
mpechner-akasa Dec 23, 2024
455d5cf
fixed return value
mpechner-akasa Dec 23, 2024
3034840
thought I fixed that.
mpechner-akasa Dec 23, 2024
9dbb6be
figured goland would have pointed these out
mpechner-akasa Dec 23, 2024
f8e3a1b
Merge branch 'main' into aws-secretsmanager-additions
mpechner-akasa Dec 23, 2024
382a309
Merge branch 'main' into aws-secretsmanager-additions
mpechner-akasa Dec 25, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,7 @@ Here is an overview of all new **experimental** features:

### Improvements

- General: Add SecretKey to AWS SecretsManager TriggerAuthentication to allow parsing JSON / Key/Value Pairs in secrets (#5940)
- TODO ([#XXX](https://github.com/kedacore/keda/issues/XXX))

### Fixes
Expand Down
2 changes: 2 additions & 0 deletions apis/keda/v1alpha1/triggerauthentication_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -376,6 +376,8 @@ type AwsSecretManagerSecret struct {
VersionID string `json:"versionId,omitempty"`
// +optional
VersionStage string `json:"versionStage,omitempty"`
// +optional
SecretKey string `json:"secretKey,omitempty"`
}

func init() {
Expand Down
2 changes: 2 additions & 0 deletions config/crd/bases/keda.sh_clustertriggerauthentications.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -183,6 +183,8 @@ spec:
type: string
parameter:
type: string
secretKey:
type: string
versionId:
type: string
versionStage:
Expand Down
2 changes: 2 additions & 0 deletions config/crd/bases/keda.sh_triggerauthentications.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -182,6 +182,8 @@ spec:
type: string
parameter:
type: string
secretKey:
type: string
versionId:
type: string
versionStage:
Expand Down
26 changes: 24 additions & 2 deletions pkg/scaling/resolver/aws_secretmanager_handler.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ package resolver

import (
"context"
"encoding/json"
"fmt"

"github.com/aws/aws-sdk-go-v2/aws"
Expand All @@ -43,9 +44,9 @@ func NewAwsSecretManagerHandler(a *kedav1alpha1.AwsSecretManager) *AwsSecretMana
}
}

// Read fetches the secret value from AWS Secret Manager using the provided secret name, version ID(optional), and version stage(optional).
// Read fetches the secret value from AWS Secret Manager using the provided secret name, version ID(optional), version stage(optional), and secretKey(optional).
// It returns the secret value as a string.
func (ash *AwsSecretManagerHandler) Read(ctx context.Context, logger logr.Logger, secretName, versionID, versionStage string) (string, error) {
func (ash *AwsSecretManagerHandler) Read(ctx context.Context, logger logr.Logger, secretName, versionID, versionStage string, secretKey string) (string, error) {
input := &secretsmanager.GetSecretValueInput{
SecretId: aws.String(secretName),
}
Expand All @@ -60,6 +61,27 @@ func (ash *AwsSecretManagerHandler) Read(ctx context.Context, logger logr.Logger
logger.Error(err, "Error getting credentials")
return "", err
}
if secretKey != "" {
// Parse the secret string as JSON
var secretMap map[string]interface{}
err = json.Unmarshal([]byte(*result.SecretString), &secretMap)
if err != nil {
logger.Error(err, "Error parsing secret string as JSON")
return "", err
}

// Check if the specified secret key exists
if val, ok := secretMap[secretKey]; ok {
// Convert the value to a string and return it
if strVal, isString := val.(string); isString {
return strVal, nil
}
logger.Error(nil, "SecretKey value is not a string")
return "", fmt.Errorf("SecretKey value is not a string")
}
logger.Error(nil, "SecretKey Not Found")
return "", fmt.Errorf("SecretKey Not Found")
}
return *result.SecretString, nil
}

Expand Down
4 changes: 2 additions & 2 deletions pkg/scaling/resolver/scale_resolvers.go
Original file line number Diff line number Diff line change
Expand Up @@ -333,10 +333,10 @@ func resolveAuthRef(ctx context.Context, client client.Client, logger logr.Logge
logger.Error(err, "error authenticating to Aws Secret Manager", "triggerAuthRef.Name", triggerAuthRef.Name)
} else {
for _, secret := range triggerAuthSpec.AwsSecretManager.Secrets {
res, err := awsSecretManagerHandler.Read(ctx, logger, secret.Name, secret.VersionID, secret.VersionStage)
res, err := awsSecretManagerHandler.Read(ctx, logger, secret.Name, secret.VersionID, secret.VersionStage, secret.SecretKey)
if err != nil {
logger.Error(err, "error trying to read secret from Aws Secret Manager", "triggerAuthRef.Name", triggerAuthRef.Name,
"secret.Name", secret.Name, "secret.Version", secret.VersionID, "secret.VersionStage", secret.VersionStage)
"secret.Name", secret.Name, "secret.Version", secret.VersionID, "secret.VersionStage", secret.VersionStage, "secret.SecretKey", secret.SecretKey)
} else {
result[secret.Parameter] = res
}
Expand Down
107 changes: 99 additions & 8 deletions tests/secret-providers/aws_secretmanager/aws_secretmanager_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ package aws_secret_manager_test
import (
"context"
"encoding/base64"
"encoding/json"
"fmt"
"os"
"testing"
Expand Down Expand Up @@ -151,6 +152,31 @@ spec:
name: {{.SecretManagerSecretName}}
`

triggerAuthenticationSecretKeyTemplate = `apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
name: {{.TriggerAuthenticationName}}
namespace: {{.TestNamespace}}
spec:
awsSecretManager:
credentials:
accessKey:
valueFrom:
secretKeyRef:
name: {{.AwsCredentialsSecretName}}
key: AWS_ACCESS_KEY_ID
accessSecretKey:
valueFrom:
secretKeyRef:
name: {{.AwsCredentialsSecretName}}
key: AWS_SECRET_ACCESS_KEY
region: {{.AwsRegion}}
secrets:
- parameter: connection
name: {{.SecretManagerSecretName}}
secretKey: connectionString
mpechner-akasa marked this conversation as resolved.
Show resolved Hide resolved
`

scaledObjectTemplate = `apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
Expand Down Expand Up @@ -257,11 +283,54 @@ spec:
)

func TestAwsSecretManager(t *testing.T) {
var useJSONSecretFormat = false
require.NotEmpty(t, awsAccessKeyID, "TF_AWS_ACCESS_KEY env variable is required for AWS Secret Manager test")
require.NotEmpty(t, awsSecretAccessKey, "TF_AWS_SECRET_KEY env variable is required for AWS Secret Manager test")

// Create the secret in AWS
err := createAWSSecret(t, useJSONSecretFormat)
assert.NoErrorf(t, err, "cannot create AWS Secret Manager secret - %s", err)

// Create kubernetes resources for PostgreSQL server
kc := GetKubernetesClient(t)
data, postgreSQLtemplates := getPostgreSQLTemplateData()

CreateKubernetesResources(t, kc, testNamespace, data, postgreSQLtemplates)

assert.True(t, WaitForStatefulsetReplicaReadyCount(t, kc, postgreSQLStatefulSetName, testNamespace, 1, 60, 3),
"replica count should be %d after 3 minutes", 1)

createTableSQL := "CREATE TABLE task_instance (id serial PRIMARY KEY,state VARCHAR(10));"
psqlCreateTableCmd := fmt.Sprintf("psql -U %s -d %s -c \"%s\"", postgreSQLUsername, postgreSQLDatabase, createTableSQL)

ok, out, errOut, err := WaitForSuccessfulExecCommandOnSpecificPod(t, postgresqlPodName, testNamespace, psqlCreateTableCmd, 60, 3)
assert.True(t, ok, "executing a command on PostreSQL Pod should work; Output: %s, ErrorOutput: %s, Error: %s", out, errOut, err)

// Create kubernetes resources for testing
data, templates := getTemplateData(useJSONSecretFormat)

KubectlApplyMultipleWithTemplate(t, data, templates)
assert.True(t, WaitForDeploymentReplicaReadyCount(t, kc, deploymentName, testNamespace, minReplicaCount, 60, 3),
"replica count should be %d after 3 minutes", minReplicaCount)

testScaleOut(t, kc, data)

// cleanup
KubectlDeleteMultipleWithTemplate(t, data, templates)
DeleteKubernetesResources(t, testNamespace, data, postgreSQLtemplates)

// Delete the secret in AWS
err = deleteAWSSecret(t)
assert.NoErrorf(t, err, "cannot delete AWS Secret Manager secret - %s", err)
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we create a function for the test and pass the required changes? 'TestAwsSecretManager' and 'TestAwsSecretManagerJSONFormat' are almost the same, maybe we can create a function like testAwsSecretManager with the changes as arguments (useJSONSecretFormat , the templates, etc)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you want testAwsSecretManager refactored? Or a new function?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My proposal is to create a new function that can be called with both configurations to reduce the amount of duplicated code, the name was tentative and just a suggestion

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok thanks.


func TestAwsSecretManagerJSONFormat(t *testing.T) {
var useJSONSecretFormat = true
require.NotEmpty(t, awsAccessKeyID, "TF_AWS_ACCESS_KEY env variable is required for AWS Secret Manager test")
require.NotEmpty(t, awsSecretAccessKey, "TF_AWS_SECRET_KEY env variable is required for AWS Secret Manager test")

// Create the secret in GCP
err := createAWSSecret(t)
// Create the secret in AWS
err := createAWSSecret(t, useJSONSecretFormat) // Create JSON formatted Secret
assert.NoErrorf(t, err, "cannot create AWS Secret Manager secret - %s", err)

// Create kubernetes resources for PostgreSQL server
Expand All @@ -280,7 +349,7 @@ func TestAwsSecretManager(t *testing.T) {
assert.True(t, ok, "executing a command on PostreSQL Pod should work; Output: %s, ErrorOutput: %s, Error: %s", out, errOut, err)

// Create kubernetes resources for testing
data, templates := getTemplateData()
data, templates := getTemplateData(useJSONSecretFormat)

KubectlApplyMultipleWithTemplate(t, data, templates)
assert.True(t, WaitForDeploymentReplicaReadyCount(t, kc, deploymentName, testNamespace, minReplicaCount, 60, 3),
Expand All @@ -292,7 +361,7 @@ func TestAwsSecretManager(t *testing.T) {
KubectlDeleteMultipleWithTemplate(t, data, templates)
DeleteKubernetesResources(t, testNamespace, data, postgreSQLtemplates)

// Delete the secret in GCP
// Delete the secret in AWS
err = deleteAWSSecret(t)
assert.NoErrorf(t, err, "cannot delete AWS Secret Manager secret - %s", err)
}
Expand Down Expand Up @@ -324,12 +393,19 @@ func getPostgreSQLTemplateData() (templateData, []Template) {
}
}

func getTemplateData() (templateData, []Template) {
func getTemplateData(useJSONFormat bool) (templateData, []Template) {
var triggerConfig string
if useJSONFormat {
triggerConfig = triggerAuthenticationSecretKeyTemplate
} else {
triggerConfig = triggerAuthenticationTemplate
}

return data, []Template{
{Name: "secretTemplate", Config: secretTemplate},
{Name: "awsCredentialsSecretTemplate", Config: awsCredentialsSecretTemplate},
{Name: "deploymentTemplate", Config: deploymentTemplate},
{Name: "triggerAuthenticationTemplate", Config: triggerAuthenticationTemplate},
{Name: "triggerAuthenticationTemplate", Config: triggerConfig},
{Name: "scaledObjectTemplate", Config: scaledObjectTemplate},
}
}
Expand All @@ -342,7 +418,7 @@ func testScaleOut(t *testing.T, kc *kubernetes.Clientset, data templateData) {
"replica count should be %d after 3 minutes", maxReplicaCount)
}

func createAWSSecret(t *testing.T) error {
func createAWSSecret(t *testing.T, useJSONFormat bool) error {
ctx := context.Background()

// Create AWS configuration
Expand All @@ -360,7 +436,22 @@ func createAWSSecret(t *testing.T) error {
client := secretsmanager.NewFromConfig(cfg)

// Create the secret value
secretString := postgreSQLConnectionString
var secretString string
if useJSONFormat {
secretObject := map[string]string{
"connectionString": postgreSQLConnectionString,
}
// Convert the map to a JSON string
jsonData, err := json.Marshal(secretObject)
if err != nil {
return fmt.Errorf("Error converting to JSON: %w", err)
}

// Print the JSON string
secretString = string(jsonData)
} else {
secretString = postgreSQLConnectionString
}
_, err = client.CreateSecret(ctx, &secretsmanager.CreateSecretInput{
Name: &secretManagerSecretName,
SecretString: &secretString,
Expand Down
Loading
Loading