FEAT: AWS secret extension

[chenqi0805]

Description

This PR adds support for AWS secret extension plugin:

pipeline_extensions:
  aws:
    secrets:
      secret1:
        name: my-es-secret
        region: us-east-1
      secret2:
        name: my-es-secret-max-retries
        region: us-east-1
log-pipeline:
  workers: 1
  source:
    http:
      path: "/log/ingest"
  buffer:
    bounded_blocking:
      batch_size: ${{aws_secrets:secret1:batch_size}} # max number of records the buffer will drain for each read
  sink:
    - opensearch:
        hosts: [ "..." ]
        index: test-es7-compression
        username: ${{aws_secrets:secret1:username}}
        password: ${{aws_secrets:secret1:password}}
        flush_timeout: -1
        max_retries: ${{aws_secrets:secret2}}

The scope of settings that support secret reference is limited to plugins (source/buffer/sink/processor) in pipeline.YAML. The reference pattern is explained as follows:

• ${{aws_secrets:secret1:username}} means the secret value under secret1 will be parsed as json and the setting value is swapped with the one according to key username in the secret json
• ${{aws_secrets:secret2}} means the secret value under secret2 is parsed as a literal string and the setting value is swapped with the literal string value

Issues Resolved

Resolves #2780

Check List

• New functionality includes testing.
• New functionality has a documentation issue. Please link to it in this PR.
  • New functionality has javadoc added
• Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Unit Test Results

  1 448 files  +  16    1 448 suites  +16   46m 7s ⏱️ + 6m 17s
  6 034 tests +  67    6 033 ✔️ +  67  1 💤 ±0  0 ❌ ±0 
11 798 runs  +160  11 796 ✔️ +160  2 💤 ±0  0 ❌ ±0 

Results for commit 73a265d. ± Comparison against base commit b2b3249.

♻️ This comment has been updated with latest results.

[dlvenable]

This should return a generic type as it can vary.

[dlvenable]

We don't want to expose this in the aws-plugin-api. It could couple plugins with AWS SecretsManager which is not desirable.

[dlvenable]

Data Prepper core should not depend on aws-plugin. The extensions framework is what will let this load dynamically.

[chenqi0805]

Thanks for the catch! It is unnecessary

[dlvenable]

What can we do to this back to 100%?

[chenqi0805]

There is one JsonProcessingException unreachable:

@Override
    public Object retrieveValue(String secretId) {
        if (!secretIdToValue.containsKey(secretId)) {
            throw new IllegalArgumentException(String.format("Unable to find secretId: %s", secretId));
        }
        try {
            final Object secretValue = secretIdToValue.get(secretId);
            return secretValue instanceof Map ? OBJECT_MAPPER.writeValueAsString(secretIdToValue.get(secretId)) :
                    secretValue;
        } catch (JsonProcessingException e) {
            throw new IllegalArgumentException(String.format("Unable to read the value under secretId: %s as string",
                    secretId));
        }
    }

This is about writing value back into a string when ${{aws_secrets:secretId}} is used but the actual secret Value is a valid json, which means we interpret it as literal value

[dlvenable]

Can you mock secretIdToValue such that get(secretId) returns non-JSON?

[dlvenable]

AWS SecretsManager uses the term SecretId, so let's name this secret_id.

https://docs.aws.amazon.com/secretsmanager/latest/apireference/API_GetSecretValue.html#API_GetSecretValue_RequestSyntax

[dlvenable]

Let's use the existing AWSCredentialsProvider as provided by the AWS Plugin.

[chenqi0805]

#3350

[dlvenable]

Why are you throwing an AWS SDK exception here?

[chenqi0805]

I was trying to wrap it with a more explicit error message. Maybe RuntimeException is more appropriate?

[dlvenable]

Are these part of another PR? This seems unrelated.

[chenqi0805]

Casting would not work for secret value substitution especially when it is a String. But it is possible with generic type in PluginConfigValueTranslator this would not be necessary