Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support topic-based encryption for Kafka buffer #3422

Closed
dlvenable opened this issue Oct 2, 2023 · 1 comment · Fixed by #3468
Closed

Support topic-based encryption for Kafka buffer #3422

dlvenable opened this issue Oct 2, 2023 · 1 comment · Fixed by #3468
Assignees
Labels
enhancement New feature or request plugin - buffer A plugin for buffering incoming data
Milestone

Comments

@dlvenable
Copy link
Member

dlvenable commented Oct 2, 2023

Is your feature request related to a problem? Please describe.

For some situations, we want to encrypt each Kafka topic with a different encryption key.

Describe the solution you'd like

Update Data Prepper's Kafka buffer to support an optional encryption key.

buffer:
  kafka:
    topics:
      - name: MyTopic
         encryption_key: gEa68HffrNhFtJkNoY0UsD6D6W4w8KUFtYJTnte+eiY=

Additionally, this key could be encrypted by Amazon KMS so that we can support envelope encryption.

buffer:
  kafka:
    topics:
      - name: MyTopic
        encryption_key: AQIDAHhBQ4iH7RP28kWDRU1yN2K73qYEE2d8i06EBly7HoDSIwFXoO+oiW+HOlam8lfIUFwLAAAAfjB8BgkqhkiG9w0BBwagbzBtAgEAMGgGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQM/j9Uf9cxYv/poV0FAgEQgDuVG9jfls3Ys7dR/cRKmdkcYDJw/XzR/ZEnZwcT9e+XB1T+SxC0YHLtc33lRwoD/UV0Ot+y8oUBqMvaXg==
        kms_key_id: alias/ExampleAlias

Data Prepper initialization

When Data Prepper starts, it reads the pipeline configuration file. If the user provided a KMS key in the configuration, the Kafka Buffer decrypts the data encryption key using KMS.

Data Prepper holds the decrypted data key in memory for future processing.

KMS-Initialization-SD

Receiving data and writing to Kafka

As users provider their data to a Data Prepper source, that source writes to the Kafka buffer. The Kafka Buffer encrypts each record using the decrypted data key; this is the same key decrypted during initialization. The Kafka Buffer sends the encrypted record to the Kafka topic.

KMS-ProcessingWrite-SD

Processing data and reading from Kafka

As Data Prepper runs, the Pipeline Worker reads from the Kafka Buffer. The Kafka Buffer polls the Kafka topic for data. Each Kafka ConsumerRecord is already encrypted as described above. Thus, the Kafka Buffer decrypts this data using the same data encryption key it loaded at initialization.

KMS-ProcessingRead-SD

Describe alternatives you've considered (Optional)**

N/A

Additional context

This builds on the work being done for #3322. The design could be extended for generic sink and source as well.

@dlvenable dlvenable added enhancement New feature or request plugin - buffer A plugin for buffering incoming data labels Oct 2, 2023
@dlvenable dlvenable added this to the v2.6 milestone Oct 6, 2023
@dlvenable dlvenable moved this from Unplanned to In progress in Data Prepper Tracking Board Oct 6, 2023
dlvenable added a commit to dlvenable/data-prepper that referenced this issue Oct 10, 2023
…es. Use a KMS key to decrypt the data encryption key, if one is provided. Resolves opensearch-project#3422

Signed-off-by: David Venable <[email protected]>
dlvenable added a commit to dlvenable/data-prepper that referenced this issue Oct 10, 2023
…es. Use a KMS key to decrypt the data encryption key, if one is provided. Resolves opensearch-project#3422

Signed-off-by: David Venable <[email protected]>
dlvenable added a commit to dlvenable/data-prepper that referenced this issue Oct 10, 2023
…es. Use a KMS key to decrypt the data encryption key, if one is provided. Resolves opensearch-project#3422

Signed-off-by: David Venable <[email protected]>
dlvenable added a commit that referenced this issue Oct 10, 2023
Encrypt and decrypt data in the Kafka buffer when the user configures. Use a KMS key to decrypt the data encryption key, if one is provided. Resolves #3422

Signed-off-by: David Venable <[email protected]>
@github-project-automation github-project-automation bot moved this from In progress to Done in Data Prepper Tracking Board Oct 10, 2023
@dlvenable dlvenable self-assigned this Oct 11, 2023
@dlvenable
Copy link
Member Author

I updated the description to show the actual format. As with the serialization format, the encryption is now configured on the topic.

@dlvenable dlvenable modified the milestones: v2.6, v2.7 Nov 1, 2023
@dlvenable dlvenable modified the milestones: v2.7, v2.6 Nov 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request plugin - buffer A plugin for buffering incoming data
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

1 participant