-
Notifications
You must be signed in to change notification settings - Fork 202
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support topic-based encryption for Kafka buffer #3422
Labels
Milestone
Comments
dlvenable
added
enhancement
New feature or request
plugin - buffer
A plugin for buffering incoming data
labels
Oct 2, 2023
dlvenable
added a commit
to dlvenable/data-prepper
that referenced
this issue
Oct 10, 2023
…es. Use a KMS key to decrypt the data encryption key, if one is provided. Resolves opensearch-project#3422 Signed-off-by: David Venable <[email protected]>
dlvenable
added a commit
to dlvenable/data-prepper
that referenced
this issue
Oct 10, 2023
…es. Use a KMS key to decrypt the data encryption key, if one is provided. Resolves opensearch-project#3422 Signed-off-by: David Venable <[email protected]>
4 tasks
dlvenable
added a commit
to dlvenable/data-prepper
that referenced
this issue
Oct 10, 2023
…es. Use a KMS key to decrypt the data encryption key, if one is provided. Resolves opensearch-project#3422 Signed-off-by: David Venable <[email protected]>
dlvenable
added a commit
that referenced
this issue
Oct 10, 2023
Encrypt and decrypt data in the Kafka buffer when the user configures. Use a KMS key to decrypt the data encryption key, if one is provided. Resolves #3422 Signed-off-by: David Venable <[email protected]>
github-project-automation
bot
moved this from In progress
to Done
in Data Prepper Tracking Board
Oct 10, 2023
I updated the description to show the actual format. As with the serialization format, the encryption is now configured on the topic. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
Is your feature request related to a problem? Please describe.
For some situations, we want to encrypt each Kafka topic with a different encryption key.
Describe the solution you'd like
Update Data Prepper's Kafka buffer to support an optional encryption key.
Additionally, this key could be encrypted by Amazon KMS so that we can support envelope encryption.
Data Prepper initialization
When Data Prepper starts, it reads the pipeline configuration file. If the user provided a KMS key in the configuration, the Kafka Buffer decrypts the data encryption key using KMS.
Data Prepper holds the decrypted data key in memory for future processing.
Receiving data and writing to Kafka
As users provider their data to a Data Prepper source, that source writes to the Kafka buffer. The Kafka Buffer encrypts each record using the decrypted data key; this is the same key decrypted during initialization. The Kafka Buffer sends the encrypted record to the Kafka topic.
Processing data and reading from Kafka
As Data Prepper runs, the Pipeline Worker reads from the Kafka Buffer. The Kafka Buffer polls the Kafka topic for data. Each Kafka ConsumerRecord is already encrypted as described above. Thus, the Kafka Buffer decrypts this data using the same data encryption key it loaded at initialization.
Describe alternatives you've considered (Optional)**
N/A
Additional context
This builds on the work being done for #3322. The design could be extended for generic sink and source as well.
The text was updated successfully, but these errors were encountered: