Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support deleting s3 objects in S3 source when using SQS #3177

Open
asifsmohammed opened this issue Aug 16, 2023 · 0 comments
Open

Support deleting s3 objects in S3 source when using SQS #3177

asifsmohammed opened this issue Aug 16, 2023 · 0 comments
Labels
enhancement New feature or request

Comments

@asifsmohammed
Copy link
Collaborator

Is your feature request related to a problem? Please describe.
Data Prepper 2.4 supports deleting S3 objects when using S3 scan when delete_s3_objects_on_read is configured. SQS messages received by S3 source are processed but the underlying S3 objects are not deleted even after successful processing. Delete only works with end-to-end acknowledgements for S3 scan.

Describe the solution you'd like
Support deleting underlying S3 objects for SQS case in S3 source.

Additional context
The current delete functionality is performed by calling S3ObjectDeleteWorker from S3ObjectWorker. The delete logic can be moved to S3ObjectWorker.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
Data Prepper Tracking Board
Status: Unplanned
Milestone
No milestone
Development

No branches or pull requests
1 participant
@asifsmohammed