Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support dynamically grouping Events together in the S3 sink #4345

Closed
graytaylor0 opened this issue Mar 27, 2024 · 1 comment
Closed

Support dynamically grouping Events together in the S3 sink #4345

graytaylor0 opened this issue Mar 27, 2024 · 1 comment
Assignees
Labels
duplicate This issue or pull request already exists enhancement New feature or request

Comments

@graytaylor0
Copy link
Member

Is your feature request related to a problem? Please describe.
S3 Sink users would like to be able to group Events together and send them to dynamic partitions of their S3 bucket. Users would like to track different groups based on what is in the Events to group them into different S3 “folders”.

Describe the solution you'd like
Proposal with path_prefix and object_name

sink:
  - s3:
      object_key:
        path_prefix: "folder-${/partition_key}/"
        object_name: "my-object-${/some_key}";
      threshold:
        // We flush and remove groups when they reach 1000 events
        event_count: 1000
        // Alternatively, flush and remove groups on time when event_count isn't reached 
        event_collect_timeout: "30s"

When dynamic values are used in either the path_prefix or the object_name, we will group Events together based on the expression evaluation result.

Describe alternatives you've considered (Optional)
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

@dlvenable
Copy link
Member

@graytaylor0 , I believe this request is already expressed in #3310. Then we also have #4099 for the configurable object name. I'll close this one as a duplicate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
duplicate This issue or pull request already exists enhancement New feature or request
Projects
Development

No branches or pull requests

2 participants