Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tumbling window aggregation :) #31

Merged
merged 23 commits into from
Dec 26, 2024
Merged

Conversation

turbolytics
Copy link
Owner

@turbolytics turbolytics commented Apr 3, 2024

Getting back up to speed after 6 months.

https://duckdb.org/docs/connect/concurrency.html

DuckDB uses optimistic concurrency control, an approach generally considered to be the best fit for read-intensive analytical database systems as it speeds up read query processing. As a result any transactions that modify the same rows at the same time will cause a transaction conflict error:

  • Initialize the tumbling window manager.
  • Setup the tumbling window manager to poll on a loop for closed windows.
  • Update sqlflow to remove conf dep and inject pipeline directly
  • Add more structure around config - every yaml property should be converted to a typed config object
  • Add invoke test cases. For windowed it should process message batch then flush the window.
  • Add Support for configurable out (including kafka)
  • Add tests to ensure windowing close logic - 0 flushes, partial flush and total flushes.
  • Add benchmark
  • Update config tables.sql.window -> tables.sql.manager and update window package -> handlers .Tumbling window

@turbolytics
Copy link
Owner Author

closes #27

@turbolytics turbolytics merged commit c1430d4 into main Dec 26, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant