-
Notifications
You must be signed in to change notification settings - Fork 569
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Tracking: Ad-hoc(batch) ingestion #18583
Comments
Hi, I will help with this issue, starting with TVFs. |
Have we reached consensus to support TVFs? To me, their use cases are duplicated with Sources, so they seem to be unnecessary. I’d like to see rationales and examples where they are more useful than sources before adding them |
|
Thanks for the explanation!
Makes me think whether also related with other
Compared with duckDB
|
The issue is not related to "shared" but it is beacuse the CDC source contains multiple tables' changes. Actually that is a "CONNECTION" |
Agree with that. cc @chenzl25. do we have plan to simplify the syntax of the TVF with connection? |
After the connection is supported, in my mind connection can be used in TVF directly like:
Connections contain the necessary information to allow TVF to query the external system. |
We will enhance the ad-hoc ingestion capability in subsequent releases, with the expectation that it will eventually be possible for users to read ad-hoc data if it is persisted on an external system.
Streaming storage
for the streaming storage, the predicate pushdown with the "offset" is required
lake
file source(object store)
Database
Currently we only support
Create table
with primary key on the CDC connector. To support it, we need design and introduce new syntax thatCREATE source
with CDC connector. In that case, the source can only be ad-hoc queried.The text was updated successfully, but these errors were encountered: