Part 4: read_table.c uses transform in ffi #614
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #614 +/- ##
==========================================
- Coverage 83.45% 83.24% -0.21%
==========================================
Files 74 74
Lines 16877 17080 +203
Branches 16877 17080 +203
==========================================
+ Hits 14084 14218 +134
- Misses 2135 2201 +66
- Partials 658 661 +3 ☔ View full report in Codecov by Sentry. |
github-actions
bot
added
the
breaking-change
scovich
reviewed
Dec 30, 2024
There was a problem hiding this comment.
What is the duckdb story with this new approach/PR? Have we explored that yet?
Asking because IIRC they push partition values down into their parquet reader, so they'll need to introspect the transforms and handle them differently than any kernel code we've written.
| if (transformed_res.tag != OkHandleExclusiveEngineData) { | ||
| print_error("Failed to transform read data.", (Error*)transformed_res.err); | ||
| free_error((Error*)transformed_res.err); | ||
| return NULL; |
There was a problem hiding this comment.
Don't we need to free data and evaluator regardless of whether this transform succeeded?
nicklan
added a commit
that referenced
this pull request
Jan 23, 2025
…and return it. (#607) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-incubator/delta-kernel-rs/blob/main/CONTRIBUTING.md 2. Run `cargo t --all-features --all-targets` to get started testing, and run `cargo fmt`. 3. Ensure you have added or run the appropriate tests for your PR. 4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 5. Be sure to keep the PR description updated to reflect all changes. --> ## What changes are proposed in this pull request? <!-- Please clarify what changes you are proposing and why the changes are needed. The purpose of this section is to outline the changes, why they are needed, and how this PR fixes the issue. If the reason for the change is already explained clearly in an issue, then it does not need to be restated here. 1. If you propose a new API or feature, clarify the use case for a new API or feature. 2. If you fix a bug, you can clarify why it is a bug. --> This is the initial part of moving to using expressions to express transformations when reading data. What this PR does is: - Compute a "static" transform, which is just a set of column expressions that need to be passed directly through without change, or enough metadata for lower levels to fill in a "fixup" expression - The static transform is passed into the iterator that parses each `Add` file - When parsing the `Add` file, if there are needed fix-ups (just partition columns today), the correct expression is created, and inserted into a row indexed map - This map is returned so the caller can find out for a given row what, if any, expression needs to be applied when reading the specified row Follow-up PRs: * #612: Propagate this information through when using `visit_scan_files` * #613: Actually use the data to do transformation and remove `transform_to_logical` entirely * #614: Make this work over ffi and use it * (TODO): Clean up any existing code that's now over complicated in the scan building Each of those are more invasive and end up touching significant code, so I'm staging this as much as possible to make reviews easier. <!-- Uncomment this section if there are any changes affecting public APIs: ### This PR affects the following public APIs If there are breaking changes, please ensure the `breaking-changes` label gets added by CI, and describe why the changes are needed. Note that _new_ public APIs are not considered breaking. --> ## How was this change tested? <!-- Please make sure to add test cases that check the changes thoroughly including negative and positive cases if possible. If it was tested in a way different from regular unit tests, please clarify how you tested, ideally via a reproducible test documented in the PR description. --> Unit tests, and inspection of resultant expressions when run on tables
nicklan
commented
Jan 25, 2025
| @@ -398,5 +429,5 @@ pub unsafe extern "C" fn visit_scan_data( | |||
| callback, | |||
| }; | |||
| // TODO: return ExternResult to caller instead of panicking? | |||
There was a problem hiding this comment.
reminder: get to this TODO!
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Stacked PR. Only review these commits
What changes are proposed in this pull request?
Use new transform functionality to transform data over FFI.
This lets us get rid of all the gross partition adding code in c :)
Still some things to work out here wrt. ffi types, so this is very much a draft.
But it works.
How was this change tested?