Support vectorized aggregation on Hypercore TAM #7655
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #7655 +/- ##
==========================================
+ Coverage 80.06% 81.33% +1.27%
==========================================
Files 190 242 +52
Lines 37181 44902 +7721
Branches 9450 11198 +1748
==========================================
+ Hits 29770 36523 +6753
- Misses 2997 3991 +994
+ Partials 4414 4388 -26 ☔ View full report in Codecov by Sentry. |
erimatnor
force-pushed
the
hypercore-vectoragg
branch
from
486e2b1 to
270fd54
Compare
akuzm
reviewed
Feb 4, 2025
erimatnor
force-pushed
the
hypercore-vectoragg
branch
from
270fd54 to
232df0a
Compare
It needs to know the value size.
erimatnor
force-pushed
the
hypercore-vectoragg
branch
7 times, most recently
from
fabdb44 to
efd1d0e
Compare
Add support for vectorized aggregation over Hypercore TAM. This includes some refactoring of the VectorAgg node in order to plan vectorized aggregation on top of ColumnarScans. Currently, only ColumnarScan can run below VectorAgg, because it is doing qual filtering. In theory, a SeqScan reading from Hypercore TAM should also work because it would produce Arrow slots. However, a SeqScan doesn't do vectorized filtering, which is currently assumed to be done before the VectorAgg node. In ColumnarScan, it necessary to turn off projection when VectorAgg is used. Otherwise, it would project the arrow slot into a virtual slot, thus losing the vector data. Ideally, a projection should never be planned to begin with, but this isn't possible since VectorAgg relies on replacing existing non-vectorized Agg plans added by PostgreSQL.
erimatnor
force-pushed
the
hypercore-vectoragg
branch
from
efd1d0e to
20de575
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add support for vectorized aggregation over Hypercore TAM. This includes some refactoring of the VectorAgg node in order to plan vectorized aggregation on top of ColumnarScans.
Currently, only ColumnarScan can run below VectorAgg, because it is doing qual filtering. In theory, a SeqScan reading from Hypercore TAM should also work because it would produce Arrow slots. However, a SeqScan doesn't do vectorized filtering, which is currently assumed to be done before the VectorAgg node.
In ColumnarScan, it necessary to turn off projection when VectorAgg is used. Otherwise, it would project the arrow slot into a virtual slot, thus losing the vector data. Ideally, a projection should never be planned to begin with, but this isn't possible since VectorAgg relies on replacing existing non-vectorized Agg plans added by PostgreSQL.
The existing vectoragg tests can be run with TAM enabled by default and in that case the tests produces the same vectoragg results as without TAM.
Closes: #7654
Disable-check: commit-count