[Spool] Introduce stage replacer and change send nodes to be able to send to more than one stage #14495
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
cc @bziobrowski |
8 tasks
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #14495 +/- ##
============================================
+ Coverage 61.75% 63.86% +2.11%
- Complexity 207 1570 +1363
============================================
Files 2436 2673 +237
Lines 133233 146738 +13505
Branches 20636 22499 +1863
============================================
+ Hits 82274 93710 +11436
- Misses 44911 46104 +1193
- Partials 6048 6924 +876
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚨 Try these New Features:
|
yashmayya
reviewed
Nov 20, 2024
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/plannode/MailboxSendNode.java
Outdated
Show resolved
Hide resolved
yashmayya
reviewed
Nov 20, 2024
| @@ -77,6 +78,8 @@ public abstract SortedSet<MailboxSendNode> getGroup(MailboxSendNode stage) | |||
| */ | |||
| public abstract SortedSet<SortedSet<MailboxSendNode>> getGroups(); | |||
|
|
|||
| public abstract Set<MailboxSendNode> getStages(); | |||
There was a problem hiding this comment.
Small doc comment might be useful here to distinguish this method from getLeaders / getGroups.
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/logical/PlanFragmenter.java
Outdated
Show resolved
Hide resolved
Comment on lines
+125
to
+127
| * The returned value of this method is ignored by default | ||
| */ | ||
| protected T preChildren(PlanNode node, C context) { |
There was a problem hiding this comment.
Why not make this a void function then? Doesn't seem like there are any future planned use cases for the return value here either?
...y-planner/src/main/java/org/apache/pinot/query/planner/logical/EquivalentStagesReplacer.java
Show resolved
Hide resolved
...-query-planner/src/main/java/org/apache/pinot/query/planner/plannode/MailboxReceiveNode.java
Outdated
Show resolved
Hide resolved
...y-planner/src/main/java/org/apache/pinot/query/planner/logical/EquivalentStagesReplacer.java
Outdated
Show resolved
Hide resolved
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/plannode/MailboxSendNode.java
Outdated
Show resolved
Hide resolved
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/plannode/MailboxSendNode.java
Show resolved
Hide resolved
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/plannode/MailboxSendNode.java
Show resolved
Hide resolved
yashmayya
approved these changes
Nov 21, 2024
...-query-planner/src/main/java/org/apache/pinot/query/planner/plannode/MailboxReceiveNode.java
Outdated
Show resolved
Hide resolved
...planner/src/test/java/org/apache/pinot/query/planner/logical/EquivalentStagesFinderTest.java
Outdated
Show resolved
Hide resolved
...planner/src/test/java/org/apache/pinot/query/planner/logical/EquivalentStagesFinderTest.java
Outdated
Show resolved
Hide resolved
...planner/src/test/java/org/apache/pinot/query/planner/logical/EquivalentStagesFinderTest.java
Outdated
Show resolved
Hide resolved
pinot-query-planner/src/test/java/org/apache/pinot/query/planner/logical/StagesTestBase.java
Outdated
Show resolved
Hide resolved
| } | ||
|
|
||
| public SimpleChildBuilder<MailboxReceiveNode> newReceiver( | ||
| Function<SimpleChildBuilder<MailboxReceiveNode>, SimpleChildBuilder<MailboxReceiveNode>> customize) { |
There was a problem hiding this comment.
What's the use case for this customization function?
pinot-query-planner/src/test/java/org/apache/pinot/query/planner/logical/StagesTestBase.java
Outdated
Show resolved
Hide resolved
...anner/src/test/java/org/apache/pinot/query/planner/logical/EquivalentStagesReplacerTest.java
Show resolved
Hide resolved
gortiz
commented
Nov 21, 2024
...planner/src/test/java/org/apache/pinot/query/planner/logical/EquivalentStagesFinderTest.java
Outdated
Show resolved
Hide resolved
rename some test methods Co-authored-by: Yash Mayya <[email protected]>
vrajat
reviewed
Nov 21, 2024
...-query-planner/src/main/java/org/apache/pinot/query/planner/plannode/MailboxReceiveNode.java
Show resolved
Hide resolved
davecromberge
pushed a commit
to davecromberge/pinot
that referenced
this pull request
Nov 22, 2024
…send to more than one stage (apache#14495)
Open
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is a continuation of #14296 and the next step on #14196.
Here we define how to modify a query to replace stages that have been found equivalent. Not all equivalent stages can be substitute. Specifically, two stages that send to the same stage cannot be substituted by a spool. The reason for that is that the mailbox service assumes there the tuple (receiveStage, sendStage) is unique in the query. Probably this can be changed, but it doesn't seem to be worth it in this initial phase.
This PR also changes the equivalence function to consider not equivalent two senders with different distribution. The design algorithm includes some ways to also support this, but again, it adds a complexity that doesn't seem to be required in the first spool version.
Several tests have been added and also a new builder API to create spools in tests. Although the spool builder code is a bit ugly, it simplifies the tests that use it. There are some examples in
EquivalentStagesReplacerTest.The next PR may be the final one, where the worker protocol and operators are changed to support these new senders that can send everywhere.