Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Investigate CoalescedHashPartitioning #9880

Open
jlowe opened this issue Nov 28, 2023 · 3 comments
Open

Investigate CoalescedHashPartitioning #9880

jlowe opened this issue Nov 28, 2023 · 3 comments
Assignees
@razajafri
Labels
audit_3.4.2 Audit related tasks for 3.4.2 audit_3.5.1 Audit related tasks for 3.5.1 audit_4.0.0 Audit related tasks for 4.0.0

Comments

@jlowe
Copy link
Contributor

jlowe commented Nov 28, 2023

apache/spark@b53c170679f added a new CoalescedHashPartitioning case class. We need to investigate whether the RAPIDS Accelerator needs to be updated to handle this new partitioning.
@jlowe jlowe added ? - Needs Triage Need team to review and classify audit_3.4.2 Audit related tasks for 3.4.2 audit_3.5.1 Audit related tasks for 3.5.1 audit_4.0.0 Audit related tasks for 4.0.0 labels Nov 28, 2023
@jlowe
Copy link
Contributor Author

jlowe commented Nov 28, 2023

See apache/spark@81639090622 for changes that were needed to the CPU BroadcastHashJoinExec that are probably relevant to the changes likely needed for the GPU version.

@mattahrens mattahrens removed the ? - Needs Triage Need team to review and classify label Dec 5, 2023
@razajafri
Copy link
Collaborator

related to #9753

@andygrove
Copy link
Contributor

See apache/spark@81639090622 for changes that were needed to the CPU BroadcastHashJoinExec that are probably relevant to the changes likely needed for the GPU version.

This commit updated the outputPartitioning logic in BroadcastHashJoinExec, but we do not currently implement this method in the GPU equivalent class. I filed #10609 to discuss whether we should also implement this, but as @revans2 pointed out, AQE will call the CPU version of this method anyway during replanning and before we replace these operators, so maybe we don't really need them.

@andygrove andygrove assigned razajafri and unassigned andygrove Apr 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@razajafri razajafri

Labels
audit_3.4.2 Audit related tasks for 3.4.2 audit_3.5.1 Audit related tasks for 3.5.1 audit_4.0.0 Audit related tasks for 4.0.0
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@andygrove @jlowe @mattahrens @razajafri