Remove Arrow CUDA IPC code #10995

shwina · 2022-05-27T15:45:31Z

Closes #10994.

This PR removes the Arrow CUDA-IPC related code we have, which has two benefits:

It deletes code (I have confirmed that no one uses this code today)
It removes our dependency on Arrow CUDA, which contributes towards removing our shared lib dependency on libcuda.so

davidwendt · 2022-05-27T16:16:52Z

One more spot I think:

cudf/conda/recipes/libcudf/meta.yaml

Line 162 in 7c26691

- test -f $PREFIX/include/cudf/ipc.hpp

jakirkham · 2022-06-02T00:21:10Z

What is the plan with the Java code? For example

cudf/java/src/main/native/src/TableJni.cpp

Line 544 in 4e66281

class native_arrow_ipc_reader_handle final {

shwina · 2022-06-03T19:28:05Z

@jakirkham I don't believe that uses any of the libcudf code being deleted here. It also doesn't use anything Arrow CUDA (i.e., nothing from the arrow::cuda namespace), although it does use the non-GPU IPC bits from Arrow.

github-actions · 2022-07-03T20:03:02Z

This PR has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this PR if it is no longer required. Otherwise, please respond with a comment indicating any updates. This PR will be labeled inactive-90d if there is no activity in the next 60 days.

vyasr · 2022-07-12T00:28:36Z

@shwina does this also close #3661, or does that also require addressing outstanding Jitify issues still?

…remove-arrow-cuda-dependency

shwina · 2022-07-18T16:27:59Z

@vyasr - this will not close #3661. We will either need to remove jitify as a dependency, or resolve #10654.

vyasr · 2022-07-18T19:07:05Z

I think removing Jitify, while definitely feasible, will require a bit more work than I anticipated. It turns out that the rolling functionality in libcudf that uses Jitify is still used by our Python rolling code. Until that is transitioned to use numba like the rest of our apply functionality, the libcudf Jitify dependency remains (this is of course completely ignoring the question of whether we will have additional uses for Jitify in the future, which I'm sure that we will).

codecov · 2022-07-19T15:48:39Z

Codecov Report

Merging #10995 (16980b2) into branch-22.08 (5f9da83) will increase coverage by 0.05%.
The diff coverage is 73.33%.

❗ Current head 16980b2 differs from pull request most recent head 8d2e5c5. Consider uploading reports for the commit 8d2e5c5 to get more accurate results

@@               Coverage Diff                @@
##           branch-22.08   #10995      +/-   ##
================================================
+ Coverage         86.34%   86.39%   +0.05%     
================================================
  Files               144      143       -1     
  Lines             22811    22742      -69     
================================================
- Hits              19696    19648      -48     
+ Misses             3115     3094      -21

Impacted Files	Coverage Δ
python/cudf/cudf/_lib/__init__.py	`100.00% <ø> (ø)`
python/cudf/cudf/core/groupby/groupby.py	`91.02% <69.23%> (-0.42%)`	⬇️
python/cudf/cudf/core/column/numerical.py	`96.19% <100.00%> (+0.31%)`	⬆️
python/cudf/cudf/core/dataframe.py	`93.57% <0.00%> (+0.04%)`	⬆️
python/cudf/cudf/core/column/string.py	`88.80% <0.00%> (+0.12%)`	⬆️
python/cudf/cudf/core/tools/datetimes.py	`84.49% <0.00%> (+0.30%)`	⬆️
python/cudf/cudf/core/column/lists.py	`91.70% <0.00%> (+0.97%)`	⬆️
python/cudf/cudf/api/types.py	`90.42% <0.00%> (+1.06%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5f9da83...8d2e5c5. Read the comment docs.

robertmaynard

The cmake changes look correct to me

bdice

All the changes look fine to me.

bdice · 2022-07-19T18:34:15Z

@shwina Does this mean we no longer need to pin the CUDA build of Arrow in the cudf conda environment?

cudf/conda/environments/cudf_dev_cuda11.5.yml

Line 24 in b2dd1bf

- pyarrow=8.0.0=*cuda

conda/recipes/cudf/meta.yaml

Co-authored-by: Bradley Dice <[email protected]>

… into remove-arrow-cuda-dependency

ajschmidt8

Approving ops-codeowner changes. Also opened the accompanying PR below.

Remove arrow-cpp-proc integration#504

shwina · 2022-07-21T14:36:03Z

rerun tests

jakirkham · 2022-07-21T18:13:18Z

@gpucibot merge

jakirkham · 2022-07-21T18:15:00Z

Thanks Ashwin for the PR and everyone for the reviews! 🙏

… cudf recipe (#11412) Arrow 8.0.1 just got built and uploaded to conda-forge and there's an associated PR to update the pinning to 8.0.1. Currently the recipes pin to 8.0.0 explicitly which will cause solving issues. Instead of just changing to 8.0.1, this PR relaxes the pinning to allow minor and patch versions to get pulled as there hasn't been any nvcc compiler issues in a while and Arrow strives to not introduce API / ABI breakages in minor and patch versions. Additionally, the cudf recipe seems to be unnecessarily setting requiring a cuda build of pyarrow which should no longer be needed after #10995. Would really like this in for 21.08 if possible, but happy to change to just pin 8.0.1 if preferred. Authors: - Keith Kraus (https://github.com/kkraus14) Approvers: - AJ Schmidt (https://github.com/ajschmidt8) URL: #11412

Remove Arrow IPC code

2fd1990

github-actions bot added CMake CMake build issue Python Affects Python cuDF API. libcudf Affects libcudf (C++/CUDA) code. labels May 27, 2022

One more spot

0e7b386

github-actions bot added the conda label May 27, 2022

github-actions bot added the inactive-30d label Jul 3, 2022

github-actions bot removed the inactive-30d label Jul 12, 2022

shwina added 2 commits July 18, 2022 10:07

Merge branch 'branch-22.08' of https://github.com/rapidsai/cudf into …

b98ff99

…remove-arrow-cuda-dependency

Drop ArrowCUDA things

80c247b

shwina added breaking Breaking change improvement Improvement / enhancement to an existing function labels Jul 18, 2022

Stylle

5104b24

shwina added 2 commits July 18, 2022 17:12

gpuarrow.pyx

fec8942

Remove more gpuarrow things

617da20

shwina marked this pull request as ready for review July 19, 2022 15:57

shwina requested review from a team as code owners July 19, 2022 15:57

shwina requested a review from trxcllnt July 19, 2022 15:57

shwina requested review from mroeschke and bdice July 19, 2022 15:57

shwina changed the title ~~[DRAFT] Remove Arrow CUDA IPC code~~ Remove Arrow CUDA IPC code Jul 19, 2022

robertmaynard approved these changes Jul 19, 2022

View reviewed changes

mroeschke approved these changes Jul 19, 2022

View reviewed changes

bdice approved these changes Jul 19, 2022

View reviewed changes

Remove CUDA build pinnings

dcf0830

bdice reviewed Jul 19, 2022

View reviewed changes

conda/recipes/cudf/meta.yaml Outdated Show resolved Hide resolved

shwina and others added 3 commits July 19, 2022 16:35

Update conda/recipes/cudf/meta.yaml

31f2668

Co-authored-by: Bradley Dice <[email protected]>

Another ArrowCUDA thing

4cd19c6

Merge branch 'remove-arrow-cuda-dependency' of github.com:shwina/cudf…

8d2e5c5

… into remove-arrow-cuda-dependency

ajschmidt8 mentioned this pull request Jul 21, 2022

Remove arrow-cpp-proc rapidsai/integration#504

Merged

ajschmidt8 approved these changes Jul 21, 2022

View reviewed changes

jakirkham approved these changes Jul 21, 2022

View reviewed changes

rapids-bot bot merged commit 3ea0638 into rapidsai:branch-22.08 Jul 21, 2022

kkraus14 mentioned this pull request Aug 1, 2022

Relax arrow pinning to just 8.x and remove cuda build dependency from cudf recipe #11412

Merged

3 tasks

shwina mentioned this pull request Aug 12, 2022

[FEA] Interchange protocol between processes on the same device. #11514

Open

guilhermeleobas mentioned this pull request Feb 13, 2023

Update IPC to support new versions of arrow/cudf heavyai/heavyai#39

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove Arrow CUDA IPC code #10995

Remove Arrow CUDA IPC code #10995

shwina commented May 27, 2022 •

edited

Loading

davidwendt commented May 27, 2022

jakirkham commented Jun 2, 2022

shwina commented Jun 3, 2022

github-actions bot commented Jul 3, 2022

vyasr commented Jul 12, 2022

shwina commented Jul 18, 2022

vyasr commented Jul 18, 2022

codecov bot commented Jul 19, 2022 •

edited

Loading

robertmaynard left a comment

bdice left a comment

bdice commented Jul 19, 2022

ajschmidt8 left a comment

shwina commented Jul 21, 2022

jakirkham commented Jul 21, 2022

jakirkham commented Jul 21, 2022

Remove Arrow CUDA IPC code #10995

Remove Arrow CUDA IPC code #10995

Conversation

shwina commented May 27, 2022 • edited Loading

davidwendt commented May 27, 2022

jakirkham commented Jun 2, 2022

shwina commented Jun 3, 2022

github-actions bot commented Jul 3, 2022

vyasr commented Jul 12, 2022

shwina commented Jul 18, 2022

vyasr commented Jul 18, 2022

codecov bot commented Jul 19, 2022 • edited Loading

Codecov Report

robertmaynard left a comment

Choose a reason for hiding this comment

bdice left a comment

Choose a reason for hiding this comment

bdice commented Jul 19, 2022

ajschmidt8 left a comment

Choose a reason for hiding this comment

shwina commented Jul 21, 2022

jakirkham commented Jul 21, 2022

jakirkham commented Jul 21, 2022

shwina commented May 27, 2022 •

edited

Loading

codecov bot commented Jul 19, 2022 •

edited

Loading