Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Installing cudatoolkit should print a warning pointing to the new packages #94

Open
1 task done
pitrou opened this issue Jul 26, 2023 · 4 comments
Open
1 task done

Installing cudatoolkit should print a warning pointing to the new packages #94

pitrou opened this issue Jul 26, 2023 · 4 comments
Labels
bug Something isn't working

Comments

@pitrou
Copy link
Member

pitrou commented Jul 26, 2023

Solution to issue cannot be found in the documentation.

  • I checked the documentation.

Issue

Installing cudatoolkit does not bring with it nvcc or other important utilities. This makes it impossible to use with e.g. CMake's FindCUDAToolkit, which requires nvcc for finding even simple stuff such as the CUDA runtime or driver libraries.

Since, AFAIU, cudatoolkit is being phased one in favour of a new package structure (which does provide nvcc in a new separate package), it would be nice if installing cudatoolkit could print a clear warning message about this. This would be a great service for users, which are currently left in an information vacuum about the situation.

Installed packages

# packages in environment at /home/antoine/mambaforge/envs/pyarrow:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
_sysroot_linux-64_curr_repodata_hack 3                   h5bd9786_13    conda-forge
abseil-cpp                20220623.0           h8cdb687_6    conda-forge
adbc-driver-manager       0.1.0                    pypi_0    pypi
aiobotocore               2.5.0           py310h06a4308_0    defaults
aiohttp                   3.8.5           py310h2372a71_0    conda-forge
aioitertools              0.11.0             pyhd8ed1ab_0    conda-forge
aiosignal                 1.3.1              pyhd8ed1ab_0    conda-forge
alabaster                 0.7.12                   pypi_0    pypi
archery                   0.1.0                     dev_0    <develop>
asttokens                 2.2.1              pyhd8ed1ab_0    conda-forge
async-timeout             4.0.2              pyhd8ed1ab_0    conda-forge
attrs                     23.1.0             pyh71513ae_1    conda-forge
autopep8                  1.7.0                    pypi_0    pypi
aws-c-auth                0.7.0                hf8751d9_2    conda-forge
aws-c-cal                 0.6.0                h93469e0_0    conda-forge
aws-c-common              0.8.23               hd590300_0    conda-forge
aws-c-compression         0.2.17               h862ab75_1    conda-forge
aws-c-event-stream        0.3.1                h9599702_1    conda-forge
aws-c-http                0.7.11               hbe98c3e_0    conda-forge
aws-c-io                  0.13.28              h3870b5a_0    conda-forge
aws-c-mqtt                0.8.14               h2e270ba_2    conda-forge
aws-c-s3                  0.3.13               heb0bb06_2    conda-forge
aws-c-sdkutils            0.1.11               h862ab75_1    conda-forge
aws-checksums             0.1.16               h862ab75_1    conda-forge
aws-crt-cpp               0.20.3               he9c0e7f_4    conda-forge
aws-sdk-cpp               1.11.68              hbc2ea52_9    conda-forge
babel                     2.10.3                   pypi_0    pypi
backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
backports                 1.1                pyhd3eb1b0_0    defaults
backports.functools_lru_cache 1.6.5              pyhd8ed1ab_0    conda-forge
backports.zoneinfo        0.2.1           py310hff52083_7    conda-forge
beautifulsoup4            4.11.1                   pypi_0    pypi
benchmark                 1.8.2                h59595ed_0    conda-forge
binutils_impl_linux-64    2.40                 hf600244_0    conda-forge
blas                      1.1                    openblas    conda-forge
boost-cpp                 1.82.0               he19a28c_1    conda-forge
boto3                     1.26.76            pyhd8ed1ab_0    conda-forge
botocore                  1.29.76         py310h06a4308_0    defaults
breathe                   4.34.0                   pypi_0    pypi
brotli                    1.0.9                h166bdaf_9    conda-forge
brotli-bin                1.0.9                h166bdaf_9    conda-forge
brotlipy                  0.7.0           py310h5764c6d_1005    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.19.1               hd590300_0    conda-forge
ca-certificates           2023.7.22            hbcca054_0    conda-forge
certifi                   2023.7.22          pyhd8ed1ab_0    conda-forge
cffi                      1.15.1          py310h255011f_3    conda-forge
charset-normalizer        2.1.1                    pypi_0    pypi
clang                     15.0.7               ha770c72_2    conda-forge
clang-15                  15.0.7          default_h7634d5b_2    conda-forge
clangxx                   15.0.7          default_h1e1816b_2    conda-forge
click                     8.1.6           unix_pyh707e725_0    conda-forge
cloudpickle               2.2.1           py310h06a4308_0    defaults
cmake                     3.26.4               hcfe8598_0    conda-forge
cmake-format              0.6.13                   pypi_0    pypi
cmakelang                 0.6.13                   pypi_0    pypi
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
crc32c                    2.3                      pypi_0    pypi
cryptography              41.0.2          py310h75e40e8_0    conda-forge
cudatoolkit               11.8.0              h37601d7_11    conda-forge
cython                    3.0.0           py310hc6cd4ac_0    conda-forge
cython-lint               0.15.0                   pypi_0    pypi
decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
defusedxml                0.7.1                    pypi_0    pypi
deprecated                1.2.13                   pypi_0    pypi
docutils                  0.19                     pypi_0    pypi
doxygen                   1.9.7                h661eb56_1    conda-forge
exceptiongroup            1.1.2              pyhd8ed1ab_0    conda-forge
executing                 1.2.0              pyhd8ed1ab_0    conda-forge
expat                     2.5.0                hcb278e6_1    conda-forge
flake8                    5.0.4                    pypi_0    pypi
flask                     2.2.4                    pypi_0    pypi
flatbuffers               23.5.26              h59595ed_1    conda-forge
frozenlist                1.4.0           py310h2372a71_0    conda-forge
fsspec                    2023.6.0           pyh1a96a4e_0    conda-forge
gcc                       12.3.0               h8d2909c_1    conda-forge
gcc_impl_linux-64         12.3.0               he2b93b0_0    conda-forge
gdb                       12.1            py310hd73dadb_0    conda-forge
gflags                    2.2.2             he1b5a44_1004    conda-forge
gitdb                     4.0.9                    pypi_0    pypi
github3-py                3.2.0                    pypi_0    pypi
gitpython                 3.1.27                   pypi_0    pypi
glog                      0.6.0                h6f12383_0    conda-forge
gmock                     1.13.0               ha770c72_1    conda-forge
gmp                       6.2.1                h58526e2_0    conda-forge
google-cloud-cpp          2.7.0                h21dfe5b_1    conda-forge
googleapis-common-protos  1.59.0                   pypi_0    pypi
googleapis-storage-testbench 0.33.0                   pypi_0    pypi
grpc-cpp                  1.51.1               h27aab58_1    conda-forge
grpcio                    1.54.0                   pypi_0    pypi
gtest                     1.13.0               h00ab1b0_1    conda-forge
gunicorn                  20.1.0                   pypi_0    pypi
gxx                       12.3.0               h8d2909c_1    conda-forge
gxx_impl_linux-64         12.3.0               he2b93b0_0    conda-forge
hypothesis                6.82.0             pyha770c72_0    conda-forge
icu                       72.1                 hcb278e6_0    conda-forge
idna                      3.4                pyhd8ed1ab_0    conda-forge
imagesize                 1.4.1                    pypi_0    pypi
importlib-metadata        6.8.0              pyha770c72_0    conda-forge
iniconfig                 2.0.0              pyhd8ed1ab_0    conda-forge
ipython                   8.14.0             pyh41d4057_0    conda-forge
itsdangerous              2.1.2                    pypi_0    pypi
jedi                      0.18.2             pyhd8ed1ab_0    conda-forge
jinja2                    3.1.2                    pypi_0    pypi
jira                      3.4.1                    pypi_0    pypi
jmespath                  1.0.1              pyhd8ed1ab_0    conda-forge
julia                     0.5.7                    pypi_0    pypi
kernel-headers_linux-64   3.10.0              h4a8ded7_13    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
krb5                      1.21.1               h659d440_0    conda-forge
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
libabseil                 20220623.0      cxx17_h05df665_6    conda-forge
libblas                   3.9.0           17_linux64_openblas    conda-forge
libbrotlicommon           1.0.9                h166bdaf_9    conda-forge
libbrotlidec              1.0.9                h166bdaf_9    conda-forge
libbrotlienc              1.0.9                h166bdaf_9    conda-forge
libcblas                  3.9.0           17_linux64_openblas    conda-forge
libclang-cpp15            15.0.7          default_h7634d5b_2    conda-forge
libcrc32c                 1.1.2                h9c3ff4c_0    conda-forge
libcurl                   8.2.0                hca28451_0    conda-forge
libedit                   3.1.20221030         h5eee18b_0    defaults
libev                     4.33                 h516909a_1    conda-forge
libevent                  2.1.12               hf998b51_1    conda-forge
libexpat                  2.5.0                hcb278e6_1    conda-forge
libffi                    3.4.4                h6a678d5_0    defaults
libgcc-devel_linux-64     12.3.0               h8bca6fd_0    conda-forge
libgcc-ng                 13.1.0               he5830b7_0    conda-forge
libgfortran-ng            13.1.0               h69a702a_0    conda-forge
libgfortran5              13.1.0               h15d22d2_0    conda-forge
libgomp                   13.1.0               he5830b7_0    conda-forge
libgoogle-cloud           2.7.0                h21dfe5b_1    conda-forge
libgrpc                   1.51.1               h4fad500_1    conda-forge
libiconv                  1.17                 h166bdaf_0    conda-forge
liblapack                 3.9.0           17_linux64_openblas    conda-forge
libllvm11                 11.1.0               he0ac6c6_5    conda-forge
libllvm14                 14.0.6               hcd5def8_3    conda-forge
libllvm15                 15.0.7               h5cf9203_2    conda-forge
libnghttp2                1.52.0               h61bc06f_0    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libnuma                   2.0.16               h0b41bf4_1    conda-forge
libopenblas               0.3.23          pthreads_h80387f5_0    conda-forge
libprotobuf               3.21.12              h3eb15da_0    conda-forge
libsanitizer              12.3.0               h0f45ef3_0    conda-forge
libsqlite                 3.42.0               h2797004_0    conda-forge
libssh2                   1.11.0               h0841786_0    conda-forge
libstdcxx-devel_linux-64  12.3.0               h8bca6fd_0    conda-forge
libstdcxx-ng              13.1.0               hfd8a6a1_0    conda-forge
libthrift                 0.18.1               h8fd135c_2    conda-forge
libutf8proc               2.8.0                h166bdaf_0    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libuv                     1.44.2               h166bdaf_0    conda-forge
libxml2                   2.11.4               h0d562d8_0    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
llvmlite                  0.39.1                   pypi_0    pypi
lz4-c                     1.9.4                h6a678d5_0    defaults
make                      4.3                  hd18ef5c_1    conda-forge
markupsafe                2.1.1                    pypi_0    pypi
matplotlib-inline         0.1.6              pyhd8ed1ab_0    conda-forge
mccabe                    0.7.0                    pypi_0    pypi
multidict                 6.0.4           py310h1fa729e_0    conda-forge
ncurses                   6.4                  h6a678d5_0    defaults
ninja                     1.11.1               h924138e_0    conda-forge
nlohmann_json             3.11.2               h27087fc_0    conda-forge
numba                     0.57.1          py310h0f6aa51_0    conda-forge
numpy                     1.23.5          py310hac523dd_0    defaults
numpy-base                1.23.5          py310h375b286_0    defaults
numpydoc                  1.1.0                    pypi_0    pypi
oauthlib                  3.2.1                    pypi_0    pypi
openblas                  0.3.23          pthreads_h855a84d_0    conda-forge
openssl                   3.1.1                hd590300_1    conda-forge
orc                       1.9.0                h2f23424_1    conda-forge
packaging                 23.1               pyhd8ed1ab_0    conda-forge
pandas                    2.0.3           py310h7cbd5c2_1    conda-forge
parso                     0.8.3              pyhd8ed1ab_0    conda-forge
pexpect                   4.8.0              pyh1a96a4e_2    conda-forge
pickleshare               0.7.5                   py_1003    conda-forge
pip                       23.2.1             pyhd8ed1ab_0    conda-forge
pkg-config                0.29.2            h36c2ea0_1008    conda-forge
pluggy                    1.2.0              pyhd8ed1ab_0    conda-forge
prompt-toolkit            3.0.39             pyha770c72_0    conda-forge
prompt_toolkit            3.0.39               hd8ed1ab_0    conda-forge
protobuf                  4.22.3                   pypi_0    pypi
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
pure_eval                 0.2.2              pyhd8ed1ab_0    conda-forge
py                        1.11.0             pyh6c4a22f_0    conda-forge
py-mini-racer             0.6.0                    pypi_0    pypi
pyarrow                   13.0.0.dev535+gbe2014a9eb          pypi_0    pypi
pycodestyle               2.9.1                    pypi_0    pypi
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pydata-sphinx-theme       0.8.0                    pypi_0    pypi
pyflakes                  2.5.0                    pypi_0    pypi
pygit2                    1.10.1                   pypi_0    pypi
pygithub                  1.55                     pypi_0    pypi
pygments                  2.15.1          py310h06a4308_1    defaults
pyjwt                     2.5.0                    pypi_0    pypi
pynacl                    1.5.0                    pypi_0    pypi
pyopenssl                 23.2.0             pyhd8ed1ab_1    conda-forge
pyparsing                 3.1.0              pyhd8ed1ab_0    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
pytest                    7.4.0              pyhd8ed1ab_0    conda-forge
pytest-faulthandler       2.0.1                      py_0    conda-forge
pytest-lazy-fixture       0.6.3                      py_0    conda-forge
python                    3.10.12         hd12c33a_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-dotenv             0.21.0                   pypi_0    pypi
python-tzdata             2023.3             pyhd8ed1ab_0    conda-forge
python-xxhash             3.2.0           py310h1fa729e_0    conda-forge
python_abi                3.10                    3_cp310    conda-forge
pytz                      2023.3             pyhd8ed1ab_0    conda-forge
rapidjson                 1.1.0             he1b5a44_1002    conda-forge
rdma-core                 28.9                 h59595ed_1    conda-forge
re2                       2023.02.01           hcb278e6_0    conda-forge
readline                  8.2                  h8228510_1    conda-forge
requests                  2.28.1                   pypi_0    pypi
requests-oauthlib         1.3.1                    pypi_0    pypi
requests-toolbelt         1.0.0                    pypi_0    pypi
rfc3986                   2.0.0                    pypi_0    pypi
rfc3987                   1.3.8                    pypi_0    pypi
rhash                     1.4.3                h166bdaf_0    conda-forge
ruamel-yaml               0.17.21                  pypi_0    pypi
ruamel-yaml-clib          0.2.6                    pypi_0    pypi
s2n                       1.3.46               h06160fa_0    conda-forge
s3fs                      2023.6.0           pyhd8ed1ab_0    conda-forge
s3transfer                0.6.1              pyhd8ed1ab_0    conda-forge
scalpl                    0.4.2                    pypi_0    pypi
scipy                     1.9.2                    pypi_0    pypi
semver                    2.13.0                   pypi_0    pypi
setuptools                68.0.0             pyhd8ed1ab_0    conda-forge
setuptools-scm            7.1.0           py310h06a4308_0    defaults
setuptools_scm            7.1.0                hd3eb1b0_0    defaults
six                       1.16.0             pyh6c4a22f_0    conda-forge
smmap                     5.0.0                    pypi_0    pypi
snappy                    1.1.10               h9fff704_0    conda-forge
snowballstemmer           2.2.0                    pypi_0    pypi
sortedcontainers          2.4.0              pyhd8ed1ab_0    conda-forge
soupsieve                 2.3.2.post1              pypi_0    pypi
sparse                    0.13.0                   pypi_0    pypi
sphinx                    5.1.1                    pypi_0    pypi
sphinx-copybutton         0.5.1                    pypi_0    pypi
sphinx-design             0.3.0                    pypi_0    pypi
sphinxcontrib-applehelp   1.0.2                    pypi_0    pypi
sphinxcontrib-devhelp     1.0.2                    pypi_0    pypi
sphinxcontrib-htmlhelp    2.0.0                    pypi_0    pypi
sphinxcontrib-jsmath      1.0.1                    pypi_0    pypi
sphinxcontrib-qthelp      1.0.3                    pypi_0    pypi
sphinxcontrib-serializinghtml 1.1.5                    pypi_0    pypi
stack_data                0.6.2              pyhd8ed1ab_0    conda-forge
sysroot_linux-64          2.17                h4a8ded7_13    conda-forge
thrift-compiler           0.18.1               h8fd135c_2    conda-forge
thrift-cpp                0.18.1               ha770c72_2    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
tokenize-rt               5.1.0                    pypi_0    pypi
toml                      0.10.2                   pypi_0    pypi
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
traitlets                 5.9.0              pyhd8ed1ab_0    conda-forge
types-cryptography        3.3.23                   pypi_0    pypi
typing-extensions         4.7.1                hd8ed1ab_0    conda-forge
typing_extensions         4.7.1              pyha770c72_0    conda-forge
tzdata                    2023c                h04d1e81_0    defaults
ucx-proc                  1.0.0                       gpu    conda-forge
uri                       2.0.1                    pypi_0    pypi
uritemplate               4.1.1                    pypi_0    pypi
urllib3                   1.26.15         py310h06a4308_0    defaults
waitress                  2.1.2                    pypi_0    pypi
wcwidth                   0.2.6              pyhd8ed1ab_0    conda-forge
werkzeug                  2.2.2                    pypi_0    pypi
wheel                     0.41.0             pyhd8ed1ab_0    conda-forge
wrapt                     1.15.0          py310h1fa729e_0    conda-forge
xsimd                     11.1.0               h00ab1b0_0    conda-forge
xxhash                    0.8.1                h0b41bf4_0    conda-forge
xz                        5.4.2                h5eee18b_0    defaults
yarl                      1.9.2           py310h2372a71_0    conda-forge
zipp                      3.16.2             pyhd8ed1ab_0    conda-forge
zlib                      1.2.13               hd590300_5    conda-forge
zstd                      1.5.5                hc292b87_0    defaults

Environment info

mamba version : 1.4.9
     active environment : pyarrow
    active env location : /home/antoine/mambaforge/envs/pyarrow
            shell level : 1
       user config file : /home/antoine/.condarc
 populated config files : /home/antoine/mambaforge/.condarc
                          /home/antoine/.condarc
          conda version : 23.5.2
    conda-build version : not installed
         python version : 3.9.16.final.0
       virtual packages : __archspec=1=x86_64
                          __cuda=12.0=0
                          __glibc=2.35=0
                          __linux=5.15.0=0
                          __unix=0=0
       base environment : /home/antoine/mambaforge  (writable)
      conda av data dir : /home/antoine/mambaforge/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://conda.anaconda.org/nodefaults/linux-64
                          https://conda.anaconda.org/nodefaults/noarch
          package cache : /home/antoine/mambaforge/pkgs
                          /home/antoine/.conda/pkgs
       envs directories : /home/antoine/mambaforge/envs
                          /home/antoine/.conda/envs
               platform : linux-64
             user-agent : conda/23.5.2 requests/2.31.0 CPython/3.9.16 Linux/5.15.0-78-generic ubuntu/22.04.2 glibc/2.35
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False
@pitrou pitrou added the bug Something isn't working label Jul 26, 2023
@pitrou pitrou mentioned this issue Jul 26, 2023
Closed
49 tasks
@isuruf
Copy link
Member

isuruf commented Jul 27, 2023

This makes it impossible to use with e.g. CMake's FindCUDAToolkit, which requires nvcc for finding even simple stuff such as the CUDA runtime or driver libraries.

In the other issue, you said that this used to work, but doesn't anymore. Can you elaborate?

@pitrou
Copy link
Member Author

pitrou commented Jul 27, 2023

In the other issue, you said that this used to work, but doesn't anymore. Can you elaborate?

I might have misremembered, but I think I used to be able to build Arrow C++ with CUDA enabled a couple years ago using the conda-forge CUDA toolkit.

@isuruf
Copy link
Member

isuruf commented Jul 28, 2023

I might have misremembered, but I think I used to be able to build Arrow C++ with CUDA enabled a couple years ago using the conda-forge CUDA toolkit.

This is certainly possible as CMake has had multiple big changes to how CUDA is found. cuda-nvcc is a new package and there will be a corresponding cudatoolkit meta-package soon.

@leofang
Copy link
Member

leofang commented Dec 16, 2023

I am not sure how useful this is, given that the CUDA 12 rebuild has been performed in many feedstocks and that cudatoolkit is only for CUDA 11 and below. The requested message is equivalent of saying "don't use CUDA 11" on conda-forge. Maybe @conda-forge/cuda folks can share some thoughts?

@jayqi jayqi mentioned this issue Aug 9, 2024
Merged
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@pitrou @isuruf @leofang