Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build instructions for KBMOD in conda/virtual environments #449

Closed
wants to merge 1 commit into from

Conversation

stevenstetzler
Copy link
Member

I had trouble building KBMOD source on the UW Hyak/Klone GPU nodes into a conda environment. The problem was that CMake's FindPython was finding the wrong version of Python (even though the conda environment was activated and the correct Python installation was the first on the PATH etc.).

It was necessary to specify:

cmake -DPython3_FIND_VIRTUALENV=ONLY -B src/kbmod -S .

when installing.

@DinoBektesevic
Copy link
Member

DinoBektesevic commented Feb 2, 2024

The CMake find_package discovery system is well busted and doesn't work more often than it does. The use of that flag is by default set to FIRST - i.e. it should strongly prefer the venv (note this is a specific tool) Python rather than the system Python since version 3.15. Before that it basically doesn't even exist as a flag. It only started supporting conda since version 3.17 and the versions in between 3.15 and 3.17 have patchy support at best.

The current latest version is 3.28, but we officially pin 3.9..3.17 in CMakeLists. The limiting factors here are the availability of new versions of cmake on different machines - especially the shared compute resources that tend to be older and do not offer a newer version of cmake. The difference of behavior between whatever version of cmake is available at a particular machine make the package discovery sometimes work and sometimes not. The fixes are individually different for many of these situations (f.e. sometimes -DPython3_FIND_STRATEGY=LOCATION needs to be set instead, or, ultimately, the Python3_ROOT).

Same is with the GPU discovery mechanisms. It needs nvcc, but also the correctly matched driver versions. This is done by exporting the installation location into PATH and let the global discovery find them.

So, I'm not sure this is the solution to the problem. I'm not sure there is a single correct fix. It would be beneficial to eventually write a general "installation FAQ/Debug" in the documentation but that is atm far from getting updated so I'm happy to add the commentary to the README. Please clarify the following before merging:

Older versions of CMAKE (<3.17) sometimes do not correctly discover the Python installation in virtual environments, causing the installation to fail. In those cases, and if you have CMAKE version >3.15, [.... your sentences ...]

@stevenstetzler
Copy link
Member Author

I'm using CMake version 3.27.6, so the error didn't occur with an older version.

Maybe this best sits in an FAQ like you mention instead of the README like you mention.

@DinoBektesevic
Copy link
Member

It feels like this was said, but I just want to do another check, @stevenstetzler and others, is this PR set to be closed or merged?

@DinoBektesevic
Copy link
Member

Since there's been no answers here and it's obvious from other experiences that the way we build and test KBMOD in different environments needs to be improved I've opened an issue #503 and am closing this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants