python3Packages.lightgbm: add GPU support

[illustris]

Description of changes

Add GPU support for the lightgbm package

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 23.05 Release Notes (or backporting 22.11 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

[illustris]

flake for easy testing: https://github.com/illustris/lightgbm-gpu-minimal-test
Works on consumer GPUs:

[illustris@desktop:/dev/shm]$ nix run github:illustris/lightgbm-gpu-minimal-test
/nix/store/cd9xrgli17m69888dn1i8i30h1vrbw7m-python3-3.10.10-env/lib/python3.10/site-packages/lightgbm/engine.py:177: UserWarning: Found `num_iterations` in params. Will use it instead of argument
  _log_warning(f"Found `{alias}` in params. Will use it instead of argument")
[LightGBM] [Info] This is the GPU trainer!!
[LightGBM] [Info] Total Bins 36
[LightGBM] [Info] Number of data points in the train set: 50, number of used features: 2
[LightGBM] [Info] Using GPU Device: NVIDIA GeForce RTX 2080 Ti, Vendor: NVIDIA Corporation
[LightGBM] [Info] Compiling OpenCL Kernel with 64 bins...
[LightGBM] [Info] GPU programs have been built
[LightGBM] [Info] Size of histogram bin entry: 8
[LightGBM] [Info] 2 dense feature groups (0.00 MB) transferred to GPU in 0.037596 secs. 0 sparse feature groups
[LightGBM] [Info] Start training from score 0.600000
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
True

It did not work on an enterprise vGPU (A100-10C):

# nix run github:illustris/lightgbm-gpu-minimal-test
/nix/store/cd9xrgli17m69888dn1i8i30h1vrbw7m-python3-3.10.10-env/lib/python3.10/site-packages/lightgbm/engine.py:177: UserWarning: Found
`num_iterations` in params. Will use it instead of argument
  _log_warning(f"Found `{alias}` in params. Will use it instead of argument")
[LightGBM] [Info] This is the GPU trainer!!
[LightGBM] [Info] Total Bins 36
[LightGBM] [Info] Number of data points in the train set: 50, number of used features: 2
False

Further testing needed.

---

EDIT: Nevermind, hardware.opengl.enable=true needs to be set for /run/opengl-driver/etc/OpenCL/vendors to exist. Adding that to the server lets ocl-icd detect the GPU.

# nix run github:illustris/lightgbm-gpu-minimal-test
/nix/store/cd9xrgli17m69888dn1i8i30h1vrbw7m-python3-3.10.10-env/lib/python3.10/site-packages/lightgbm/engine.py:177: UserWarning: Found `num_iterations` in params. Will use it instead of argument
  _log_warning(f"Found `{alias}` in params. Will use it instead of argument")
[LightGBM] [Info] This is the GPU trainer!!
[LightGBM] [Info] Total Bins 36
[LightGBM] [Info] Number of data points in the train set: 50, number of used features: 2
[LightGBM] [Info] Using GPU Device: GRID A100-10C, Vendor: NVIDIA Corporation
[LightGBM] [Info] Compiling OpenCL Kernel with 64 bins...
[LightGBM] [Info] GPU programs have been built
[LightGBM] [Info] Size of histogram bin entry: 8
[LightGBM] [Info] 2 dense feature groups (0.00 MB) transferred to GPU in 0.001206 secs. 0 sparse feature groups
[LightGBM] [Info] Start training from score 0.560000
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
True

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/prs-ready-for-review/3032/1971

[SuperSandro2000]

@ofborg build python310Packages.lightgbm