Segmentation Fault Error in LidarCenterNet Import

[amanysh99]

When running train.py with python -m pdb, a segmentation fault error occurs upon importing LidarCenterNet from model.py. The error arises after successfully importing PyTorch and other dependencies. This suggests a potential issue within the LidarCenterNet module or its interactions with PyTorch/CUDA. Steps to reproduce:
Run python -m pdb train.py with specified arguments.
Observe segmentation fault error at LidarCenterNet import.
Environment:
PyTorch version: 1.13.1+cu117
CUDA version: 12.0 (V12.0.140)
OS: Ubuntu (WSL2) 5.15.167.4-microsoft-standard
Expected Behavior:
Successful execution of train.py without segmentation fault errors.
Actual Behavior:
Segmentation fault error upon importing LidarCenterNet.
The result of debugging and the error snip are attached.

Can anyone help?
Thanks in advance

[Kait0]

You probably have the wrong version of mmcv or torchscatter installed. Those two are also dependent of pytorch and cuda so if you change the versions of these two you need to change their versions as well.

I don't know if the code will work without changes if you upgrade the pytorch, mmcv, cuda versions etc.
If you don't actually need the newer versions I would recommend you use the same as use in our repo.

[amanysh99]

Thank you, I look forward to trying this out. @Kait0

[amanysh99]

I'm experiencing a compatibility issue between CUDA and PyTorch. My GPU supports CUDA 12.5 (screenshot 1: nvidia-smi output). Following the installation guide (screenshot 2: CUDA Toolkit 12.5.0 instructions), I successfully installed CUDA 12.5 on WSL (Ubuntu) and verified its availability using python -c "import torch; print(torch.cuda.is_available())".
However, when installing PyTorch, I discovered the latest compatible version only supports CUDA 12.4 (screenshot 3). Attempting to downgrade to CUDA 12.4 resulted in errors (screenshot 4).
Please advise on resolving this compatibility issue or provide alternative solutions.

[Kait0]

The support that is shown with nvidia-smi means support up to cuda 12.5. using older versions should be fine.
For most things you don't need a local installation of cuda, pytorch comes with cuda packaged. Torch-scatter might need it but you most likely won't use the lib you can also not install it and comment this line.

[amanysh99]

I followed your suggestions, but unfortunately, I'm still encountering issues:
ImportError: cannot import name 'force_fp32' from 'mmcv.runner' (unknown location)
PyTorch version is: 1.11.0+cu102
MMCV version is: 1.5.3
AssertionError: MMCV==1.5.3 is used but incompatible. Please install mmcv>=2.0.0rc4, <2.2.0.

[Kait0]

Never seen this issue before. You will need to debug / google.
MMCV is usually a pain to setup correctly which is why I removed this dependency in our followup work TransFuser++.

[amanysh99]

I'm currently debugging the train.py script and I encountered an error when running the command python train.py --batch_size 5 --logdir ./log --root_dir ./data --parallel_training 1. The error message indicates an issue with MMCV compatibility. Specifically, it seems that PyTorch 1.11.0+cu102 requires MMCV 2.x, but I have MMCV 1.5.3 installed. I'm investigating possible solutions, including downgrading PyTorch or upgrading MMCV."

[Kait0]

pytorch does not require mmcv. mmcv depends on pytorch.
perhaps you have something else like mmdetection installed that requires a different version of mmcv.

[amanysh99]