Auto thread detection in pbmm2 too high on slurm HPC

[mrvollger]

Operating system
NAME="Rocky Linux"
VERSION="8.9 (Green Obsidian)"

Package name
pbmm2 v1.13.0

issue
When I run pbmm2 I am finding that pbmm2 detects the total number of cores on the machine rather than what I have access to via my job on a slurm HPC. For example, here, pbmm2 detects 192 cores, but I only have 64, really:

$ pbmm2 align -m 16g --sort --unmapped denovo-asm/GM12878.v0.1.0.scaffold.fa  ../../k-mer-variant-phasing/results/GM12878/GM12878.haplotagged.bam  tmp.bam --log-level INFO
>|> 20240816 19:19:00.715 -|- INFO -|- AlignSettings -|- 0x7f68af197140|| -|- Using 184 threads for alignments, 8 threads for sorting, and 128G bytes RAM for sorting.

$ nproc
64

Thanks,
Mitchell

[armintoepfer]

Then it sounds like slurm won't properly allocate and restrict number of CPUs to your job. There are several ways to adjust the number of threads, for example using -j

[mrvollger]

Happy to use -j, but just in case I wasn't clear, the node can only use 64 cores, so when pbmm2 detects 192, its CPU utilization is still only going to those 64 cores. Mostly, I was pointing out that there is some detection difference between e.g. nproc and pbmm2 that allows nproc to detect slurm's limit of 64 in this example.

[armintoepfer]

I've created an internal ticket to look at this issue.

[armintoepfer]

We took a look at it. It's a problem that std::thread::hardware_concurrency() detects the number of virtual CPU cores of the physical CPU and not what has been dedicated to the job. I decided that we are not going to overfit our thread auto detection to different job dispatchers. Instead, we will train our support of this shortcoming.

We haven't had issue internally, because we always define the number of threads for each tool.

Thank you for bringing this to our attention, even though I'm not going to add special logic. I hope you understand.