sporadic segfaults running lima

[jamestwebber]

Operating system
Linux, Debian 10 on a GCP VM

Package name

lima 2.6.0

Using:
  lima      : 2.6.0 (commit v2.6.0)
  pbbam     : 2.1.0 (commit v2.1.0)
  pbcopper  : 2.0.0 (commit v2.0.0-59-g580e770)
  boost     : 1.77
  htslib    : 1.15
  zlib      : 1.2.11

Conda environment

conda list output

# packages in environment at /opt/conda/envs/pacbio:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.18.1               h7f98852_0    conda-forge
ca-certificates           2022.12.7            ha878542_0    conda-forge
htslib                    1.16                 h6bc39ce_0    bioconda
isoseq3                   3.8.2                h9ee0642_0    bioconda
keyutils                  1.6.1                h166bdaf_0    conda-forge
krb5                      1.20.1               hf9c8cef_0    conda-forge
ld_impl_linux-64          2.39                 hcc3a1bd_1    conda-forge
libcurl                   7.87.0               h6312ad2_0    conda-forge
libdeflate                1.13                 h166bdaf_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 12.2.0              h65d4601_19    conda-forge
libgomp                   12.2.0              h65d4601_19    conda-forge
libnghttp2                1.51.0               hdcd2b5c_0    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libsqlite                 3.40.0               h753d276_0    conda-forge
libssh2                   1.10.0               haa6b8db_3    conda-forge
libstdcxx-ng              12.2.0              h46fd767_19    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libzlib                   1.2.13               h166bdaf_4    conda-forge
lima                      2.6.0                h9ee0642_0    bioconda
ncurses                   6.3                  h27087fc_1    conda-forge
openssl                   1.1.1s               h0b41bf4_1    conda-forge
palette                   0.2                      pypi_0    pypi
pbbam                     2.1.0                h218f6fc_3    bioconda
pbcopper                  2.0.0                ha04c180_1    bioconda
pbmm2                     1.9.0                h9ee0642_0    bioconda
pbpigeon                  1.0.0                hdfd78af_0    bioconda
pbskera                   0.1.0                hdfd78af_0    bioconda
pbtk                      1.0.0                h9ee0642_0    bioconda
pip                       22.3.1             pyhd8ed1ab_0    conda-forge
pysam                     0.19.1          py310hff46b53_1    bioconda
python                    3.10.6          h582c2e5_0_cpython    conda-forge
python_abi                3.10                    3_cp310    conda-forge
readline                  8.1.2                h0f457ee_0    conda-forge
samtools                  1.16.1               h6899075_1    bioconda
setuptools                65.6.3             pyhd8ed1ab_0    conda-forge
stringtie                 2.2.1                ha04fe3b_3    bioconda
tk                        8.6.12               h27826a3_0    conda-forge
tqdm                      4.64.1                   pypi_0    pypi
tzdata                    2022g                h191b570_0    conda-forge
wheel                     0.38.4             pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
zlib                      1.2.13               h166bdaf_4    conda-forge

Describe the bug
I am getting a segfault when running lima to trim 10x adapters for a MAS-iso-seq dataset:

lima --isoseq -j 2 input.bam 10x_3kit_primers.fasta output.lima.bam

At first I thought this was due to a specific read, but it seems to be sporadic--on a small file (~10k reads) it will only fail 10% of the time or so, but on a larger file (50k) it consistently crashes and on the full size file (millions of reads) it always happens.

My best guess is some kind of multithreading issue and it happens with low frequency. I don't know why this particular dataset is having this issue when others did not.

Error message
It's just Segmentation fault

To Reproduce
I should be able to share a small BAM file with you but will need some way to transfer it.

Expected behavior
I just want to trim adapters 😢

[armintoepfer]

Thank you for reporting!

Please upload data to reproduce and I'll fix it asap https://github.com/PacificBiosciences/pbbioconda#file-sharing

[jamestwebber]

Okay I've uploaded a BAM file with ~50k reads. I fails sometimes but not all the time–maybe 1/2 or 1/3 of runs. A larger file would fail more often.

[armintoepfer]

Are you using the primers from https://isoseq.how/umi/examples.html? I would assume, but not sure ;)

Reads input                    : 49993
Reads above all thresholds (A) : 49944
Reads below any threshold  (B) : 49

[jamestwebber]

Yes should be the 10x 3' kit primers. And it does run successfully sometimes...I found that if I ran lima 2-3 times I'd usually get a segfault on one. On the full data it seemed unavoidable.

[armintoepfer]

First step, I'm able to reproduce with the released version:

$ which lima
/pbi/flash/atoepfer/miniconda3-mamba/bin/lima
$ lima --version
lima 2.6.0

Using:
  lima      : 2.6.0 (commit v2.6.0)
  pbbam     : 2.1.0 (commit v2.1.0)
  pbcopper  : 2.0.0 (commit v2.0.0-59-g580e770)
  boost     : 1.77
  htslib    : 1.15
  zlib      : 1.2.11
$ md5sum $(which lima)
4e9aa50df821a2c72212949e5883bb98  /pbi/flash/atoepfer/miniconda3-mamba/bin/lima
$ for i in {1..100}; do echo $i; lima ~/pdac.sub.bam primers.fasta out.bam --isoseq -j 2; done
1
2
3
4
5
Segmentation fault (core dumped)
6
7
8
9
10
11
12
13
Segmentation fault (core dumped)
14
15
16
17
18
...

Good news, with our latest development version, no more hiccups. We've found the same bug internally, but haven't updated conda yet. And even better for you, I'm going to release it tomorrow if everything goes according to my plan :)

[jamestwebber]

Great, at least I know I'm not crazy 😂

[armintoepfer]

Latest lima v2.7.1 is up, please try

[jamestwebber]

Seems like it works now, thanks!

[armintoepfer]

Great. Please let us know when you find more problems.

[nathanweeks]

lima 2.9.0 consistently segfaults for me on Rocky Linux 8.7 / 8.8, using Linux kernel 4.18.0-425.10.1.el8_7.x86_64 & 4.18.0-477.27.1.el8_8.cloud.x86_64, respectively (the latter being a Google Cloud Platform VM instance).

e.g., on a new GCP VM instance running Rocky 8.8:

$ sudo curl -o /usr/local/bin/micromamba -L https://github.com/mamba-org/micromamba-releases/releases/download/1.5.3-0/micromamba-linux-64
$ sudo chmod +x /usr/local/bin/micromamba
$ micromamba create -c conda-forge -c bioconda -n lima lima
...
$ micromamba list -n lima
warning  libmamba 'root_prefix' set with default value: /home/user/micromamba
List of packages in environment: "/home/user/micromamba/envs/lima"

  Name  Version  Build       Channel 
───────────────────────────────────────
  lima  2.9.0    h9ee0642_0  bioconda
$ micromamba run -n lima lima
warning  libmamba 'root_prefix' set with default value: /home/user/micromamba
$ echo $?
139
$ dmesg -T | tail -n 1
[Mon Nov 20 19:54:48 2023] 5110 (lima): Uhuuh, elf segment at 0000000000406000 requested but the memory is mapped already

The error message logged at the end has a few online references that indicate specific kernel versions + build toolchain used may be contributing factors (e.g., https://stackoverflow.com/a/51661122, Homebrew/homebrew-core#136432)

The segfault can be observed when using a Debian 12 container (specifically, biocontainers image: https://quay.io/repository/biocontainers/lima) on Rocky 8.7 host OS; however, the issue does not impact Rocky Linux 9.2, nor does it impact a Rocky 8.8 container running on the default kernel available in a GitHub Codespace (6.2.0-1016-azure) --- so the kernel version seems to be important.

[armintoepfer]

So you are telling me that it aborts even with empty input? This is on a x86_64 system, correct?

[jamestwebber]

lima 2.9.0 works for me although I don't know what the changes are.

[nathanweeks]

So you are telling me that it aborts even with empty input? This is on a x86_64 system, correct?

That's correct.

[nathanweeks]

Update: it must have something to do with the lima conda package. The lima executables from the tarball downloaded from the GitHub release work OK (so far); e.g., on RHEL 8.9 (where the conda-packaged version of lima also segfaults):

$ uname -r
4.18.0-513.5.1.el8_9.x86_64
$ cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.9 (Ootpa)
$ curl -L https://github.com/PacificBiosciences/barcoding/releases/download/v2.9.0/lima.tar.gz | tar -xzf -
...
$ ./lima --version
lima 2.9.0

Using:
  lima      : 2.9.0 (commit v2.9.0)
  pbbam     : 2.5.0 (commit v2.5.0)
  pbcopper  : 2.4.0 (commit v2.4.0)
  boost     : 1.81
  htslib    : 1.17
  zlib      : 1.2.13```

[armintoepfer]

I'm getting more and more annoyed at conda changing our binaries...

[nathanweeks]

I'm getting more and more annoyed at conda changing our binaries...

At least an RPATH has been added (by patchelf) to the conda-packged lima executable:

$ readelf -d ./lima

Dynamic section at offset 0x4e9aa8 contains 27 entries:
  Tag        Type                         Name/Value
 0x0000000000000003 (PLTGOT)             0x8eafe8
...
$ readelf -d micromamba/envs/lima/bin/lima

Dynamic section at offset 0x6e28 contains 28 entries:
  Tag        Type                         Name/Value
 0x000000000000000f (RPATH)              Library rpath: [$ORIGIN/../lib]
 0x0000000000000003 (PLTGOT)             0x8eafe8
...

Squinting at the conda docs & source, I'm think it may be possible to disable this by setting binary_relocation: False in the meta.yml?

https://docs.conda.io/projects/conda-build/en/latest/resources/define-metadata.html#binary-relocation

[armintoepfer]

Let me test waters with a different distribution channel idea. What if I'd provide you with a docker image that contains all PacBio secondary application tools? The binaries in there would be still statically compiled, but I'd distribute the docker image via a registry.

[nathanweeks]

Let me test waters with a different distribution channel idea. What if I'd provide you with a docker image that contains all PacBio secondary application tools? The binaries in there would be still statically compiled, but I'd distribute the docker image via a registry.

A container image with the latest versions (and/or an "officially-supported" container image with the SMRT Tools versions of each application) would be convenient. It might not obviate the need to fix the bioconda packages, unless they're to be deprecated.

[nathanweeks]

Squinting at the conda docs & source, I'm think it may be possible to disable this by setting binary_relocation: False in the meta.yml?

https://docs.conda.io/projects/conda-build/en/latest/resources/define-metadata.html#binary-relocation

Done in the bioconda-recipes PR linked to above. The updated build 1 version of the lima 2.9.0 bioconda package seems to fix the immediate-segfault issue on EL8.