caught illegal operation: addreses 0x7f5438636bda, cause 'illegal operand'

[dstueckm]

I'm getting a cryptic error message when trying to run Scanorama through reticulate in R on an HPC cluster. I've been using the Seurat-compatible method suggested by @gdagstn on the open Seurat compatibility issue (#38 (comment)).

After running scanorama$integrate(assaylist, genelist), I get the error message

*** caught illegal operation ***
addreses 0x7f5438636bda, cause 'illegal operand'

I've tried generating the assaylist in a few different ways using a few different datasets, so I do not believe it is an issue with the data or a memory issue.

I'll include possibly relevant information below, happy to provide any further details
R version 3.6.1 (2019-07-05)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.2 LTS

Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1

reticulate v 1.16
Seurat v 3.1.4
Python3 v 3.7.2
scanorama v 1.6

[brianhie]

Thanks @dstueckm, at least in the Python version there seems to be something wrong with the latest versions of the annoy Python package causing segfaults or other illegal memory accesses (https://github.com/brianhie/scanorama#troubleshooting). You could experiment with different annoy versions (we tested the pure Python version with annoy version 1.11.5 on Ubuntu 18.04, which works).

Have you gotten Scanorama running at all (e.g., on toy data on a local machine) but it fails on the HPC? Or is the same configuration failing across environments?

[dstueckm]

Thanks for the reply! I tried uninstalling annoy and installing v 1.11.5 and still receive the same error message. I'll see if I can get it working locally and get back to you.

[brianhie]

Hi @dstueckm, just following up on this, did you ever manage to resolve the issue? Just wondering if there are any action items for me regarding this issue

[cvanderaa]

Hi @brianhie,

Sorry to leave a comment in a closed issue, but I experience the same error. I also use scanorama from within R using the reticulate package. I run into the following error:

*** caught illegal operation ***
address 0x7f37ece0aad0, cause 'illegal operand'

Traceback:
 1: py_call_impl(callable, dots$args, dots$keywords)
 2: scanorama$correct(datasets, genes_list, verbose = TRUE)
An irrecoverable exception occurred. R is aborting now ...
Illegal instruction (core dumped)

The error is the same as @dstueckm, except for a different address. Actually, the address changes at each of my attempts. Note that I'm working on my institution's HPC and this error only occurs when I'm sending the job to a cluster node. The exact same script and data work perfectly on an interactive node from the same server. In both cases, I use multiple cores.

I must admit I have very little experience with Python and I can't figure out whether the error is coming from the interface between reticulate and scanorama, from scanorama itself or from other modules and libraries called from within scanorama. Would you have an intuition about the origin of the error that could help me find the bug?

[brianhie]

Hi @cvanderaa, can you try one thing for me -- see if the error happens when you pass the approx=False flag to scanorama. If the error does not happen, it's probably a problem with the annoy library.

[cvanderaa]

Thanks a lot for pointing to this. Indeed, it works with approx=False, so I know where to look now. What version of annoy would you recommend. I currently have annoy-1.17 (installed last week using pip).

As a side note and FYI, I now also see several times this warning:

.local/share/r-miniconda/envs/r-reticulate/lib/python3.6/site-packages/sklearn/utils/validation.py:72: FutureWarning: Pass n_neighbors=20 as keyword args. From version 1.0 (renaming of 0.25) passing these as positional arguments will result in an error
  "will result in an error", FutureWarning)

[brianhie]

Great! If the algorithm is not too slow, then you can just consider using that for now. I think annoy fails on some computer architectures for unknown reasons.

[cvanderaa]

Ok thanks a lot for the info! The issue is solved for me.