PythonCall/juliacall doesn't work in conda environment

[nardi]

I first noticed this because I tried to import juliacall from a plain Python script, running in a conda environment. I then got this error:

> python -c "import juliacall"
fatal: error thrown and no exception handler available.
InitError(mod=:Sys, error=ErrorException("could not load library "libpcre2-8"
The specified module could not be found. "))
jl_errorf at /cygdrive/c/buildbot/worker/package_win64/build/src\rtutils.c:77
jl_load_dynamic_library at /cygdrive/c/buildbot/worker/package_win64/build/src\dlload.c:277
jl_get_library_ at /cygdrive/c/buildbot/worker/package_win64/build/src\runtime_ccall.cpp:52
jl_get_library_ at /cygdrive/c/buildbot/worker/package_win64/build/src\runtime_ccall.cpp:41 [inlined]
jl_load_and_lookup at /cygdrive/c/buildbot/worker/package_win64/build/src\runtime_ccall.cpp:63
jlplt_pcre2_compile_8_32718.clone_1 at c:\users\nardi\appdata\local\programs\julia-1.6.1\lib\julia\sys.dll (unknown line)
compile at .\pcre.jl:124
compile at .\regex.jl:79
match at .\regex.jl:297
match at .\regex.jl:297 [inlined]
match at .\regex.jl:316 [inlined]
splitdrive at .\path.jl:38
joinpath at .\path.jl:256
abspath at .\path.jl:410 [inlined]
__init_build at .\sysinfo.jl:125
__init__ at .\sysinfo.jl:117
jfptr___init___38133.clone_1 at c:\users\nardi\appdata\local\programs\julia-1.6.1\lib\julia\sys.dll (unknown line)
jl_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\julia.h:1703 [inlined]
jl_module_run_initializer at /cygdrive/c/buildbot/worker/package_win64/build/src\toplevel.c:72
_julia_init at /cygdrive/c/buildbot/worker/package_win64/build/src\init.c:794
jl_init_with_image__threading at /cygdrive/c/buildbot/worker/package_win64/build/src\jlapi.c:74 [inlined]
jl_init_with_image__threading at /cygdrive/c/buildbot/worker/package_win64/build/src\jlapi.c:63 [inlined]
jl_init__threading at /cygdrive/c/buildbot/worker/package_win64/build/src\jlapi.c:90
DllCanUnloadNow at C:\tools\miniconda3\envs\hgo_3.7\DLLs\_ctypes.pyd (unknown line)
DllCanUnloadNow at C:\tools\miniconda3\envs\hgo_3.7\DLLs\_ctypes.pyd (unknown line)
DllCanUnloadNow at C:\tools\miniconda3\envs\hgo_3.7\DLLs\_ctypes.pyd (unknown line)
DllCanUnloadNow at C:\tools\miniconda3\envs\hgo_3.7\DLLs\_ctypes.pyd (unknown line)
unknown function (ip: 00007ffaa94e6de9)
PyObject_FastCallKeywords at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyMethodDef_RawFastCallKeywords at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyEval_EvalFrameDefault at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyMethodDef_RawFastCallKeywords at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyEval_EvalFrameDefault at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyEval_EvalCodeWithName at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyImport_Import at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyImport_Import at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyMethodDef_RawFastCallDict at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PySlice_New at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyEval_EvalFrameDefault at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyEval_EvalCodeWithName at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyMethodDef_RawFastCallKeywords at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyEval_EvalFrameDefault at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyMethodDef_RawFastCallKeywords at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyEval_EvalFrameDefault at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyMethodDef_RawFastCallKeywords at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyEval_EvalFrameDefault at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyMethodDef_RawFastCallKeywords at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyEval_EvalFrameDefault at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyFunction_FastCallDict at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyObject_CallMethodIdObjArgs at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyObject_CallMethodIdObjArgs at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyMember_SetOne at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyImport_ImportModuleLevelObject at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyObject_LengthHint at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyEval_EvalFrameDefault at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyEval_EvalCodeWithName at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyEval_EvalCodeEx at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyEval_EvalCode at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyArena_Free at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyRun_StringFlags at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyRun_SimpleStringFlags at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
Py_Main at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
Py_Main at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyBytes_AsString at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
Py_Main at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
unknown function (ip: 00007ff6b69e1267)
BaseThreadInitThunk at C:\WINDOWS\System32\KERNEL32.DLL (unknown line)
RtlUserThreadStart at C:\WINDOWS\SYSTEM32\ntdll.dll (unknown line)

It looks like it can't load libpcre, which seems weird, because the conda environment should only add to the paths/environment variables, right?
On the other hand, using PythonCall from Julia was working fine, and it was using the Python executable from the environment as well. However, when I switch it to the conda environment explicitly using ENV["JULIA_PYTHONCALL_EXE"] = "CONDA:..." it also crashes:

julia> @pyv `1+1`
ERROR: UndefRefError: access to undefined reference
Stacktrace:
 [1] Py_CompileString(code::String, filename::String, mode::Int64)
   @ PythonCall.CPython C:\Users\Nardi\Programming\PythonCall.jl\src\cpython\fundamentals.jl:64
 [2] pyptr
   @ C:\Users\Nardi\Programming\PythonCall.jl\src\PyCode.jl:27 [inlined]
 [3] eval_impl(::Type{PyObject}, code::PyCode, globals::PyDict{String, PyObject}, locals::Nothing, extrakeys::Tuple{}, extravals::Tuple{})
   @ PythonCall C:\Users\Nardi\Programming\PythonCall.jl\src\eval.jl:3
 [4] top-level scope
   @ REPL[14]:1

I am running on Windows 10, and the conda environment has Python 3.7 installed. However, running julia when the environment is activated works fine, and regexes (so libpcre) also work. The only thing I can think of is that there is a mismatch between the libpcre version inside and outside the conda environment. But I'm not still not sure why that only shows up when setting up the Julia runtime from "inside" Python.

[nardi]

Oh, from the PythonCall side I think it already goes wrong during initialization (probably because the python interpreter crashes upon doing import juliacall):

julia> PythonCall.CONFIG
dlopenflags: 0x00000046
exepath: nothing
libpath: nothing
libptr: Ptr{Nothing} @0x0000000000000000
pyhome: nothing
pyhome_w: UInt16[]
pyprogname: nothing
pyprogname_w: UInt16[]
isstackless: false
isembedded: false
isinitialized: false
preinitialized: false
version: v"0.0.0"
isconda: true
condaenv: "hgo_3.7"
pyplotautoshow: true
qtfix: true
sysautolasttraceback: true
inputhookrunning: false
ipythonintegration: true
isipython: false

However there is no error/warning message when doing using PythonCall.

[nardi]

python -c "import subprocess; subprocess.run(['julia'])" also works fine.

[cjdoris]

Your initial problem looks like it's Julia that can't find libpcre, not Python.

So I guess the way juliacall is loading libjulia means it can't always link to all the libraries it needs.

I take it julia loads OK normally? I wonder if the paths to some dynamic libraries are embedded into the julia executable.

Can you locate julia and the libpcre it is using?

[cjdoris]

Hi I'm rewriting the package over on the 'rewrite' branch and came across the same problem. I fixed it there.

[timmysilv]

hi @cjdoris, I just hit this issue on Windows. I tried to call from juliacall import Main and this error came up. At first, I didn't have any Julia version installed. I manually installed a compatible version after that, but got the same issue. Ultimately, I followed this comment and it worked, but that's a rather tedious process. What was the fix on the rewrite branch? Is it merged in anywhere?