Segmentation Fault - maybe because of setting globals inside async

[schlichtanders]

Hello,

I am trying to use Python with Julia's async tasks.

I've set

const pyglobals = get!(PythonCall.pydict, PythonCall.MODULE_GLOBALS, Main)

and just bring julia variables to python via pyglobals["variable"] = value. Maybe this is false?

Somehow when calling PythonCall inside Julia's async and redefining pyglobals inbetween leads to the following segmentation fault.

[24149] signal (11.1): Segmentation fault
in expression starting at none:1
_PyFrame_IsIncomplete at /usr/local/src/conda/python-3.12.0/Include/internal/pycore_frame.h:176 [inlined]
_PyFrame_GetFirstComplete at /usr/local/src/conda/python-3.12.0/Include/internal/pycore_frame.h:182 [inlined]
_PyThreadState_GetFrame at /usr/local/src/conda/python-3.12.0/Include/internal/pycore_frame.h:191 [inlined]
PyEval_GetGlobals at /usr/local/src/conda/python-3.12.0/Python/ceval.c:2355 [inlined]
PyImport_Import at /usr/local/src/conda/python-3.12.0/Python/import.c:2998
PyImport_Import at /home/ssahm/.julia/packages/PythonCall/wXfah/src/cpython/pointers.jl:299 [inlined]
macro expansion at /home/ssahm/.julia/packages/PythonCall/wXfah/src/Py.jl:131 [inlined]
pyimport at /home/ssahm/.julia/packages/PythonCall/wXfah/src/concrete/import.jl:11
deserialize_py at /home/ssahm/.julia/packages/PythonCall/wXfah/src/compat/serialization.jl:19
deserialize at /home/ssahm/.julia/packages/PythonCall/wXfah/src/compat/serialization.jl:28
unknown function (ip: 0x7fa2f13eff62)
_jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
handle_deserialize at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Serialization/src/Serialization.jl:880
deserialize at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Serialization/src/Serialization.jl:816
deserialize_expr at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Serialization/src/Serialization.jl:1277
handle_deserialize at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Serialization/src/Serialization.jl:896
deserialize at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Serialization/src/Serialization.jl:816
deserialize_expr at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Serialization/src/Serialization.jl:1277
handle_deserialize at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Serialization/src/Serialization.jl:896
deserialize at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Serialization/src/Serialization.jl:816
deserialize_expr at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Serialization/src/Serialization.jl:1277
handle_deserialize at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Serialization/src/Serialization.jl:896
deserialize at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Serialization/src/Serialization.jl:816
deserialize_expr at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Serialization/src/Serialization.jl:1277
handle_deserialize at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Serialization/src/Serialization.jl:896
deserialize at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Serialization/src/Serialization.jl:1499
unknown function (ip: 0x7fa2f1324406)
_jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
handle_deserialize at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Serialization/src/Serialization.jl:962
deserialize at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Serialization/src/Serialization.jl:816
deserialize_expr at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Serialization/src/Serialization.jl:1277
handle_deserialize at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Serialization/src/Serialization.jl:896
deserialize at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Serialization/src/Serialization.jl:816
deserialize_expr at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Serialization/src/Serialization.jl:1277
handle_deserialize at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Serialization/src/Serialization.jl:896
deserialize at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Serialization/src/Serialization.jl:816
#5 at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Serialization/src/Serialization.jl:975
ntupleany at ./ntuple.jl:43
deserialize_tuple at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Serialization/src/Serialization.jl:975
handle_deserialize at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Serialization/src/Serialization.jl:859
deserialize at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Serialization/src/Serialization.jl:816 [inlined]
deserialize_msg at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Distributed/src/messages.jl:87
unknown function (ip: 0x7fa2dcbb5eb2)
_jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
jl_f__call_latest at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/builtins.c:774
#invokelatest#2 at ./essentials.jl:819 [inlined]
invokelatest at ./essentials.jl:816 [inlined]
message_handler_loop at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:176
process_tcp_streams at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:133
#103 at ./task.jl:514
unknown function (ip: 0x7fa2f131f25f)
_jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
start_task at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/task.c:1092
Allocations: 33967797 (Pool: 33946601; Big: 21196); GC: 49

Any help is highly appreciated

[schlichtanders]

I think I found the root cause: It is a three-level nesting Julia - Python - Julia which breaks

Concretely, if

1. I am calling a python function from Julia, and
2. within the python function I am using julias sleep` for better task interactions, and
3. I am running another python code while the original function is in the julia sleep part

Then the above segmentation fault goes.

So I guess, one needs to make sure that python functions are always completely executed before a new python function is called.

Closing this, because this workaround sounds reasonable and works so far.