Ctrl-C interrupts not working inside python execution

[axsk]

Affects: PythonCall / JuliaCall

Describe the bug
Interrupting during python execution is not working properly (not even on one thread).
Keeping ctrl+c pressed interrupts the execution, but the next python call segfaults julia.
This applies to both, PythonCall and PyCall.

julia> using PyCall                                                                                                                                                                                          
                                                                                                                                                                                                             
julia> py"""                                                                                                                                                                                                 
       import time                                                                                                                                                                                           
       for i in range(100):                                                                                                                                                                                  
           time.sleep(1)"""                                                                                                                                                                                  
^C^C^C^C^C^C^C^C^C^C^CWARNING: Force throwing a SIGINT                                                                                                                                                       
^C^CERROR: InterruptException:                                                                                                                                                                               
Stacktrace:                                                                                                                                                                                                  
 [1] macro expansion                                                                                                                                                                                         
   @ ~/.julia/packages/PyCall/1gn3u/src/exception.jl:108 [inlined]                                                                                                                                           
 [2] #117                                                                                                                                                                                                    
   @ ~/.julia/packages/PyCall/1gn3u/src/pyeval.jl:38 [inlined]                                                                                                                                               
 [3] disable_sigint                                                                                                                                                                                          
   @ ./c.jl:473 [inlined]                                                                                                                                                                                    
 [4] pyeval_(s::String, globals::PyDict{String, PyObject, true}, locals::PyDict{String, PyObject, true}, input_type::Int64, fname::String)                                                                   
   @ PyCall ~/.julia/packages/PyCall/1gn3u/src/pyeval.jl:37                                                                                                                                                  
 [5] macro expansion                                                                                                                                                                                         
   @ ~/.julia/packages/PyCall/1gn3u/src/pyeval.jl:230 [inlined]                                                                                                                                              
 [6] top-level scope                                                                                                                                                                                         
   @ REPL[2]:1

julia> py"""
       import time
       for i in range(100):
           time.sleep(1)"""

[473012] signal (11.1): Segmentation fault
in expression starting at REPL[2]:1
_PyInterpreterState_GET at /usr/local/src/conda/python-3.10.14/Include/internal/pycore_pystate.h:117 [inlined]
PyUnicode_DecodeFSDefaultAndSize at /usr/local/src/conda/python-3.10.14/Objects/unicodeobject.c:4095
Py_CompileStringExFlags at /usr/local/src/conda/python-3.10.14/Python/pythonrun.c:1390
macro expansion at /home/htc/bzfsikor/.julia/packages/PyCall/1gn3u/src/exception.jl:108 [inlined]
pyeval_ at /home/htc/bzfsikor/.julia/packages/PyCall/1gn3u/src/pyeval.jl:34

julia> using PythonCall                                                                                                                                                                                                                                                                                                                                                                       
                                                                                                                                                                                                             
julia> @pyexec """                                                                                                                                                                                           
              import time                                                                                                                                                                
              for i in range(100):                                                                                                                                                           
                time.sleep(1)"""                                                                                                                                                                             
                                                                                                                                                                                                             
^C^C^C^C^CWARNING: Force throwing a SIGINT                                                                                                                                                                   
^CERROR: InterruptException:                                                                                                                                                                                 
Stacktrace:                                                                                                                                                                                                  
 [1] PyObject_CallObject                                                                                                                                                                                     
   @ ~/.julia/packages/PythonCall/S5MOg/src/C/pointers.jl:297 [inlined]                                                                                                                                      
 [2] macro expansion                                                                                                                                                                                         
   @ ~/.julia/packages/PythonCall/S5MOg/src/Core/Py.jl:132 [inlined]                                                                                                                                         
 [3] pycallargs(f::Py, args::Py)                                                                                                                                                                             
   @ PythonCall.Core ~/.julia/packages/PythonCall/S5MOg/src/Core/builtins.jl:212                                                                                                                             
 [4] pycall(::Py, ::Py, ::Vararg{Py}; kwargs::@Kwargs{})                                                                                                                                                     
   @ PythonCall.Core ~/.julia/packages/PythonCall/S5MOg/src/Core/builtins.jl:230                                                                                                                             
 [5] pycall                                                                                                                                                                                                  
   @ ~/.julia/packages/PythonCall/S5MOg/src/Core/builtins.jl:220 [inlined]                                                                                                                                   
 [6] Py                                                                                                                                                                                                      
   @ ~/.julia/packages/PythonCall/S5MOg/src/Core/Py.jl:339 [inlined]                                                                                                                                         
 [7] pyexec(::Type{Nothing}, code::Py, globals::Module, locals::Tuple{})                                                                                                                                     
   @ PythonCall.Core ~/.julia/packages/PythonCall/S5MOg/src/Core/builtins.jl:1212                                                                                                                            
 [8] top-level scope                                                                                                                                                                                         
   @ ~/.julia/packages/PythonCall/S5MOg/src/Core/builtins.jl:1260                                                                                                                                            
                                                                                                                                                                                                             
julia> @pyexec """                                                                                                                                                                                           
              import time                                                                                                                                                                                           
              for i in range(100):                                                                                                                                                                   
                time.sleep(1)"""                                                                                                                                                                             
                                                                                                                                                                                                             
[472692] signal (11.1): Segmentation fault                                                                                                                                                                   
in expression starting at REPL[2]:1                                                                                                                                                                          
_PyInterpreterState_GET at /usr/local/src/conda/python-3.12.4/Include/internal/pycore_pystate.h:133 [inlined]                                                                                                
maybe_freelist_pop at /usr/local/src/conda/python-3.12.4/Objects/tupleobject.c:1133 [inlined]                                                                                                                
tuple_alloc at /usr/local/src/conda/python-3.12.4/Objects/tupleobject.c:43 [inlined]                                                                                                                         
PyTuple_New at /usr/local/src/conda/python-3.12.4/Objects/tupleobject.c:74                                                                                                                                   
PyTuple_New at /home/htc/bzfsikor/.julia/packages/PythonCall/S5MOg/src/C/pointers.jl:297 [inlined]                                                                                                           
pynulltuple at /home/htc/bzfsikor/.julia/packages/PythonCall/S5MOg/src/Core/builtins.jl:824 [inlined]

Your system
Please provide detailed information about your system:

• Linux, Julia 1.10.4, both PyCall and PythonCall on newest versions.

[cjdoris]

It's true that Python code is not currently interruptible when called from PythonCall. That requires some signal handling magic that I think isn't currently possible in Python and Julia.

In your example you threw SIGINT by holding down Ctrl-C, I'm not surprised this would crash things as it's generally intended as a last resort to stop a program.

What happens if you just press Ctrl-C once and wait?

[kleinschmidt]

I can reliably get a segfault on one Ctrl-C, which looks like this:

[ Info: Received message: nothing
^C
[82624] signal (2): Interrupt: 2
in expression starting at /Users/dkleinschmidt/work/REDACTED/demo.jl:22
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
Allocations: 3128814 (Pool: 3125853; Big: 2961); GC: 4

[82624] signal (11.2): Segmentation fault: 11
in expression starting at /Users/dkleinschmidt/work/REDACTED/demo.jl:22
_PyUnicode_FromId at /Users/dkleinschmidt/.pyenv/versions/3.10.13/lib/libpython3.10.dylib (unknown line)
_PyImport_GetModuleId at /Users/dkleinschmidt/.pyenv/versions/3.10.13/lib/libpython3.10.dylib (unknown line)
wait_for_thread_shutdown at /Users/dkleinschmidt/.pyenv/versions/3.10.13/lib/libpython3.10.dylib (unknown line)
Py_FinalizeEx at /Users/dkleinschmidt/.pyenv/versions/3.10.13/lib/libpython3.10.dylib (unknown line)
Py_FinalizeEx at /Users/dkleinschmidt/.julia/packages/PythonCall/flx5V/src/C/pointers.jl:303 [inlined]
#37 at /Users/dkleinschmidt/.julia/packages/PythonCall/flx5V/src/C/context.jl:194
unknown function (ip: 0x139e3444f)
_jl_invoke at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/src/gf.c:0 [inlined]
ijl_apply_generic at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/src/gf.c:3077
_atexit at ./initdefs.jl:428
jfptr__atexit_79411.3 at /Users/dkleinschmidt/.julia/juliaup/julia-1.10.4+0.aarch64.apple.darwin14/lib/julia/sys.dylib (unknown line)
_jl_invoke at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/src/gf.c:0 [inlined]
ijl_apply_generic at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/src/gf.c:3077
jl_apply at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/src/./julia.h:1982 [inlined]
ijl_atexit_hook at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/src/init.c:280
jl_exit_thread0_cb at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-10/src/./signals-mach.c:464
Allocations: 3128814 (Pool: 3125853; Big: 2961); GC: 4

I can't share the exact code unfortunately, but there's no multi-threading happening in julia and the python code that is executing is a call to confluent_python.Consumer.poll() (which ultimately is a shim for a librdkafka C function call...). I do think that librdkafka is multithreaded though if that makes a difference...

This is with

• Julia 1.9.4 1.10.4
• PythonCall 0.9.22
• confluent_kafka 2.5.0
• Python 3.10.13
• MacOS 14.5 (23F79) (Macbook Air M1)