[Bug] Cython deadlock on upon being called back with non-callee thread

[junrushao]

No description provided.

[tqchen]

Was due to the fact that non-python started thread needs to call an explicit state ensure function. Perhaps we should register such functions e.g. runtime.ThreadInit as PackedFunc from cython side

https://docs.python.org/3/c-api/init.html#non-python-created-threads

[junrushao]

@tqchen this is interesting finding! Just curious, is it required by only Cython, or both ctypes and cython?

[tqchen]

I am not too sure, ctypes's mechanism might already supported this initialization

[junrushao]

Related issue: apache/tvm#7919

Hmmm...okay, it is not so related...

[tqchen]

It is related since we can introduce EnvThreadInitialize in the same registry as a followup

[junrushao]

Looks like we have to somehow link to Python to make sure the following snippet could run:

PyGILState_STATE gstate;
gstate = PyGILState_Ensure();

/* Perform Python actions here. */
result = CallSomeFunction();
/* evaluate result or handle exception */

/* Release the thread. No Python API allowed beyond this point. */
PyGILState_Release(gstate);

It might be possible with cython if we register them into two packed functions, but I don't have much idea about ctypes :-(

[junrushao]

@zxybazh Looks like we can actually register these APIs with ctypes: apache/tvm#7919 (comment)

[zxybazh]

Something related: A viable solution for Python concurrency

[junrushao]

It’s a fancy story, which won’t become a real solution in short term