Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix segmentation fault caused by using libraries depending on pybind11 #9

Merged
merged 2 commits into from
Feb 25, 2025

Conversation

jonatanklosko
Copy link
Member

This is actually a bug upstream in pybind11, which is a popular library for implementing C extension modules in Python. The bug concerns the GIL, it has been reported a long time ago (pybind/pybind11#2888) and still exists. I found another viable way for us to manage the GIL, such that we don't run into that issue. More details in the code comment.

For the record, below is the code that reliably reproduced the issue for me.

script.exs
IO.gets("PID: #{:os.getpid()} Press enter...")

Mix.install([
  {:pythonx, path: "~/git/pythonx"},
])

Pythonx.uv_init("""
[project]
name = "project"
version = "0.0.0"
requires-python = "==3.13.*"
dependencies = [
  "matplotlib"
]
""")

globals = %{}

# Start a long-running evaluation that releases the GIL, it likely
# runs on the first dirty scheduler thread. This way we can start
# another evaluation and it will run on another dirty scheduler
# thread.
spawn(fn ->
  {_, _globals} =
    Pythonx.eval(
      """
      import time
      time.sleep(20)
      """,
      %{}
    )
end)

Process.sleep(100)

{_, globals} =
  Pythonx.eval(
    """
    import matplotlib
    matplotlib.use("Agg")

    import matplotlib.pyplot as plt
    plt.plot([1, 2], [1, 2])
    """,
    globals
  )

{_, _globals} =
  Pythonx.eval(
    """
    import io
    buffer = io.BytesIO()
    plt.gcf().savefig(buffer, format="png", bbox_inches="tight")
    """,
    globals
  )

(I also removed the interpreter finalization NIF, because we don't use and it's unlikely we ever will, so it's not worth keeping it in sync).

@jonatanklosko
Copy link
Member Author

Looks like it doesn't work on Windows Python 3.10, I will keep investigating.

@jonatanklosko jonatanklosko merged commit 1dd28d0 into main Feb 25, 2025
9 checks passed
@jonatanklosko jonatanklosko deleted the jk-gil branch February 25, 2025 06:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant