gh-124984: Fix ssl thread safety

[ZeroIntensity]

As it turns out, OpenSSL doesn't like being called in multiple threads. This adds a per-socket (and per-context and per-session) lock for all OpenSSL calls.

• Issue: Segfault on Python 3.13.0rc3 (free-threaded) with requests #124984

[JelleZijlstra]

Thanks, that's a lot of locks.

[vstinner]

Tests / Thread sanitizer (free-threading)

test_ssl fails with:

0:03:08 load avg: 6.09 [15/22/1] test_ssl failed (1 failure) (33.9 sec) -- running (1): test_socket (1 min 15 sec)
test test_ssl failed -- Traceback (most recent call last):
  File "/home/runner/work/cpython/cpython/Lib/test/test_ssl.py", line 2848, in test_ssl_in_multiple_threads
    self.assertIsNone(cm.exc_value)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
AssertionError: SkipTest("resource 'walltime' is not enabled") is not None

[ZeroIntensity]

Oh, I didn't see that one, nevermind. I'll fix that in the next hour or so.

[vstinner]

Merged, thank you! I replaced "Fix" with "Enhance" in the commit message: "Enhance ssl thread safety" :-)

[vstinner]

Remaining question: should we backport this enhancement (bugfix?) to Python 3.13?

[ZeroIntensity]

Remaining question: should we backport this enhancement (bugfix?) to Python 3.13?

Yeah, I had the same thought. I'll leave that decision to @Yhg1s

[Yhg1s]

Backport this to 3.13? Uuhm...

No.

[vstinner]

Without this change, the ssl module is not usable on a Free Threaded build (it does easily crash). The change only affects the Free Threaded build: adding @critical_section does nothing in the regular build, and the added test is ignored on regular Python (only run on Free Threading). I would be fine with no backporting it, since Free Threading remains experimental.

[colesbury]

I think that fixing the SSL module crash in the 3.13 free threading build is important -- lots of basic tasks around HTTP requests are likely to crash without it.

I don't think "lines of code changed" is a good measure of the complexity here -- the Modules/_ssl.c changes are mostly mechanical, and the only behavioral changes is additional locking in the free threading build.

If backporting this PR is a nonstarter than we should consider a smaller, more targeted change that only adds @critical_section to SSLContext methods (without changing getters and setters). That seems less ideal to me, but should fix the common SSL crash and be a pretty small code change.

[colesbury]

Also, thank you @ZeroIntensity for fixing this bug and @vstinner, @corona10, and everyone else that reviewed the PR.

[miss-islington-app]

Thanks @ZeroIntensity for the PR, and @vstinner for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13.
🐍🍒⛏🤖

[miss-islington-app]

Sorry, @ZeroIntensity and @vstinner, I could not cleanly backport this to 3.13 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 4c53b2577531c77193430cdcd66ad6385fcda81f 3.13

[vstinner]

@ZeroIntensity: Automated backport failed. Would you mind to backport the change manually?

With a backport, it might be easier to take a decision on fixing 3.13 or not.

[ZeroIntensity]

Yeah, I can do it later today.

[bedevere-app]

GH-125780 is a backport of this pull request to the 3.13 branch.

[vstinner]

@Yhg1s: Are you still against the backport to 3.13 after @colesbury's comment?

[Yhg1s]

I'm okay with a backport of just the @critical_section changes (since those expand to nothing in the normal build, and the free-threaded build is experimental anyway). It's the larger refactorings that worry me.

[vstinner]

It's the larger refactorings that worry me.

Other changes are tests and changes to use the code declared with @critical_section.

[ZeroIntensity]

There isn't any other refactoring going on here, I just had to switch the getters and setters over to AC for the critical section. Another issue is that not backporting this to 3.13 will also hurt any automatic backports for ssl in the future.