Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve testing story for Python 3.14 and free-threading builds #4028

Open
1 of 6 tasks
Zac-HD opened this issue Jul 4, 2024 · 4 comments
Open
1 of 6 tasks

Improve testing story for Python 3.14 and free-threading builds #4028

Zac-HD opened this issue Jul 4, 2024 · 4 comments
Labels
tests/build/CI about testing or deployment *of* Hypothesis

Comments

@Zac-HD
Copy link
Member

Zac-HD commented Jul 4, 2024

#4025 got some initial wins - like "running 3.14 in CI" and "ensuring that our auto-updates set up free-threading environments for us", but there's plenty more to do

  • turn on CI for the free-threading builds (e.g. 3.13.0t-dev) and see what breaks. Might be blocked on upstream pydata packages for extensions? We also don't guarantee thread-safety at the moment, so I expect some chaos here and we might just turn the tests off again for a while.
  • search for FIXME-py314 and... fix them
    • with the removal of typing.Bytestring, we no longer generate bytes instances from Sequence[int]; I think we probably should still do this
    • test_suggests_elements_instead_of_annotations is failing, so it seems something about dataclass introspection changed
    • some attrs introspection tests in Upgrade pinned dependencies #4069
  • fix any other 3.14-specific issues that come up if or when they arise
@Zac-HD Zac-HD added the tests/build/CI about testing or deployment *of* Hypothesis label Jul 4, 2024
@andfoy
Copy link

andfoy commented Aug 26, 2024

@Zac-HD I've been testing Hypothesis against the SciPy testsuite using the pytest-run-parallel (https://github.com/Quansight-Labs/pytest-run-parallel) plugin on 313t , which enables any test to be run concurrently using N threads.

Specifically, I've ran into issues originating from Hypothesis, which appear from time to time when using multiple threads to run this test:

            gc.collect()
            if not gc.get_referrers(r):
                if sys.getrefcount(r) <= _PLATFORM_REF_COUNT:
>                   raise ReferenceError(
                        f"`register_random` was passed `r={r}` which will be "
                        "garbage collected immediately after `register_random` creates a "
                        "weakref to it. This will prevent Hypothesis from managing this "
                        "PRNG. See the docs for `register_random` for more "
                        "details."
E                       ReferenceError: `register_random` was passed `r=<random.Random object at 0x4305a3f0a10>` which will be garbage collected immediately after `register_random` creates a weakref to it. This will prevent Hypothesis from managing this PRNG. See the docs for `register_random` for more details.

r          = <random.Random object at 0x4305a3f0a10>

../../../../.pyenv/versions/3.13.0rc1t/lib/python3.13t/site-packages/hypothesis/internal/entropy.py:123: ReferenceError
 def process_arguments_to_given(wrapped_test, arguments, kwargs, given_kwargs, params):
        selfy = None
        arguments, kwargs = convert_positional_arguments(wrapped_test, arguments, kwargs)
    
        # If the test function is a method of some kind, the bound object
        # will be the first named argument if there are any, otherwise the
        # first vararg (if any).
>       posargs = [p.name for p in params.values() if p.kind is p.POSITIONAL_OR_KEYWORD]
E       RecursionError: maximum recursion depth exceeded

arguments  = ()
given_kwargs = {'data': data(), 'dtype': sampled_from((<class 'numpy.float32'>, <class 'numpy.float64'>)), 'n_arrays': integers(min_value=1, max_value=3), 'p': floats(min_value=0, max_value=1), ...}
kwargs     = {'self': <scipy._lib.tests.test__util.TestLazywhere object at 0x4a0c3cc21d0>, 'xp': <module 'numpy' from '/home/andfoy/.pyenv/versions/3.13.0rc1t/lib/python3.13t/site-packages/numpy/__init__.py'>}
params     = mappingproxy(OrderedDict({'self': <Parameter "self">, 'xp': <Parameter "xp">}))
selfy      = None
wrapped_test = <function accept.<locals>.test_basic at 0x4a0c25f87a0>

../../../../.pyenv/versions/3.13.0rc1t/lib/python3.13t/site-packages/hypothesis/core.py:660: RecursionError

Other times, the test passes successfully, which suggests that some kind of race condition or non-isolation of parameters is occurring when running Hypothesis tests under parallel loads in the new free-threaded CPython. I wanted to ask you for any helpful pointers to help the project solve this kind of issues

@jobh
Copy link
Contributor

jobh commented Aug 27, 2024

@andfoy

The first one is probably a race here, or in a similar section,

if hypothesis.core._hypothesis_global_random is None: # pragma: no cover
hypothesis.core._hypothesis_global_random = random.Random()
register_random(hypothesis.core._hypothesis_global_random)

where a second thread reassigns _hypothesis_global_random (and hence unreferences the first thread's random instance) before the first thread is finished registering it.

Fixable (if my guess is correct), but I suspect there may be many such cases and some of them may fail in less obvious ways...

@andfoy
Copy link

andfoy commented Aug 27, 2024

@jobh, thanks for the explanation! I've just opened a PR (#4094) that addresses this issue, which in turn enables the aforementioned test to be run successfully.

@Zac-HD
Copy link
Member Author

Zac-HD commented Jan 25, 2025

3.14 is looking decent in this build, but I'm leaving disabled for now because the alphas do tend to break stuff pretty regularly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
tests/build/CI about testing or deployment *of* Hypothesis
Projects
None yet
Development

No branches or pull requests

3 participants