diff --git a/tests/REGRESSION.md b/tests/REGRESSION.md index d0694f8e2762..e0d37edd56f1 100644 --- a/tests/REGRESSION.md +++ b/tests/REGRESSION.md @@ -1,10 +1,11 @@ ## Regression tests for typeshed Regression tests for the standard library stubs can be found in the -`stdlib/@tests/test_cases` directory. Stubs for third-party libraries that do -have test cases can be found in `@tests/test_cases` subdirectories for each -stubs package. For example, the test cases for `requests` can be found in the -`stubs/requests/@tests/test_cases` directory. +`stdlib/@tests/test_cases` directory. Not all third-party-library stub +packages in typeshed have test cases, and not all of them need test cases -- +but for those that do, their test cases can be found in `@tests/test_cases` +subdirectories for each stubs package. For example, the test cases for +`requests` can be found in the `stubs/requests/@tests/test_cases` directory. **Regression test cases should only be written for functions and classes which are known to have caused problems in the past, where the stubs are difficult to @@ -14,14 +15,16 @@ multiple other mechanisms for spotting errors in the stubs. ### The purpose of these tests -Different test cases in this directory serve different purposes. For some stubs in -typeshed, the type annotations are complex enough that it's useful to have +Different test cases in typeshed serve different purposes. For some typeshed +stubs, the type annotations are complex enough that it's useful to have sanity checks that test whether a type checker understands the intent of the annotations correctly. Examples of tests like these are -`builtins/check_pow.py` and `asyncio/check_gather.py`. +`stdlib/@tests/test_cases/builtins/check_pow.py` and +`stdlib/@tests/test_cases/asyncio/check_gather.py`. -Other test cases, such as the samples for `ExitStack` in `check_contextlib.py` -and the samples for `LogRecord` in `check_logging.py`, do not relate to +Other test cases, such as the samples for `ExitStack` in +`stdlib/@tests/test_cases/check_contextlib.py` and the samples for `LogRecord` +in `stdlib/@tests/test_cases/check_logging.py`, do not relate to stubs where the annotations are particularly complex, but they *do* relate to stubs where decisions have been taken that might be slightly unusual. These test cases serve a different purpose: to check that type checkers do not emit @@ -29,19 +32,28 @@ false-positive errors for idiomatic usage of these classes. ## Running the tests -To verify the test cases in this directory pass with mypy, run `python tests/regr_test.py stdlib` -from the root of the typeshed repository. This assumes that the development -environment has been set up as described in the [CONTRIBUTING.md](../CONTRIBUTING.md) -document. +To verify the stdlib test cases pass with mypy, run +`python tests/regr_test.py stdlib` from the root of the typeshed repository. +This assumes that the development environment has been set up as described in +the [CONTRIBUTING.md](../CONTRIBUTING.md) document. + +For third-party-library stubs, pass the name of the runtime library the stubs +are for. For example, to run the tests for our `requests` stubs, run +`python tests/regr_test.py requests`. + +Run `python tests/regr_test.py -h` for the full range of CLI options this script +supports. There is no equivalent script for pyright currently; for pyright, the +tests are checked in CI using a GitHub Action. ### How the tests work -The code in this directory is not intended to be directly executed. Instead, -type checkers are run on the code, to check that typing errors are -emitted at the correct places. +The code in typeshed's test cases is not intended to be directly executed. +Instead, type checkers are run on the code, to check that type checker +diagnostics are emitted at the correct locations. -Some files in this directory simply contain samples of idiomatic Python, which -should not (if the stubs are correct) cause a type checker to emit any errors. +Some files in these directories simply contain samples of idiomatic Python, +which should not (if the stubs are correct) cause a type checker to emit any +diagnostics. Many test cases also make use of [`assert_type`](https://docs.python.org/3.11/library/typing.html#typing.assert_type), @@ -54,8 +66,8 @@ mypy's setting and pyright's [`reportUnnecessaryTypeIgnoreComment`](https://github.com/microsoft/pyright/blob/main/docs/configuration.md#type-check-diagnostics-settings) setting) to test instances where a type checker *should* emit some kind of -error, if the stubs are correct. Both settings are enabled by default for the entire -subdirectory. +error, if the stubs are correct. Both settings are enabled by default for +all `@tests/test_cases/` subdirectories in typeshed. For more information on using `assert_type` and `--warn-unused-ignores`/`reportUnnecessaryTypeIgnoreComment` to test type @@ -65,21 +77,21 @@ provides a useful guide. ### Naming convention -Use the same top-level name for the module / package you would like to test. +Use the same top-level name for the module or package you would like to test. Use the `check_${thing}.py` naming pattern for individual test files. By default, test cases go into a file with the same name as the stub file, prefixed with `check_`. -For example: `check_contextlib.py`. +For example: `stdlib/@tests/test_cases/check_contextlib.py`. If that file becomes too big, we instead create a directory with files named after individual objects being tested. -For example: `builtins/check_dict.py`. +For example: `stdlib/@tests/test_cases/builtins/check_dict.py`. ### Differences to the rest of typeshed -Unlike the rest of typeshed, this directory largely contains `.py` files. This -is because the purpose of this folder is to test the implications of typeshed -changes for end users, who will mainly be using `.py` files rather than `.pyi` -files. +Unlike the rest of typeshed, the `@tests/test_cases` directories largely +contain `.py` files. This is because the purpose of these directories is to +test the implications of typeshed changes for end users, who will mainly be +using `.py` files rather than `.pyi` files. Another difference to the rest of typeshed (which stems from the fact that the test-case files are all `.py` files @@ -109,7 +121,8 @@ with a specific Python version passed on the command line to the `tests/regr_tes To mark a test-case file as being skippable on lower versions of Python, append `-py3*` to the filename. For example, if `foo` is a stdlib feature that's new in Python 3.11, -test cases for `foo` should be put in a file named `check_foo-py311.py`. +test cases for `foo` should be put in a file named +`stdlib/@tests/test_cases/check_foo-py311.py`. This means that mypy will only run the test case if `--python-version 3.11`, `--python-version 3.12`, etc. is passed on the command line to `tests/regr_test.py`, @@ -118,4 +131,4 @@ is passed on the command line. However, `if sys.version_info >= (3, target):` is still required for `pyright` in the test file itself. -Example: [`check_exception_group-py311.py`](../stdlib/@tests/test_cases/builtins/check_exception_group-py311.py) +Example: [`stdlib/@tests/test_cases/check_exception_group-py311.py`](../stdlib/@tests/test_cases/builtins/check_exception_group-py311.py)