Calling pytest.main() repeatedly (possible fix for ImportPathMismatchError)

[ptyork]

I am using pytest to grade (a lot of) student assignments. Though documented as "not recommended" (https://docs.pytest.org/en/stable/how-to/usage.html#pytest-main-usage), the benefit for me of calling a bunch of tests within the same Python process were significant enough for me to bang my head against the wall on this for a couple of days.

If you don't (or can't) call pytest using the --import-mode=importlib option, then in my use case you always get the dreaded ImportPathMismatchError exception. I'm sure I'm oversimplifying, but this is because once a module is loaded, it's cached in memory and referenced by the name of the module. Assuming the modules are just files (as will be the case with the test files) then there will be duplicates as I'm running the same test files against student projects repeatedly. And Python won't overwrite the cached modules with the new versions.

To get around this, I'm "restoring" a pre-invocation version of the in-memory modules by deleting any that are loaded during the pytest run. But it's overly complicated since I also want to optimize performance by avoiding modules that pytest needs and would have to reload. So the code looks something like this:

keep_packages = ['_asyncio',
                 '_contextvars',
                 '_elementtree',
                 '_pytest',
                 '_ssl',
                 'asyncio',
                 'attr',
                 'cmd',
                 'code',
                 'codeop',
                 'contextvars',
                 'faulthandler',
                 'pdb',
                 'pkgutil',
                 'pyexpat',
                 'pytest_metadata',
                 'pytest_subtests',
                 'readline',
                 'ssl',
                 'unittest',
                 'xml']

pretest_modules = [key for key in sys.modules.keys()]

pytest.main(
    ['--tb=no', '-q']
)

posttest_modules = [key for key in sys.modules.keys()]
for module_name in posttest_modules:
    if module_name in pretest_modules:
        continue
    package = module_name.split('.')[0]
    if package in keep_packages:
        continue
    del sys.modules[module_name]

It works like a charm, but if something like this could be added directly to pytest.main()--perhaps as an optional argument ike remove-loaded-modules=True to avoid unexpected side effects for others)--it could be much more efficient. Simply capture the environment state immediately prior to invoking tests and restoring it immediately following.

And of course it could save a lot of bruised foreheads and damaged walls.

I looked at the source and was too overwhelmed to attempt a pull request. But someone familiar with it likely could make this happen quite easily if it makes sense.

[webknjaz]

Honestly, I don't feel like such hacks belong in the project. This is something to exist externally. Especially, since import machinery manipulation can be dangerous, and it'd be costly to maintain. I doubt there's enough interest to warrant accepting such a change.

[RonnyPfannschmidt]

The pytester fixture which is used for in process running of pytest in its own testsuite can serve as inspiration

The outlined usecase is not something pytest itself ought to solve as a general solution has a certain fragility whose maintenance burden ought not to be on the pytest project itself

[ptyork]

I understand the hesitance to add in maintenance. I may not have been clear since my code was complicated by a specific list of exclusions. The point was that you wouldn't need them at all if this were an embedded option. Conceptually, the code would need only be something like:

if preserve_env:
    pretest_modules = [key for key in sys.modules.keys()]
    pretest_path = sys.path.copy()

# ...MODIFY PATH AND EXEC TESTS...

if preserve_env:
    posttest_modules = [key for key in sys.modules.keys() if key not in pretest_modules]
    for module_name in posttest_modules:
        del sys.modules[module_name]
    sys.path = pretest_path.copy()

Could even be written as a simple context manager.

Why?

1. there are times when you want in-memory access to plugins. Running tests out-of-process means that plugins have to serialize state and it must subsequently be deserialized in order to access it.
2. it's vastly faster than starting new processes when running potentially hundreds of short tests.
3. it makes use via API behave in a way that is consistent with how people expect, and reverts the sys "hacks" that pytest already does.

I don't see it as being overly hacky or unsafe. Discovered test files (and any additional modules loaded by the test files) will always be "last in" and should be safely deleted. At least they were in my testing. And if it's somehow unsafe for your particular test code, you just don't add the optional flag. I guess I don't see it as any more of a hack than mocking or the existing --import-mode flag. I mean, unit testing is one big hack by design, right? 😆

[RonnyPfannschmidt]

the fact that one needs lists of modules that need break prevention is enough to make this unsafe by default

As such it is best provided as something that correctly enforces informed consent to the mechanisms

[ptyork]

@RonnyPfannschmidt I may not fully understand the alternative(s). Could you point me to the pytester fixture that you referenced in your earlier response? Thanks!

[RonnyPfannschmidt]

https://github.com/pytest-dev/pytest/blob/main/src%2F_pytest%2Fpytester.py#L477