pytest hangs in Docker

[dmurvihill]

docker-compose.yml:

python_version = "3.7"
version: '3.7'
services:
  test:
    build:
      context: .

Dockerfile:

FROM python:3.7
RUN pip install pipenv
COPY Pipfile .
RUN pipenv install --dev
COPY test_samedir.py .
COPY tests/ tests/
COPY module_dir/ module_dir/
ENTRYPOINT pipenv run pytest

Pipfile:

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]
pytest="*"

[packages]

[requires]
python_version = "3.7"
version: '3.7'

test_samedir.py

def test_samedir():
    assert 1 == 1

tests/test_subdir.py

def test_subdir():
    assert 1 == 1

Executing this environment, you get this:

Dolans-MacBook-Pro:pytest-hang dolan$ docker-compose up --build test
Building test
Step 1/8 : FROM python:3.7
 ---> fbf9f709ca9f
Step 2/8 : RUN pip install pipenv
 ---> Using cache
 ---> bafd472f23d4
Step 3/8 : COPY Pipfile .
 ---> Using cache
 ---> 64e6afb84f88
Step 4/8 : RUN pipenv install --dev
 ---> Using cache
 ---> 983a04e2cc42
Step 5/8 : COPY test_samedir.py .
 ---> Using cache
 ---> d9914e9938bd
Step 6/8 : COPY tests/ tests/
 ---> Using cache
 ---> ac8361abc237
Step 7/8 : COPY module_dir/ module_dir/
 ---> Using cache
 ---> 4a5afb7725dd
Step 8/8 : ENTRYPOINT pipenv run python -m pytest
 ---> Using cache
 ---> 737460493a9a

Successfully built 737460493a9a
Successfully tagged pytest-hang_test:latest
Starting pytest-hang_test_1 ... done
Attaching to pytest-hang_test_1
test_1  | ============================= test session starts ==============================
test_1  | platform linux -- Python 3.7.5, pytest-5.3.2, py-1.8.0, pluggy-0.13.1
test_1  | rootdir: /

And it hangs there forever. I'd expect it to report 1 or 2 tests passed.

[nicoddemus]

Hi @dmurvihill,

That's weird. What happens if you change your entry point to:

ENTRYPOINT pipenv run pytest /tests test_samedir.py -s

[RonnyPfannschmidt]

could be related to python buffering, i suggest to use PYTHONUNBUFFERED to see what happens

[MateusAmin]

Always freezes in python:3.7-stretch on the second run for me.

root@asdad# python -v -m pytest
import _frozen_importlib # frozen
import _imp # builtin
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import '_warnings' # <class '_frozen_importlib.BuiltinImporter'>
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
# installing zipimport hook
import 'zipimport' # <class '_frozen_importlib.BuiltinImporter'>
# installed zipimport hook
import '_frozen_importlib_external' # <class '_frozen_importlib.FrozenImporter'>
import '_io' # <class '_frozen_importlib.BuiltinImporter'>
import 'marshal' # <class '_frozen_importlib.BuiltinImporter'>
import 'posix' # <class '_frozen_importlib.BuiltinImporter'>
import _thread # previously loaded ('_thread')
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import _weakref # previously loaded ('_weakref')
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
# /usr/local/lib/python3.7/encodings/__pycache__/__init__.cpython-37.pyc matches /usr/local/lib/python3.7/encodings/__init__.py
# code object from '/usr/local/lib/python3.7/encodings/__pycache__/__init__.cpython-37.pyc'
# /usr/local/lib/python3.7/__pycache__/codecs.cpython-37.pyc matches /usr/local/lib/python3.7/codecs.py
# code object from '/usr/local/lib/python3.7/__pycache__/codecs.cpython-37.pyc'
import '_codecs' # <class '_frozen_importlib.BuiltinImporter'>
import 'codecs' # <_frozen_importlib_external.SourceFileLoader object at 0x7f984bb542d0>
# /usr/local/lib/python3.7/encodings/__pycache__/aliases.cpython-37.pyc matches /usr/local/lib/python3.7/encodings/aliases.py
# code object from '/usr/local/lib/python3.7/encodings/__pycache__/aliases.cpython-37.pyc'
import 'encodings.aliases' # <_frozen_importlib_external.SourceFileLoader object at 0x7f984bb72550>
import 'encodings' # <_frozen_importlib_external.SourceFileLoader object at 0x7f984bbcf710>
# /usr/local/lib/python3.7/encodings/__pycache__/utf_8.cpython-37.pyc matches /usr/local/lib/python3.7/encodings/utf_8.py
...

# /usr/local/lib/python3.7/logging/__pycache__/__init__.cpython-37.pyc matches /usr/local/lib/python3.7/logging/__init__.py
# code object from '/usr/local/lib/python3.7/logging/__pycache__/__init__.cpython-37.pyc'
import 'logging' # <_frozen_importlib_external.SourceFileLoader object at 0x7f984729aa90>
import '_pytest.logging' # <_frozen_importlib_external.SourceFileLoader object at 0x7f9847288b90>
# /usr/local/lib/python3.7/site-packages/_pytest/__pycache__/faulthandler.cpython-37.pyc matches /usr/local/lib/python3.7/site-packages/_pytest/faulthandler.py
# code object from '/usr/local/lib/python3.7/site-packages/_pytest/__pycache__/faulthandler.cpython-37.pyc'
import '_pytest.faulthandler' # <_frozen_importlib_external.SourceFileLoader object at 0x7f9847252250>
# /usr/local/lib/python3.7/site-packages/_pytest/__pycache__/_argcomplete.cpython-37.pyc matches /usr/local/lib/python3.7/site-packages/_pytest/_argcomplete.py
# code object from '/usr/local/lib/python3.7/site-packages/_pytest/__pycache__/_argcomplete.cpython-37.pyc'
# /usr/local/lib/python3.7/__pycache__/glob.cpython-37.pyc matches /usr/local/lib/python3.7/glob.py
# code object from '/usr/local/lib/python3.7/__pycache__/glob.cpython-37.pyc'
import 'glob' # <_frozen_importlib_external.SourceFileLoader object at 0x7f9847252d50>
import '_pytest._argcomplete' # <_frozen_importlib_external.SourceFileLoader object at 0x7f9847252750>
# /usr/local/lib/python3.7/__pycache__/_bootlocale.cpython-37.pyc matches /usr/local/lib/python3.7/_bootlocale.py
# code object from '/usr/local/lib/python3.7/__pycache__/_bootlocale.cpython-37.pyc'
import '_bootlocale' # <_frozen_importlib_external.SourceFileLoader object at 0x7f984720c490>

[David-Guillot]

Hi there,

We just encountered a very similar issue when starting working with Github Actions CI. A Django project with something like 150 tests involving PostgreSQL. Everything running fine on local developers computers (macOS, Ubuntu 18.04, Ubuntu 20.04) and on our previous CI (Codeship). With Github Actions (which use Docker), pytests hangs at a specific test (always the same).

I tried:

• PYTHONUNBUFFERED=1: no effect
• pytest-forked: no effect
• pytest --trace: works (produces horrible output, but I don't care for now) 🎉

[David-Guillot]

Correction: my idea of pytest --trace does not work anymore, pytest started hanging again on some test, only on Github Actions.

[nicoddemus]

Just throwing some suggestions that I would use to investigate further:

1. Add an assert 0 as the very first line of the test; if it fails, then it means it is hanging in the middle of the test. If it still hangs, it means the problem is during the test setup (in a fixture for example).

2. Looking at the fixtures used by the hanging test, do you spot anything that might suggest why it is hanging?

[David-Guillot]

Thanks @nicoddemus for your suggestions. The test in question is a bit big in terms of fixtures, but nothing enormous either: about 20 Django model objects that are saved to PostgreSQL database, some of them being then modified by factory_boy post-generation operations. Your simple-yet-I-didn't-think-of-it idea of debugging with assert False is a big help on this: the fixtures are not the problem, the test starts hanging after a first pass of action/assertions. I'll let you know how it goes, because as for other people here, I have no problem when running my test suite outside of Docker/Github Actions.

[nicoddemus]

Another tip: try using pytest-timeout, it will at least blow up the suite and print a traceback where the code is hanging.

[dominusmi]

In my case, even the timeout doesn't actually timeout! I can confirm it is setup properly, as this appears when pytest is run

platform linux -- Python 3.8.6, pytest-6.1.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/runner/work/repo
collected 37 items
plugins: timeout-1.4.2
timeout: 60.0s
timeout method: signal
timeout func_only: False

It still was going after several minutes, after which I stopped the workflow.

[xperrylinn]

@dominusmi, I've run into the same issue with pytest-timeout not actually doing anything. Did you ever figure this out?

[apollopattern]

I'm struggling with this issue. In my case, it appears that the hang (or sometimes crash) occurs during migrations. When I run
python3 manage.py test -v 3 tests.path.filename.TestClassName.test_method
the "-v 3" outputs all the migrations as they happen; it gets through 20 or 30, hangs for a while, and then crashes with a
django.db.utils.InternalError: stuck spinlock detected at LWLockWaitListLock

(The crash only seems to happen when running from the command line like this; if I run from within PyCharm, it's just a hang.)

[nicoddemus]

I suggest to try with -s to completely disable pytest's output capture.

[thetorpedodog]

We ran into a similar problem where it looked like tests were hanging when running on Azure Pipelines or in GitHub Actions—they would run up to Collected XX items and then appear to stop. The apparent hang before our test code was a red herring, as the tests were actually running. One of our tests from the first file was hanging. Because both Azure Pipelines and GitHub Actions do not send a line of output to the log until a newline is written, the first line would never complete. So if you were to run an identical test locally, you would see:

Collected XX items

test_basic.py ....... [HANG]

but because the first line never wrote, it looked like

Collected XX items

[NOTHING]

Using -v helped us identify the problem, because we could see that other tests were running until the one that hung.