Improve speed of the test harness by using "pytest-xdist"

[amotl]

Hi there,

the speed of the test harness can be tremendously improved by using pytest-xdist's parallelization feature. While @kmuehlbauer reported to have different results on CI, at least on my workstation I am seeing an improvement by factor 3.5.

With kind regards,
Andreas.

Closes #253.

[amotl]

Problem

@kmuehlbauer also gave us a heads-up regarding potential concurrency issues:

One thing to keep in mind, if unspecific errors are occurring, this might be due to race conditions between different parallel tests.

Guilty as charged!

______________________ ERROR collecting tests/test_cli.py ______________________
tests/test_cli.py:8: in <module>
    from wetterdienst import cli
wetterdienst/__init__.py:4: in <module>
    from wetterdienst import dwd
wetterdienst/dwd/__init__.py:2: in <module>
    import wetterdienst.dwd.pandas
wetterdienst/dwd/pandas.py:7: in <module>
    from wetterdienst.dwd.observations.metadata import DWDObservationResolution
wetterdienst/dwd/observations/__init__.py:1: in <module>
    from wetterdienst.dwd.observations.api import (
wetterdienst/dwd/observations/api.py:11: in <module>
    from wetterdienst.dwd.index import _create_file_index_for_dwd_server
wetterdienst/dwd/index.py:13: in <module>
    from wetterdienst.util.cache import (
wetterdienst/util/cache.py:28: in <module>
    arguments={"filename": os.path.join(cache_dir, "metaindex.dbm")},
.nox/tests-3-6/lib/python3.6/site-packages/dogpile/cache/region.py:491: in configure
    self.backend = backend_cls(arguments or {})
.nox/tests-3-6/lib/python3.6/site-packages/dogpile/cache/backends/file.py:160: in __init__
    self._init_dbm_file()
.nox/tests-3-6/lib/python3.6/site-packages/dogpile/cache/backends/file.py:183: in _init_dbm_file
    fh = dbm.open(self.filename, "c")
/opt/hostedtoolcache/Python/3.6.12/x64/lib/python3.6/dbm/__init__.py:94: in open
    return mod.open(file, flag, mode)
E   _gdbm.error: [Errno 11] Resource temporarily unavailable

-- https://github.com/earthobservations/wetterdienst/pull/254/checks?check_run_id=1455730816#step:5:79

Research

Others are observing the same things when accessing a dbmfile-based cache concurrently.

• _gdbm.error: [Errno 11] Resource temporarily unavailable RIPE-NCC/ripe-atlas-tools#177
• Regression issue: _gdbm.error: [Errno 11] Resource temporarily unavailable dvarounis/perfservmon#11
• https://access.redhat.com/discussions/4638681
• https://stackoverflow.com/questions/52381091/resource-temporarily-unavailable-on-python-shelve-open

Thoughts

This is yet another reason to get rid of the dbmfile-based cache, see also #243.

[gutzbenj]

the speed of the test harness can be tremendously improved by using pytest-xdist's parallelization feature. While @kmuehlbauer reported to have different results on CI, at least on my workstation I am seeing an improvement by factor 3.5.

Anyway this is a nice improvement to have looking at the uncountable types when one of us is working on a feature and just has to wait for the test results on their local machine. So go for it!