Why is the test_convert_deps_to_pip flaky, especially on Mac OS?

[matteius]

I am considering we should skip this test so we don't appear to have broken CI builds 90% of the time, and use this ticket to document the fact we still need to understand why this test sometimes fails on Mac OS CI.

Issue description

Example: https://github.com/pypa/pipenv/runs/8057466423?check_suite_focus=true

FAILED tests/unit/test_utils.py::test_convert_deps_to_pip[deps0-requests] - A...
FAILED tests/unit/test_utils.py::test_convert_deps_to_pip[deps1-requests[socks]]

test_convert_deps_to_pip[deps1-requests[socks]] failed; it passed 0 out of the required 1 times.
	<class 'AssertionError'>
	assert ['requests[socks]==2.19.1'] == ['requests[socks]']
  At index 0 diff: 'requests[socks]==2.19.1' != 'requests[socks]'
  Full diff:
  - ['requests[socks]']
  + ['requests[socks]==2.19.1']
  ?                  ++++++++

Expected result

The test used to pass consistently.

[matteius]

@dqkqd I was digging into this locally, and I discovered that one of the test cases you added test_get_constraints_from_deps doesn't appear to run in the CI: https://github.com/pypa/pipenv/runs/8054256358?check_suite_focus=true

I ran it locally in pycharm, and one of the test cases fail, specifically: ({"FooProject": {"path": ".", "editable": "true"}}, []),
I get this error:

FAILED    [ 19%]
tests\unit\test_utils.py:137 (test_get_constraints_from_deps[deps1-expected1])
self = <[AttributeError("'Requirement' object has no attribute 'name'") raised in repr()] Requirement object at 0x27e9cd79de0>
requirement_string = '.'

    def __init__(self, requirement_string: str) -> None:
        try:
>           req = REQUIREMENT.parseString(requirement_string)

..\..\..\..\.virtualenvs\pipenv-ZKcqGg2F\lib\site-packages\pkg_resources\_vendor\packaging\requirements.py:102: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = {string_start Combine:({W:(0-9A-Za-z) [{W:(0-9A-Za-z) | {[W:(-._)]... W:(0-9A-Za-z)}}]...}) [Suppress:('[') [Combine:(... {string enclosed in "'" | string enclosed in '"'}) | Group:({{Suppress:('(') : ...} Suppress:(') Empty}]}} string_end}
instring = '.', parse_all = False

    def parse_string(
        self, instring: str, parse_all: bool = False, *, parseAll: bool = False
    ) -> ParseResults:
        """
        Parse a string with respect to the parser definition. This function is intended as the primary interface to the
        client code.
    
        :param instring: The input string to be parsed.
        :param parse_all: If set, the entire input string must match the grammar.
        :param parseAll: retained for pre-PEP8 compatibility, will be removed in a future release.
        :raises ParseException: Raised if ``parse_all`` is set and the input string does not match the whole grammar.
        :returns: the parsed data as a :class:`ParseResults` object, which may be accessed as a `list`, a `dict`, or
          an object with attributes if the given parser includes results names.
    
        If the input string is required to match the entire grammar, ``parse_all`` flag must be set to ``True``. This
        is also equivalent to ending the grammar with :class:`StringEnd`().
    
        To report proper column numbers, ``parse_string`` operates on a copy of the input string where all tabs are
        converted to spaces (8 spaces per tab, as per the default in ``string.expandtabs``). If the input string
        contains tabs and the grammar uses parse actions that use the ``loc`` argument to index into the string
        being parsed, one can ensure a consistent view of the input string by doing one of the following:
    
        - calling ``parse_with_tabs`` on your grammar before calling ``parse_string`` (see :class:`parse_with_tabs`),
        - define your parse action using the full ``(s,loc,toks)`` signature, and reference the input string using the
          parse action's ``s`` argument, or
        - explicitly expand the tabs in your input string before calling ``parse_string``.
    
        Examples:
    
        By default, partial matches are OK.
    
        >>> res = Word('a').parse_string('aaaaabaaa')
        >>> print(res)
        ['aaaaa']
    
        The parsing behavior varies by the inheriting class of this abstract class. Please refer to the children
        directly to see more examples.
    
        It raises an exception if parse_all flag is set and instring does not match the whole grammar.
    
        >>> res = Word('a').parse_string('aaaaabaaa', parse_all=True)
        Traceback (most recent call last):
        ...
        pyparsing.ParseException: Expected end of text, found 'b'  (at char 5), (line:1, col:6)
        """
        parseAll = parse_all or parseAll
    
        ParserElement.reset_cache()
        if not self.streamlined:
            self.streamline()
        for e in self.ignoreExprs:
            e.streamline()
        if not self.keepTabs:
            instring = instring.expandtabs()
        try:
            loc, tokens = self._parse(instring, 0)
            if parseAll:
                loc = self.preParse(instring, loc)
                se = Empty() + StringEnd()
                se._parse(instring, loc)
        except ParseBaseException as exc:
            if ParserElement.verbose_stacktrace:
                raise
            else:
                # catch and re-raise exception from here, clearing out pyparsing internal stack trace
>               raise exc.with_traceback(None)
E               pkg_resources._vendor.pyparsing.exceptions.ParseException: Expected W:(0-9A-Za-z), found '.'  (at char 0), (line:1, col:1)

..\..\..\..\.virtualenvs\pipenv-ZKcqGg2F\lib\site-packages\pkg_resources\_vendor\pyparsing\core.py:1141: ParseException

During handling of the above exception, another exception occurred:

self = <Line (editable=True, name=None, path=None, uri=None, extras=(), markers=None, vcs=None, specifier=None, pyproject=None, pyproject_requires=None, pyproject_backend=None, ireq=None)>

    def _parse_name_from_line(self):
        # type: () -> Optional[STRING_TYPE]
        if not self.is_named:
            pass
        try:
>           self._requirement = init_requirement(self.line)

..\..\pipenv\vendor\requirementslib\models\requirements.py:966: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

name = '.'

    def init_requirement(name):
        # type: (AnyStr) -> TRequirement
    
        if not isinstance(name, str):
            raise TypeError("must supply a name to generate a requirement")
        from pkg_resources import Requirement
    
>       req = Requirement.parse(name)

..\..\pipenv\vendor\requirementslib\models\utils.py:194: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

s = '.'

    @staticmethod
    def parse(s):
>       req, = parse_requirements(s)

..\..\..\..\.virtualenvs\pipenv-ZKcqGg2F\lib\site-packages\pkg_resources\__init__.py:3147: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <[AttributeError("'Requirement' object has no attribute 'name'") raised in repr()] Requirement object at 0x27e9cd79de0>
requirement_string = '.'

    def __init__(self, requirement_string):
        """DO NOT CALL THIS UNDOCUMENTED METHOD; use Requirement.parse()!"""
>       super(Requirement, self).__init__(requirement_string)

..\..\..\..\.virtualenvs\pipenv-ZKcqGg2F\lib\site-packages\pkg_resources\__init__.py:3102: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <[AttributeError("'Requirement' object has no attribute 'name'") raised in repr()] Requirement object at 0x27e9cd79de0>
requirement_string = '.'

    def __init__(self, requirement_string: str) -> None:
        try:
            req = REQUIREMENT.parseString(requirement_string)
        except ParseException as e:
>           raise InvalidRequirement(
                f'Parse error at "{ requirement_string[e.loc : e.loc + 8]!r}": {e.msg}'
            )
E           pkg_resources.extern.packaging.requirements.InvalidRequirement: Parse error at "'.'": Expected W:(0-9A-Za-z)

..\..\..\..\.virtualenvs\pipenv-ZKcqGg2F\lib\site-packages\pkg_resources\_vendor\packaging\requirements.py:104: InvalidRequirement

During handling of the above exception, another exception occurred:

deps = {'FooProject': {'editable': 'true', 'path': '.'}}, expected = []

    @pytest.mark.utils
    @pytest.mark.parametrize(
        "deps, expected",
        [
            ({"requests": {}}, ["requests"]),
            ({"FooProject": {"path": ".", "editable": "true"}}, []),
            ({"FooProject": {"version": "==1.2"}}, ["fooproject==1.2"]),
            ({"requests": {"extras": ["security"]}}, []),
            ({"requests": {"extras": []}}, ["requests"]),
            ({"extras": {}}, ["extras"]),
            ({"uvicorn[standard]": {}}, [])
        ],
    )
    def test_get_constraints_from_deps(deps, expected):
>       assert dependencies.get_constraints_from_deps(deps) == expected

test_utils.py:152: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
..\..\pipenv\utils\dependencies.py:280: in get_constraints_from_deps
    new_dep = Requirement.from_pipfile(dep_name, dep)
..\..\pipenv\vendor\requirementslib\models\requirements.py:2746: in from_pipfile
    r = FileRequirement.from_pipfile(name, pipfile)
..\..\pipenv\vendor\requirementslib\models\requirements.py:1838: in from_pipfile
    arg_dict["parsed_line"] = Line(line)
..\..\pipenv\vendor\requirementslib\models\requirements.py:173: in __init__
    self.parse()
..\..\pipenv\vendor\requirementslib\models\requirements.py:1304: in parse
    self.parse_name()
..\..\pipenv\vendor\requirementslib\models\requirements.py:1027: in parse_name
    name = self._parse_name_from_line()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <Line (editable=True, name=None, path=None, uri=None, extras=(), markers=None, vcs=None, specifier=None, pyproject=None, pyproject_requires=None, pyproject_backend=None, ireq=None)>

    def _parse_name_from_line(self):
        # type: () -> Optional[STRING_TYPE]
        if not self.is_named:
            pass
        try:
            self._requirement = init_requirement(self.line)
        except Exception:
>           raise RequirementError(
                "Failed parsing requirement from {0!r}".format(self.line)
            )
E           pipenv.vendor.requirementslib.exceptions.RequirementError: Failed parsing requirement from '.'

..\..\pipenv\vendor\requirementslib\models\requirements.py:968: RequirementError

Seems maybe the test runner just isn't running it? When you run that locally what happens? This is not specifically the cause of the above issue, but the tests are co-located, so figured I would just tag you here. Also do you have any interest in joining the Python Developers slack group? That is where @oz123 and I hang out and collaborate via chat.

[matteius]

I suspect that test failure is expected based on: sarugaku/requirementslib#306
and #4900

[matteius]

Actually though it did run and passed in the CI, I just wasn't searching the test output properly. I am confused on this one.

[matteius]

https://github.com/pypa/pipenv/pull/5305/files

[dqkqd]

I would love to join slack group.

About the test_get_constraints_from_deps , I tested in my laptop, it passed all 7 cases.
The packages template "FooProject": {"path": ".", "editable": "true"} I took from https://github.com/pypa/pipfile#pipfile

gw0 [7] / gw1 [7] / gw2 [7] / gw3 [7] / gw4 [7] / gw5 [7] / gw6 [7] / gw7 [7] / gw8 [7] / gw9 [7] / gw10 [7] / gw11 [7] / gw12 [7] / gw13 [7] / gw14 [7] / gw15 [7]
scheduling tests via LoadScheduling

tests/unit/test_utils.py::test_get_constraints_from_deps[deps2-expected2]
tests/unit/test_utils.py::test_get_constraints_from_deps[deps5-expected5]
tests/unit/test_utils.py::test_get_constraints_from_deps[deps3-expected3]
tests/unit/test_utils.py::test_get_constraints_from_deps[deps0-expected0]
tests/unit/test_utils.py::test_get_constraints_from_deps[deps6-expected6]
tests/unit/test_utils.py::test_get_constraints_from_deps[deps1-expected1]
tests/unit/test_utils.py::test_get_constraints_from_deps[deps4-expected4]
[gw0] [ 14%] PASSED tests/unit/test_utils.py::test_get_constraints_from_deps[deps0-expected0]
[gw2] [ 28%] PASSED tests/unit/test_utils.py::test_get_constraints_from_deps[deps2-expected2]
[gw3] [ 42%] PASSED tests/unit/test_utils.py::test_get_constraints_from_deps[deps3-expected3]
[gw5] [ 57%] PASSED tests/unit/test_utils.py::test_get_constraints_from_deps[deps5-expected5]
[gw4] [ 71%] PASSED tests/unit/test_utils.py::test_get_constraints_from_deps[deps4-expected4]
[gw6] [ 85%] PASSED tests/unit/test_utils.py::test_get_constraints_from_deps[deps6-expected6]
[gw1] [100%] PASSED tests/unit/test_utils.py::test_get_constraints_from_deps[deps1-expected1]

UPDATE:
I try with this Pipfile and pipenv failed to install.

[packages]
"e1839a8" = {path = ".", editable = true}

Dockerfile

FROM archlinux
ENV LANG C.UTF-8
RUN pacman -Syu git python3 python-pip --noconfirm
RUN python -m pip install pipenv
COPY Pipfile .
RUN pipenv install

result

Step 6/6 : RUN pipenv install
 ---> Running in 74f5d7f4f2fd
Creating a virtualenv for this project...
Pipfile: /Pipfile
Using /usr/sbin/python (3.10.6) to create virtualenv...
⠼ Creating virtual environment...created virtual environment CPython3.10.6.final.0-64 in 201ms
  creator CPython3Posix(dest=/root/.local/share/virtualenvs/-x-v5uFv0, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
    added seed packages: pip==22.2.2, setuptools==63.4.1, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

✔ Successfully created virtual environment!
Virtualenv location: /root/.local/share/virtualenvs/-x-v5uFv0
Pipfile.lock not found, creating...
Locking [packages] dependencies...
Building requirements...
Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/pkg_resources/_vendor/packaging/requirements.py", line 102, in __init__
    req = REQUIREMENT.parseString(requirement_string)
  File "/usr/lib/python3.10/site-packages/pkg_resources/_vendor/pyparsing/core.py", line 1141, in parse_string
    raise exc.with_traceback(None)
pkg_resources._vendor.pyparsing.exceptions.ParseException: Expected W:(0-9A-Za-z), found '.'  (at char 0), (line:1, col:1)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 966, in _parse_name_from_line
    self._requirement = init_requirement(self.line)
  File "/usr/lib/python3.10/site-packages/pipenv/vendor/requirementslib/models/utils.py", line 194, in init_requirement
    req = Requirement.parse(name)
  File "/usr/lib/python3.10/site-packages/pkg_resources/__init__.py", line 3147, in parse
    req, = parse_requirements(s)
  File "/usr/lib/python3.10/site-packages/pkg_resources/__init__.py", line 3102, in __init__
    super(Requirement, self).__init__(requirement_string)
  File "/usr/lib/python3.10/site-packages/pkg_resources/_vendor/packaging/requirements.py", line 104, in __init__
    raise InvalidRequirement(
pkg_resources.extern.packaging.requirements.InvalidRequirement: Parse error at "'.'": Expected W:(0-9A-Za-z)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/sbin/pipenv", line 8, in <module>
    sys.exit(cli())
  File "/usr/lib/python3.10/site-packages/pipenv/vendor/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3.10/site-packages/pipenv/cli/options.py", line 56, in main
    return super().main(*args, **kwargs, windows_expand_args=False)
  File "/usr/lib/python3.10/site-packages/pipenv/vendor/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3.10/site-packages/pipenv/vendor/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3.10/site-packages/pipenv/vendor/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3.10/site-packages/pipenv/vendor/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/lib/python3.10/site-packages/pipenv/vendor/click/decorators.py", line 84, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/usr/lib/python3.10/site-packages/pipenv/vendor/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/lib/python3.10/site-packages/pipenv/cli/command.py", line 233, in install
    do_install(
  File "/usr/lib/python3.10/site-packages/pipenv/core.py", line 2064, in do_install
    do_init(
  File "/usr/lib/python3.10/site-packages/pipenv/core.py", line 1325, in do_init
    do_lock(
  File "/usr/lib/python3.10/site-packages/pipenv/core.py", line 1122, in do_lock
    venv_resolve_deps(
  File "/usr/lib/python3.10/site-packages/pipenv/utils/resolver.py", line 1026, in venv_resolve_deps
    deps = convert_deps_to_pip(deps, project, r=False, include_index=True)
  File "/usr/lib/python3.10/site-packages/pipenv/utils/dependencies.py", line 261, in convert_deps_to_pip
    new_dep = Requirement.from_pipfile(dep_name, dep)
  File "/usr/lib/python3.10/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 2746, in from_pipfile
    r = FileRequirement.from_pipfile(name, pipfile)
  File "/usr/lib/python3.10/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 1838, in from_pipfile
    arg_dict["parsed_line"] = Line(line)
  File "/usr/lib/python3.10/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 173, in __init__
    self.parse()
  File "/usr/lib/python3.10/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 1304, in parse
    self.parse_name()
  File "/usr/lib/python3.10/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 1027, in parse_name
    name = self._parse_name_from_line()
  File "/usr/lib/python3.10/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 968, in _parse_name_from_line
    raise RequirementError(
pipenv.vendor.requirementslib.exceptions.RequirementError: Failed parsing requirement from '.'

[matteius]

@dqkqd Try joining the slack group here: pythondev.slack.com
Otherwise you can email me and I'll request you be added via email.

[matteius]

I think the reason that test passes maybe is because it is generally being invoked from the pipenv root which has an installable thing at path ="."

[matteius]

Ok the tests have been passing since I made this change to the tests: 46f8c86

What I discovered is that requirementslib will inspect into a directory named requests to find the setup.py version info and the test that I skipped is likely the biggest culprit of this because it copies in the requests tarball and extracts it. We would like to get that test back to not-skipped, but it needs to cleanup after itself. We should also look into if this is a bug in requirementslib that it can get confused like this.