Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multiple tests fail with napalm 4.0.0 #34

Open
mcepl opened this issue Jan 3, 2023 · 0 comments
Open

Multiple tests fail with napalm 4.0.0 #34

mcepl opened this issue Jan 3, 2023 · 0 comments

Comments

@mcepl
Copy link

mcepl commented Jan 3, 2023

While packaging for openSUSE/Factory with napalm 4.0.0 some tests are failing:

[   28s] =================================== FAILURES ===================================
[   28s] ______________________ TestGetter.test_method_signatures _______________________
[   28s] 
[   28s] self = <test.unit.test_getters.TestGetter object at 0x7fe5c6be0610>
[   28s] 
[   28s]     def test_method_signatures(self):
[   28s]         """
[   28s]         Test that all methods have the same signature.
[   28s]     
[   28s]         The type hint annotations are ignored here because the import paths might differ."""
[   28s]         errors = {}
[   28s]         cls = self.driver
[   28s]         # Create fictional driver instance (py3 needs bound methods)
[   28s]         tmp_obj = cls(hostname="test", username="admin", password="pwd")
[   28s]         attrs = [m for m, v in inspect.getmembers(tmp_obj)]
[   28s]         for attr in attrs:
[   28s]             func = getattr(tmp_obj, attr)
[   28s]             if attr.startswith("_") or not inspect.ismethod(func):
[   28s]                 continue
[   28s]             try:
[   28s]                 orig = getattr(NetworkDriver, attr)
[   28s]                 orig_spec = inspect.getfullargspec(orig)[:4]
[   28s]             except AttributeError:
[   28s]                 orig_spec = "Method does not exist in napalm.base"
[   28s]             func_spec = inspect.getfullargspec(func)[:4]
[   28s]             if orig_spec != func_spec:
[   28s]                 errors[attr] = (orig_spec, func_spec)
[   28s]     
[   28s]         EXTRA_METHODS = ["__init__"]
[   28s]         for method in EXTRA_METHODS:
[   28s]             orig_spec = inspect.getfullargspec(getattr(NetworkDriver, method))[:4]
[   28s]             func_spec = inspect.getfullargspec(getattr(cls, method))[:4]
[   28s]             if orig_spec != func_spec:
[   28s]                 errors[attr] = (orig_spec, func_spec)
[   28s]     
[   28s] >       assert not errors, "Some methods vary. \n{}".format(errors.keys())
[   28s] E       AssertionError: Some methods vary. 
[   28s] E       dict_keys(['cli'])
[   28s] 
[   28s] /usr/lib/python3.8/site-packages/napalm/base/test/getters.py:141: AssertionError
[   28s] ______________________ TestGetter.test_get_facts[normal] _______________________
[   28s] 
[   28s] self = <test.unit.test_getters.TestGetter object at 0x7fe5c6be0e80>
[   28s] test_case = 'normal'
[   28s] 
[   28s]     @wrap_test_cases
[   28s]     def test_get_facts(self, test_case):
[   28s]         """Test get_facts method."""
[   28s]         facts = self.device.get_facts()
[   28s] >       assert helpers.test_model(models.FactsDict, facts)
[   28s] E       AssertionError
[   28s] 
[   28s] /usr/lib/python3.8/site-packages/napalm/base/test/getters.py:154: AssertionError
[   28s] _______________________ TestGetter.test_get_facts[stack] _______________________
[   28s] 
[   28s] self = <test.unit.test_getters.TestGetter object at 0x7fe5c6bc0760>
[   28s] test_case = 'stack'
[   28s] 
[   28s]     @wrap_test_cases
[   28s]     def test_get_facts(self, test_case):
[   28s]         """Test get_facts method."""
[   28s]         facts = self.device.get_facts()
[   28s] >       assert helpers.test_model(models.FactsDict, facts)
[   28s] E       AssertionError
[   28s] 
[   28s] /usr/lib/python3.8/site-packages/napalm/base/test/getters.py:154: AssertionError
[   28s] ____________________ TestGetter.test_get_interfaces[normal] ____________________
[   28s] 
[   28s] self = <test.unit.test_getters.TestGetter object at 0x7fe5c6bc0a00>
[   28s] test_case = 'normal'
[   28s] 
[   28s]     @wrap_test_cases
[   28s]     def test_get_interfaces(self, test_case):
[   28s]         """Test get_interfaces."""
[   28s]         get_interfaces = self.device.get_interfaces()
[   28s]         assert len(get_interfaces) > 0
[   28s]     
[   28s]         for interface, interface_data in get_interfaces.items():
[   28s] >           assert helpers.test_model(models.InterfaceDict, interface_data)
[   28s] E           AssertionError
[   28s] 
[   28s] /usr/lib/python3.8/site-packages/napalm/base/test/getters.py:164: AssertionError
[   28s] ______________________ TestGetter.test_traceroute[normal] ______________________
[   28s] 
[   28s] cls = <test.unit.test_getters.TestGetter object at 0x7fe5c6bea130>
[   28s] test_case = 'normal'
[   28s] 
[   28s]     @functools.wraps(func)
[   28s]     def mock_wrapper(cls, test_case):
[   28s]         for patched_attr in cls.device.patched_attrs:
[   28s]             attr = getattr(cls.device, patched_attr)
[   28s]             attr.current_test = func.__name__
[   28s]             attr.current_test_case = test_case
[   28s]     
[   28s]         try:
[   28s]             # This is an ugly, ugly, ugly hack because some python objects don't load
[   28s]             # as expected. For example, dicts where integers are strings
[   28s]             result = json.loads(json.dumps(func(cls, test_case)))
[   28s]         except IOError:
[   28s]             if test_case == "no_test_case_found":
[   28s]                 pytest.fail("No test case for '{}' found".format(func.__name__))
[   28s]             else:
[   28s]                 raise
[   28s]         except NotImplementedError:
[   28s]             pytest.skip("Method not implemented")
[   28s]             return
[   28s]     
[   28s]         # This is an ugly, ugly, ugly hack because some python objects don't load
[   28s]         # as expected. For example, dicts where integers are strings
[   28s]     
[   28s]         try:
[   28s]             expected_result = attr.expected_result
[   28s]         except IOError as e:
[   28s]             raise Exception("{}. Actual result was: {}".format(e, json.dumps(result)))
[   28s]         if isinstance(result, list):
[   28s]             diff = list_dicts_diff(result, expected_result)
[   28s]         else:
[   28s]             diff = dict_diff(result, expected_result)
[   28s]         if diff:
[   28s]             print("Resulting JSON object was: {}".format(json.dumps(result)))
[   28s] >           raise AssertionError(
[   28s]                 "Expected result varies on some keys {}".format(json.dumps(diff))
[   28s]             )
[   28s] E           AssertionError: Expected result varies on some keys {"success": {"5": {"probes": {"1": {"host_name": {"result": "", "expected": "dns.google"}}, "3": {"host_name": {"result": "", "expected": "dns.google"}}, "2": {"host_name": {"result": "", "expected": "dns.google"}}}}}}
[   28s] 
[   28s] /usr/lib/python3.8/site-packages/napalm/base/test/getters.py:83: AssertionError
[   28s] =============================== warnings summary ===============================
[   28s] ../../../../../usr/lib/python3.8/site-packages/_pytest/config/__init__.py:1252
[   28s]   /usr/lib/python3.8/site-packages/_pytest/config/__init__.py:1252: PytestConfigWarning: Unknown config option: json_report
[   28s]   
[   28s]     self._warn_or_fail_if_strict(f"Unknown config option: {key}\n")
[   28s] 
[   28s] ../../../../../usr/lib/python3.8/site-packages/_pytest/config/__init__.py:1252
[   28s]   /usr/lib/python3.8/site-packages/_pytest/config/__init__.py:1252: PytestConfigWarning: Unknown config option: jsonapi
[   28s]   
[   28s]     self._warn_or_fail_if_strict(f"Unknown config option: {key}\n")
[   28s] 
[   28s] -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
[   28s] 
[   28s] ---------- coverage: platform linux, python 3.8.16-final-0 -----------
[   28s] Name                                                  Stmts   Miss  Cover   Missing
[   28s] -----------------------------------------------------------------------------------
[   28s] napalm_arubaoss/ArubaOS.py                              141     29    79%   71-72, 75, 94-96, 116-118, 126-128, 138, 147, 155, 163-166, 504, 512-514, 529-533, 551-555, 569, 577-585, 620, 628, 636-638
[   28s] napalm_arubaoss/__init__.py                               2      0   100%
[   28s] napalm_arubaoss/helper/__init__.py                       24      0   100%
[   28s] napalm_arubaoss/helper/base.py                           97     68    30%   50-101, 105-114, 124-126, 136-138, 148-150, 160-162, 171-192, 205-227, 236-237
[   28s] napalm_arubaoss/helper/commit_config.py                  27     19    30%   26-60
[   28s] napalm_arubaoss/helper/compare_config.py                 22     17    23%   19-52
[   28s] napalm_arubaoss/helper/confirm_commit.py                 10      4    60%   19-23
[   28s] napalm_arubaoss/helper/get_arp_table.py                  16      0   100%
[   28s] napalm_arubaoss/helper/get_config.py                     13      2    85%   19-21
[   28s] napalm_arubaoss/helper/get_facts.py                      37      0   100%
[   28s] napalm_arubaoss/helper/get_interfaces.py                 43      5    88%   28-29, 32-33, 56
[   28s] napalm_arubaoss/helper/get_interfaces_ip.py              22      0   100%
[   28s] napalm_arubaoss/helper/get_lldp_neighbors.py             15      0   100%
[   28s] napalm_arubaoss/helper/get_lldp_neighbors_detail.py      15      0   100%
[   28s] napalm_arubaoss/helper/get_mac_address_table.py          12      0   100%
[   28s] napalm_arubaoss/helper/get_ntp_servers.py                16      0   100%
[   28s] napalm_arubaoss/helper/get_ntp_stats.py                  33      9    73%   29-38, 58, 74
[   28s] napalm_arubaoss/helper/get_route_to.py                   33      6    82%   68-83, 103
[   28s] napalm_arubaoss/helper/has_pending_commit.py              9      5    44%   18-24
[   28s] napalm_arubaoss/helper/is_alive.py                        9      5    44%   16-22
[   28s] napalm_arubaoss/helper/load_merge_candidate.py           13      8    38%   25-36
[   28s] napalm_arubaoss/helper/load_replace_candidate.py         14      9    36%   28-37
[   28s] napalm_arubaoss/helper/ping.py                           15      1    93%   25
[   28s] napalm_arubaoss/helper/rollback.py                       11      6    45%   18-25
[   28s] napalm_arubaoss/helper/traceroute.py                     23      1    96%   24
[   28s] napalm_arubaoss/helper/utils.py                          61     46    25%   19-20, 30-32, 43-62, 79-101, 112-122, 133-140
[   28s] -----------------------------------------------------------------------------------
[   28s] TOTAL                                                   733    240    67%
[   28s] 
[   28s] =========================== short test summary info ============================
[   28s] FAILED test/unit/test_getters.py::TestGetter::test_method_signatures - Assert...
[   28s] FAILED test/unit/test_getters.py::TestGetter::test_get_facts[normal] - Assert...
[   28s] FAILED test/unit/test_getters.py::TestGetter::test_get_facts[stack] - Asserti...
[   28s] FAILED test/unit/test_getters.py::TestGetter::test_get_interfaces[normal] - A...
[   28s] FAILED test/unit/test_getters.py::TestGetter::test_traceroute[normal] - Asser...
[   28s] ====== 5 failed, 18 passed, 15 skipped, 2 deselected, 2 warnings in 1.42s ======

Complete build log with all details of packages used and steps taken.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant