Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: TypeError seen during stress/test_stress_routes.py #16801

Open
kaushiksrivatsav opened this issue Feb 5, 2025 · 1 comment
Open

[Bug]: TypeError seen during stress/test_stress_routes.py #16801

kaushiksrivatsav opened this issue Feb 5, 2025 · 1 comment

Comments

@kaushiksrivatsav
Copy link

Issue Description

TypeError seen during stress/test_stress_routes.py

Results you see

duthost = , daemon_list = ['bgpd', 'zebra']

def get_frr_daemon_memory_usage(duthost, daemon_list):
    frr_daemon_memory_dict = {}
    for daemon in daemon_list:
        frr_daemon_memory_output = duthost.shell(f'vtysh -c "show memory {daemon}"')["stdout"]
        logging.info(f"{daemon} memory status: \n%s", frr_daemon_memory_output)
        output = duthost.shell(f'vtysh -c "show memory {daemon}" | grep "Free ordinary blocks"')["stdout"]
        frr_daemon_memory = output.split()[-2]
        unit = output.split()[-1]
        if unit == "KiB":
          frr_daemon_memory = frr_daemon_memory / 1000

E TypeError: unsupported operand type(s) for /: 'str' and 'int'

Results you expected to see

test case to pass no error

Is it platform specific

generic

Relevant log output

duthosts = [<MultiAsicSonicHost DUT-1>], localhost = <tests.common.devices.local.Localhost object at 0x7fc63f9c03a0>
tbinfo = {'auto_recover': 'True', 'comment': 'Tests Dell S6000 vms', 'conf-name': 'ptf6-m', 'duts': ['DUT-1'], ...}
get_function_completeness_level = None, withdraw_and_announce_existing_routes = (22, 24)
loganalyzer = {'DUT-1': <tests.common.plugins.loganalyzer.loganalyzer.LogAnalyzer object at 0x7fc63f9a1fa0>}
enum_rand_one_per_hwsku_frontend_hostname = 'DUT-1', enum_rand_one_frontend_asic_index = None, rotate_syslog = None

    def test_announce_withdraw_route(duthosts, localhost, tbinfo, get_function_completeness_level,
                                     withdraw_and_announce_existing_routes, loganalyzer,
                                     enum_rand_one_per_hwsku_frontend_hostname, enum_rand_one_frontend_asic_index,
                                     rotate_syslog):
        ptf_ip = tbinfo["ptf_ip"]
        topo_name = tbinfo["topo"]["name"]
        duthost = duthosts[enum_rand_one_per_hwsku_frontend_hostname]
        asichost = duthost.asic_instance(enum_rand_one_frontend_asic_index)
        namespace = asichost.namespace

        ignoreRegex = [
            ".*ERR route_check.py:.*",
            ".*ERR.* 'routeCheck' status failed.*",
            ".*Process \'orchagent\' is stuck in namespace \'host\'.*",
            ".*ERR rsyslogd: .*"
        ]

        hwsku = duthost.facts['hwsku']
        if hwsku in ['Arista-7050-QX-32S', 'Arista-7050QX32S-Q32', 'Arista-7050-QX32', 'Arista-7050QX-32S-S4Q31']:
            ignoreRegex.append(".*ERR memory_threshold_check:.*")
            ignoreRegex.append(".*ERR monit.*memory_check.*")
            ignoreRegex.append(".*ERR monit.*mem usage of.*matches resource limit.*")

        # Ignore errors in ignoreRegex for *all* DUTs
        for dut in duthosts.frontend_nodes:
            if dut.loganalyzer:
                loganalyzer[dut.hostname].ignore_regex.extend(ignoreRegex)

        normalized_level = get_function_completeness_level
        if normalized_level is None:
            normalized_level = "debug"

        ipv4_route_used_before, ipv6_route_used_before = withdraw_and_announce_existing_routes

        loop_times = LOOP_TIMES_LEVEL_MAP[normalized_level]

        frr_demons_to_check = ['bgpd', 'zebra']
>       start_time_frr_daemon_memory = get_frr_daemon_memory_usage(duthost, frr_demons_to_check)

asichost   = <SonicAsic 0>
dut        = <MultiAsicSonicHost DUT-1>
duthost    = <MultiAsicSonicHost DUT-1>
duthosts   = [<MultiAsicSonicHost DUT-1>]
enum_rand_one_frontend_asic_index = None
enum_rand_one_per_hwsku_frontend_hostname = 'DUT-1'
frr_demons_to_check = ['bgpd', 'zebra']
get_function_completeness_level = None
hwsku      = 'rd98DX35xx_cn9131'
ignoreRegex = ['.*ERR route_check.py:.*', ".*ERR.* 'routeCheck' status failed.*", ".*Process 'orchagent' is stuck in namespace 'host'.*", '.*ERR rsyslogd: .*']
ipv4_route_used_before = 22
ipv6_route_used_before = 24
localhost  = <tests.common.devices.local.Localhost object at 0x7fc63f9c03a0>
loganalyzer = {'DUT-1': <tests.common.plugins.loganalyzer.loganalyzer.LogAnalyzer object at 0x7fc63f9a1fa0>}
loop_times = 1
namespace  = None
normalized_level = 'debug'
ptf_ip     = '10.28.32.144'
rotate_syslog = None
tbinfo     = {'auto_recover': 'True', 'comment': 'Tests Dell S6000 vms', 'conf-name': 'ptf6-m', 'duts': ['DUT-1'], ...}
topo_name  = 'm0'
withdraw_and_announce_existing_routes = (22, 24)

stress/test_stress_routes.py:80:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

duthost = <MultiAsicSonicHost DUT-1>, daemon_list = ['bgpd', 'zebra']

    def get_frr_daemon_memory_usage(duthost, daemon_list):
        frr_daemon_memory_dict = {}
        for daemon in daemon_list:
            frr_daemon_memory_output = duthost.shell(f'vtysh -c "show memory {daemon}"')["stdout"]
            logging.info(f"{daemon} memory status: \n%s", frr_daemon_memory_output)
            output = duthost.shell(f'vtysh -c "show memory {daemon}" | grep "Free ordinary blocks"')["stdout"]
            frr_daemon_memory = output.split()[-2]
            unit = output.split()[-1]
            if unit == "KiB":
>               frr_daemon_memory = frr_daemon_memory / 1000
E               TypeError: unsupported operand type(s) for /: 'str' and 'int'

daemon     = 'bgpd'
daemon_list = ['bgpd', 'zebra']
duthost    = <MultiAsicSonicHost DUT-1>
frr_daemon_memory = '6368'
frr_daemon_memory_dict = {}
frr_daemon_memory_output = "System allocator statistics:\n  Total heap allocated:  17 MiB\n  Holding block headers: 19 MiB\n  Used small blocks: ...    296         296        1       296\nRFAPI Import Table            :        1    208         216        1       216"
output     = '  Free ordinary blocks:  6368 KiB'
unit       = 'KiB'

stress/test_stress_routes.py:134: TypeError

Output of show version

SONiC Software Version: SONiC.202411.0-dirty-20250203.183031
SONiC OS Version: 12
Distribution: Debian 12.9
Kernel: 6.1.0-22-2-arm64
Build commit: bed58c457
Build date: Mon Feb  3 18:57:03 UTC 2025
Built by: marvell@cpss-arm-vm04

Platform: arm64-marvell_rd98DX35xx_cn9131-r0
HwSKU: rd98DX35xx_cn9131
ASIC: marvell
ASIC Count: 1
Serial Number: NA
Model Number: NA
Hardware Revision: N/A
Uptime: 08:09:32 up 13:12,  2 users,  load average: 0.98, 3.69, 3.21
Date: Wed 05 Feb 2025 08:09:32

Attach files (if any)

No response

@bingwang-ms
Copy link
Collaborator

Seems a regression caused by PR #14240
@nhe-NV Can you help take a look?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants