[Snappi][IXIA]: Trying to apply configuration with MACSEC traffic items of fixed duration leads to stuck window on "Applying traffic"

[amitpawar12]

Issue Description

This is an IXIA issue and seen trying to run the SONIC-Snappi IXIA testcases along with MACSEC protocol. This is seen with IxNetwork 10.80 version.

Trying to create a MACSEC configuration file with IxNetwork docker. The traffic item has 4 flows, two lossless flows and 2 lossy flows with pause being sent from receiver to the transmitter.

The flows are set to fixed duration. Pause flow is set to fixed duration with a start-delay.

On trying to apply configuration, the window is stuck forever. The configuration window never recovers. Only way out, is to refresh the browser and immediately click on 'IxNetwork' tab and terminate the session.

On further observation, there seems to be a dotnet process running on host server, which clocks more than 100% CPU utilization when the apply-configuration is running.

top - 14:24:30 up  4:16,  8 users,  load average: 4.94, 19.48, 27.75
Tasks: 6049 total,   3 running, 6046 sleeping,   0 stopped,   0 zombie
%Cpu(s):  5.0 us,  1.0 sy,  0.0 ni, 93.9 id,  0.0 wa,  0.0 hi,  0.1 si,  0.0 st
MiB Mem : 257660.3 total, 155062.7 free,  85488.0 used,  17109.6 buff/cache
MiB Swap:  16384.0 total,  16384.0 free,      0.0 used. 164040.2 avail Mem 
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                        
 3515 root      20   0   22.3g 820840 149980 S 199.0   0.3  44:34.29 dotnet 

ansible@sonic-xxxx/proc/3515$ ps -ef | grep 3515
root      3515  3510 99 13:57 ?        00:45:41 dotnet IxNetwork.Middleware.dll -allChassis -restTrace -restInsecure -username admin-6 -restport 9096 -shutdownfile /tmp/java_root/ixnrest_6.lock -readyfile /tmp/java_root/WafProcessReady/ixnrest_6.ready -logpath /var/log/ixia/apps/ixnetwork/logs/session_6 -sessionType wafgui -collectFeatureUsageData -useAPIServer

We have case opened with IXIA - #01619852 to track this issue.

@kamalsahu0001 - This is P0 issue.

Results you see

The apply configuration window is stuck forever, if we do - "Generate all", followed by "Apply". To avoid this, tried to simply start the traffic, skipping the generate-apply, since starting traffic should also initiate generate-apply. I see that this is getting stuck at the same point as well.

Results you expected to see

The generate-apply should have been easily finished and proceeded to start the traffic.

Is it platform specific

generic

Relevant log output

Output of show version

Attach files (if any)

[kamalsahu0001]

Hi Amit, Is it the same server which had kernel panic where this IxNetwork docker is running? If possible, can you remove/delete all the unused containers which are not running and not required? If you still see after that, we can have a debug session tomorrow morning. But feel it’s something with the server which is hosting these containers. Thanks, Kamal. From: Amit Pawar ***@***.***> Date: Thursday, January 30, 2025 at 11:28 AM To: sonic-net/sonic-mgmt ***@***.***> Cc: Kamal Sahu ***@***.***>, Mention ***@***.***> Subject: [sonic-net/sonic-mgmt] [Snappi][IXIA]: Trying to apply configuration with MACSEC traffic items of fixed duration leads to stuck window on "Applying traffic" (Issue #16735) Issue Description This is an IXIA issue and seen trying to run the SONIC-Snappi IXIA testcases along with MACSEC protocol. This is seen with IxNetwork 10. 80 version. Trying to create a MACSEC configuration file with IxNetwork docker. The traffic ZjQcmQRYFpfptBannerStart This Message is From an External Sender: Use caution opening files, clicking links or responding to requests. ZjQcmQRYFpfptBannerEnd Issue Description This is an IXIA issue and seen trying to run the SONIC-Snappi IXIA testcases along with MACSEC protocol. This is seen with IxNetwork 10.80 version. Trying to create a MACSEC configuration file with IxNetwork docker. The traffic item has 4 flows, two lossless flows and 2 lossy flows with pause being sent from receiver to the transmitter. The flows are set to fixed duration. Pause flow is set to fixed duration with a start-delay. On trying to apply configuration, the window is stuck forever. The configuration window never recovers. Only way out, is to refresh the browser and immediately click on 'IxNetwork' tab and terminate the session. On further observation, there seems to be a dotnet process running on host server, which clocks more than 100% CPU utilization when the apply-configuration is running. top - 14:24:30 up 4:16, 8 users, load average: 4.94, 19.48, 27.75 Tasks: 6049 total, 3 running, 6046 sleeping, 0 stopped, 0 zombie %Cpu(s): 5.0 us, 1.0 sy, 0.0 ni, 93.9 id, 0.0 wa, 0.0 hi, 0.1 si, 0.0 st MiB Mem : 257660.3 total, 155062.7 free, 85488.0 used, 17109.6 buff/cache MiB Swap: 16384.0 total, 16384.0 free, 0.0 used. 164040.2 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 3515 root 20 0 22.3g 820840 149980 S 199.0 0.3 44:34.29 dotnet ***@***.***/proc/3515$ ps -ef | grep 3515 root 3515 3510 99 13:57 ? 00:45:41 dotnet IxNetwork.Middleware.dll -allChassis -restTrace -restInsecure -username admin-6 -restport 9096 -shutdownfile /tmp/java_root/ixnrest_6.lock -readyfile /tmp/java_root/WafProcessReady/ixnrest_6.ready -logpath /var/log/ixia/apps/ixnetwork/logs/session_6 -sessionType wafgui -collectFeatureUsageData -useAPIServer We have case opened with IXIA - #01619852 to track this issue. @kamalsahu0001<https://urldefense.com/v3/__https:/github.com/kamalsahu0001__;!!I5pVk4LIGAfnvw!jplPpuOc--SBnJzK1QrraVREi64E2g6SFfNZyao0NhzrWmG1QGmHqoIejsWjMd_bKgcCuAprf_ZB9UcC_xPeuuUjLZE$> - This is P0 issue. Results you see The apply configuration window is stuck forever, if we do - "Generate all", followed by "Apply". To avoid this, tried to simply start the traffic, skipping the generate-apply, since starting traffic should also initiate generate-apply. I see that this is getting stuck at the same point as well. Results you expected to see The generate-apply should have been easily finished and proceeded to start the traffic. Is it platform specific generic Relevant log output Output of show version Attach files (if any) MACSEC-configuration-stuck-on-apply.JPG (view on web)<https://urldefense.com/v3/__https:/github.com/user-attachments/assets/fe5c24ae-839a-43f3-8cc0-c9a8b06ffeb2__;!!I5pVk4LIGAfnvw!jplPpuOc--SBnJzK1QrraVREi64E2g6SFfNZyao0NhzrWmG1QGmHqoIejsWjMd_bKgcCuAprf_ZB9UcC_xPeo7qqSJ8$> — Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https:/github.com/sonic-net/sonic-mgmt/issues/16735__;!!I5pVk4LIGAfnvw!jplPpuOc--SBnJzK1QrraVREi64E2g6SFfNZyao0NhzrWmG1QGmHqoIejsWjMd_bKgcCuAprf_ZB9UcC_xPeNihB2Bc$>, or unsubscribe<https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/ANUC6S6MTUCHW5O4PKRLXBD2NJ4OZAVCNFSM6AAAAABWGAFBHSVHI2DSMVQWIX3LMV43ASLTON2WKOZSHAZDCNRUGU3DKNQ__;!!I5pVk4LIGAfnvw!jplPpuOc--SBnJzK1QrraVREi64E2g6SFfNZyao0NhzrWmG1QGmHqoIejsWjMd_bKgcCuAprf_ZB9UcC_xPeaNBl8KQ$>. You are receiving this because you were mentioned.Message ID: ***@***.***>