Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[browser] Increase timeout and clean up after failed attempts of launching browser when testing #107865

Closed
wants to merge 34 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
a6d7f7a
Increase timeout + clean up before next attempt.
ilonatommy Sep 16, 2024
180fe0e
Prepare for cleanup to fail.
ilonatommy Sep 16, 2024
50b077a
Log more than just the message.
ilonatommy Sep 16, 2024
87fe62b
Update src/mono/wasm/Wasm.Build.Tests/BrowserRunner.cs
lewing Sep 16, 2024
197fe7d
Check if other chrome instances are running and forcefully close them.
ilonatommy Sep 17, 2024
9093136
Missing namespace
ilonatommy Sep 17, 2024
e8d3636
Merge branch 'main' into fix-107771
ilonatommy Sep 19, 2024
e7798a5
Merge branch 'main' into fix-107771
ilonatommy Sep 19, 2024
7c540c7
Merge branch 'main' into fix-107771
lewing Sep 21, 2024
9608d0c
Merge branch 'main' into fix-107771
ilonatommy Sep 23, 2024
d837c9f
Don't terminate if tests run in parallel.
ilonatommy Sep 23, 2024
a817b0c
Merge branch 'main' into fix-107771
ilonatommy Sep 26, 2024
91077dd
Elevate Playwright logs.
ilonatommy Sep 26, 2024
1b2b569
Merge error.
ilonatommy Sep 26, 2024
236b693
Merge branch 'main' into fix-107771
ilonatommy Sep 27, 2024
4d1dd6f
Bump playwright, previous was > 2 years old.
ilonatommy Sep 30, 2024
a26687d
Switch off debugging to restore failure observability.
ilonatommy Sep 30, 2024
6221f7e
Merge branch 'main' into fix-107771
ilonatommy Sep 30, 2024
414e7cd
Too new version, changing to the nearest available.
ilonatommy Sep 30, 2024
80cc9bf
Acoid "Permission denied" on node on CI.
ilonatommy Sep 30, 2024
76170dd
Fix condition for detecting unix.
ilonatommy Sep 30, 2024
312b537
Fix the syntax for OR.
ilonatommy Sep 30, 2024
fe3d5cb
Debug Agent.OS
ilonatommy Sep 30, 2024
40e18ea
More debugging
ilonatommy Oct 1, 2024
ea9853e
Azure condition is evaluated correctly, move it to bash that works fine.
ilonatommy Oct 1, 2024
415773b
Wrong path
ilonatommy Oct 1, 2024
25e088d
Fix the reason for failure: 'linux' changed to 'linux-x64'
ilonatommy Oct 1, 2024
54ccd98
Dump env variables.
ilonatommy Oct 1, 2024
4cec530
Go back to default timeout + give up on our own chrome.
ilonatommy Oct 1, 2024
5988c90
Nullable
ilonatommy Oct 1, 2024
2289c01
Trying to fix default chromium on linux
ilonatommy Oct 2, 2024
1fa0a83
Merge branch 'main' into fix-107771
ilonatommy Oct 2, 2024
ab4bcc6
Merge branch 'main' into fix-107771
ilonatommy Oct 2, 2024
0de13d2
Merge branch 'main' into fix-107771
ilonatommy Oct 3, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 53 additions & 9 deletions src/mono/wasm/Wasm.Build.Tests/BrowserRunner.cs
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,10 @@
using System;
using System.IO;
using System.Collections.Generic;
using System.Diagnostics;
using System.Text.RegularExpressions;
using System.Threading.Tasks;
using System.Runtime.InteropServices;
using Microsoft.Playwright;
using Wasm.Tests.Internal;
using Xunit.Abstractions;
Expand All @@ -19,6 +21,7 @@ internal class BrowserRunner : IAsyncDisposable
private static Regex s_blazorUrlRegex = new Regex("Now listening on: (?<url>https?://.*$)");
private static Regex s_appHostUrlRegex = new Regex("^App url: (?<url>https?://.*$)");
private static Regex s_exitRegex = new Regex("WASM EXIT (?<exitCode>-?[0-9]+)$");
private static bool s_runsOnWindows = RuntimeInformation.IsOSPlatform(OSPlatform.Windows);
private static readonly Lazy<string> s_chromePath = new(() =>
{
string artifactsBinDir = Path.Combine(Path.GetDirectoryName(typeof(BuildTestBase).Assembly.Location)!, "..", "..", "..", "..");
Expand Down Expand Up @@ -103,12 +106,11 @@ public async Task<string> StartServerAndGetUrlAsync(
public async Task<IBrowser> SpawnBrowserAsync(
string browserUrl,
bool headless = true,
int timeout = 10000,
int maxRetries = 3,
int? timeout = null,
int maxRetries = 3,
string language = "en-US"
) {
var url = new Uri(browserUrl);
Playwright = await Microsoft.Playwright.Playwright.CreateAsync();
// codespaces: ignore certificate error -> Microsoft.Playwright.PlaywrightException : net::ERR_CERT_AUTHORITY_INVALID
string[] chromeArgs = new[] { $"--explicitly-allowed-ports={url.Port}", "--ignore-certificate-errors", $"--lang={language}" };
_testOutput.WriteLine($"Launching chrome ('{s_chromePath.Value}') via playwright with args = {string.Join(',', chromeArgs)}");
Expand All @@ -118,8 +120,14 @@ public async Task<IBrowser> SpawnBrowserAsync(
{
try
{
// widnows runs are sequential, we cannot terminate when we run in parallel
if (s_runsOnWindows)
{
TerminateExistingChromeInstances();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we also terminate after each test ?
do we believe that chrome process survived on the agent from another run ? this is docker/VM right ?

Copy link
Member Author

@ilonatommy ilonatommy Sep 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't believe after is needed. The problem is with LaunchAsync, so the idea was to make sure no chrome is running directly before calling that method. And yes, the suspicion was that some leftovers from the previous test was left and that's why new browser launch is time outing.
This failure is only on CI, I have never managed to reproduce it locally.

}
Playwright = await Microsoft.Playwright.Playwright.CreateAsync();
Browser = await Playwright.Chromium.LaunchAsync(new BrowserTypeLaunchOptions {
ExecutablePath = s_chromePath.Value,
// ExecutablePath = s_chromePath.Value,
Headless = headless,
Args = chromeArgs,
Timeout = timeout
Expand All @@ -129,17 +137,53 @@ public async Task<IBrowser> SpawnBrowserAsync(
Browser = null;
_testOutput.WriteLine("Browser has been disconnected");
};
break;
return Browser!;
}
catch (System.TimeoutException ex)
{
attempt++;
_testOutput.WriteLine($"Attempt {attempt} failed with TimeoutException: {ex.Message}");
_testOutput.WriteLine($"Attempt {attempt} failed with TimeoutException: {ex}");

try
{
// Cleanup before retrying
if (Browser != null)
{
await Browser.CloseAsync();
Browser = null;
}
if (Playwright != null)
{
Playwright.Dispose();
Playwright = null;
}
}
catch(Exception cleanupException)
{
_testOutput.WriteLine($"Attempt to clean up failed with {cleanupException}");
}
}
}
throw new Exception($"Failed to launch browser after {maxRetries} attempts");
}

private void TerminateExistingChromeInstances()
{
try
{
var chromeProcesses = Process.GetProcessesByName("chrome");
_testOutput.WriteLine($"Found {chromeProcesses.Length} old Chrome processes.");
foreach (var process in chromeProcesses)
{
_testOutput.WriteLine($"Terminating existing Chrome process: {process.Id}");
process.Kill();
process.WaitForExit();
}
}
if (attempt == maxRetries)
throw new Exception($"Failed to launch browser after {maxRetries} attempts");
return Browser!;
catch (Exception ex)
{
_testOutput.WriteLine($"Failed to terminate existing Chrome instances: {ex}");
}
}

// FIXME: options
Expand Down
8 changes: 8 additions & 0 deletions src/mono/wasm/Wasm.Build.Tests/Wasm.Build.Tests.csproj
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,14 @@
<RunScriptCommands Condition="'$(OS)' != 'Windows_NT'" Include="export IS_RUNNING_ON_CI=true" />
<RunScriptCommands Condition="'$(OS)' == 'Windows_NT'" Include="set IS_RUNNING_ON_CI=true" />

<!-- avoids "Executable doesn't exist. Looks like Playwright was just installed or updated." -->
<RunScriptCommands Condition="'$(OS)' != 'Windows_NT'" Include="npx playwright install" />

<!-- prividing info for the issue in Playwright -->
<RunScriptCommands Include="npx envinfo --preset playwright" />

<!-- <RunScriptCommands Condition="'$(OS)' == 'Windows_NT'" Include="set DEBUG=pw:browser*" /> -->

<RunScriptCommands Condition="'$(OS)' != 'Windows_NT'" Include="chmod +x $HELIX_WORKITEM_ROOT/.playwright/node/linux-x64/playwright.sh" />
<RunScriptCommands Condition="'$(OS)' != 'Windows_NT'" Include="chmod +x $HELIX_WORKITEM_ROOT/.playwright/node/linux-x64/node" />
</ItemGroup>
Expand Down
Loading