[Flaky Test]: Multiple tests – failed to fetched test output at ...

[rdner]

Failing test case

It's not a test case, the error happens after tests finished

Error message

Failed for instance windows-amd64-2022-fleet (@ 35.232.65.142): failed to execute tests on instance ogc-windows-amd64-2022-fleet-2486: error running sudo tests: failed to fetched test output at %home%\agent\build\TEST-go-remote-windows-amd64-2022-fleet-sudo.integration.out

Build

https://buildkite.com/elastic/elastic-agent/builds/7261#018dc1f1-d326-455c-bd41-fcad4fa2e0af

OS

Linux, Mac, Windows

Stacktrace and notes

>> (windows-amd64-2022-fleet) Failed for instance windows-amd64-2022-fleet (@ 35.232.65.142): failed to execute tests on instance ogc-windows-amd64-2022-fleet-2486: error running sudo tests: failed to fetched test output at %home%\agent\build\TEST-go-remote-windows-amd64-2022-fleet-sudo.integration.out

Looks like the SSH session stopped working since this code is failing:

elastic-agent/pkg/testing/runner/windows.go

Lines 311 to 322 in f23c279

	resultPkg.Output, err = sshClient.GetFileContents(ctx, outputPath+".out", WithContentFetchCommand("type"))
	if err != nil {
	return OSRunnerPackageResult{}, fmt.Errorf("failed to fetched test output at %s.out", outputPath)
	}
	resultPkg.JSONOutput, err = sshClient.GetFileContents(ctx, outputPath+".out.json", WithContentFetchCommand("type"))
	if err != nil {
	return OSRunnerPackageResult{}, fmt.Errorf("failed to fetched test output at %s.out.json", outputPath)
	}
	resultPkg.XMLOutput, err = sshClient.GetFileContents(ctx, outputPath+".xml", WithContentFetchCommand("type"))
	if err != nil {
	return OSRunnerPackageResult{}, fmt.Errorf("failed to fetched test output at %s.xml", outputPath)
	}

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[pazone]

I see it's passed after 2 retries. Shouldn't we add retries to GetFileContents in the ssh.go?

[rdner]

@pazone we can, but I think we should also identify why the connection to hosts was unreliable, it happened twice again in https://buildkite.com/elastic/elastic-agent/builds/7264#018dc323-a3fd-423d-bed4-75fcf4027ffc

Looks like the issue occurs periodically and lasts for a few hours.

[cmacknz]

I think we add retries here regardless in case it improves the situation. It also looks like we drop the underlying error from resultPkg.Output, err so we can't tell exactly what the failure is.

[cmacknz]

If we can catch this while the VM is still alive we can check what the instance is doing in GCP.

[rdner]

Same failure again in https://buildkite.com/elastic/elastic-agent/builds/7296#018dc731-ae08-4a86-86f8-0f25e6edde84

[rdner]

Closing in favor of #4356