[Flaky Test] VM orchestration is unstable in integration tests

[rdner]

The failures can be categorized in following groups:

Firewall resource not found or already exists (quite often) (should be fixed by #4740)

This has been reported in the OGC repository adam-stokes/ogc#28

libcloud.common.google.ResourceNotFoundError: {'message': "The resource 'projects/elastic-platform-ingest/global/firewalls/linux-amd64-ubuntu-2204-upgrade' was not found", 'domain': 'global', 'reason': 'notFound'}

libcloud.common.google.ResourceExistsError: {'message': "The resource 'projects/elastic-platform-ingest/zones/us-central1-a/instances/ogc-linux-amd64-ubuntu-2204-fleet-airgapped-2315' already exists", 'domain': 'global', 'reason': 'alreadyExists'}

Examples:

• https://buildkite.com/elastic/elastic-agent/builds/7992#018e803c-886f-44d4-b614-16c750c44c5e
• https://buildkite.com/elastic/elastic-agent/builds/7989#018e801f-3723-472a-865c-f1c7531e7003
• https://buildkite.com/elastic/elastic-agent/builds/7481#018de444-4124-4dcb-984b-cea4cf33943e
• https://buildkite.com/elastic/elastic-agent/builds/7538#018e0db4-e88e-4c37-8095-8317ca15f411
• https://buildkite.com/elastic/elastic-agent/builds/7661#018e25ab-1fc6-4003-8550-d841d8b1512b
• https://buildkite.com/elastic/elastic-agent/builds/8172#018ebc63-5d61-44d4-994f-a768bd49a0e0
• https://buildkite.com/elastic/elastic-agent/builds/8258#018ec37e-0ee8-4273-aa1e-e341ab3d567c
• https://buildkite.com/elastic/elastic-agent/builds/8342#018ee242-7f8a-4902-b077-8759b47b3b9c
• https://buildkite.com/elastic/elastic-agent/builds/8351#018ee2c3-fea5-4086-a6ed-a071e0d5f9ee
• https://buildkite.com/elastic/elastic-agent/builds/8344#018ee62b-819d-453c-ad10-c623c3aa848c
• https://buildkite.com/elastic/elastic-agent/builds/8440#018ef10e-0d1f-4671-bf7f-cc9e2718037b
• https://buildkite.com/elastic/elastic-agent/builds/8511#018f1038-7e83-412c-b622-2ced349d7dfa

I believe it might be some kind of race condition, we should investigate further.

Networking issues

Tracked by #4794

Permission errors (serverless)

Error: error running clean: got unexpected response code [403] from deployment shutdown API: {
   "errors": [
       {
           "message": "To access the resource [u:/deployments/cc41c0a61a474f3aa6d890df111925d5], the user must have the required authorization.",
           "code": "root.permission_denied"
       }
   ]
}

Examples:

• https://buildkite.com/elastic/elastic-agent/builds/8267#018ec54f-d7cf-4c6a-9db6-1b87552472da
• https://buildkite.com/elastic/elastic-agent/builds/8268#018ec566-5553-4582-a52a-b13beb3d2654
• https://buildkite.com/elastic/elastic-agent/builds/8286#018eca76-50f3-4bd7-94db-8e05556dcdbb

SQL error

sqlite3.OperationalError: no such table: layouts

Examples:

• https://buildkite.com/elastic/elastic-agent/builds/8265#018ec498-b906-4c0d-ae50-2a5aa1d4f3d6
• https://buildkite.com/elastic/elastic-agent/builds/8268#018ec566-5551-4b6c-b4d7-9b5669482f35

GCP just fails with 500 (rare)

libcloud.common.google.GoogleBaseError: {'message': "Internal error. Please try again or contact Google Support. (Code: '')", 'domain': 'global', 'reason': 'backendError'}

Examples:

• https://buildkite.com/elastic/elastic-agent/builds/8091#018ea9bd-ee66-4a10-b1bc-b8f8030d80bc
• https://buildkite.com/elastic/elastic-agent/builds/8409#018eed67-8594-43f5-802c-0e4b61dc5a7e

Job did not complete in 180 seconds

libcloud.common.types.LibcloudError: <LibcloudError in None 'Job did not complete in 180 seconds'>

Examples:

• https://buildkite.com/elastic/elastic-agent/builds/7601#018e1638-25cd-4803-8d5e-d7f647879785

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[blakerouse]

Another OGC failure https://buildkite.com/elastic/elastic-agent/builds/7651#018e1f0f-c712-4721-baf7-f13f8ba8477e

Error: error running test: failed to prepare instance ogc-windows-amd64-2022-fleet-e3f0: failed to install curl: could not run "choco install -y curl" though SSH: Process exited with status 1 (stdout: , stderr: 'choco' is not recognized as an internal or external command, operable program or batch file.

Not OGC's fault, that is the integration testing framework preparing the instance. OGC doesn't do that.

[blakerouse]

And another one https://buildkite.com/elastic/elastic-agent/builds/7654#018e1f49-c387-4973-b8a0-dd10dba598f2

Error: error running test: failed to check for cloud 8.13.0-SNAPSHOT [stack_id: 8130-SNAPSHOT, deployment_id: <REDACTED>] to be ready: error calling deployment retrieval API: Get "https://cloud.elastic.co/api/v1/deployments/<REDACTED>": context deadline exceeded

Not OGC, OGC doesn't create or prepare any stack.

[blakerouse]

Another OGC-related failure https://buildkite.com/elastic/elastic-agent/builds/7744#018e36cf-934b-4a2f-aaca-de800860be5e

Failed to execute tests on instance: error running sudo tests: failed to fetched test output at %home%\agent\build\TEST-go-remote-windows-amd64-2022-upgrade-sudo.integration.out

Not OGC, OGC doesn't run the tests or fetch the results.

[blakerouse]

Just to be clear, OGC only creates the instance with the cloud providers nothing else. Everything else is done by the integration testing framework and is our code.

[rdner]

@blakerouse would "VM orchestration" be a better term? I will rename this issue then.

[rdner]

When it comes to OGC failures, sometimes we have something like this:

https://buildkite.com/elastic/elastic-agent/builds/8091#018ea9bd-ee66-4a10-b1bc-b8f8030d80bc

libcloud.common.google.GoogleBaseError: {'message': "Internal error. Please try again or contact Google Support. (Code: '')", 'domain': 'global', 'reason': 'backendError'}

Not sure we can do anything about it.

[pierrehilbert]

For this one, yeah not sure we can do anything.

[rdner]

I updated the description to organize known failures by categories and clean up my comments on this issue.

[belimawr]

I believe this is a new VM orchestration issue:

Error: error running test: failed to connect to instance ogc-linux-amd64-ubuntu-2204-default-f3e5: error NewClientConn for ssh to "34.41.144.218:22" :ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

https://buildkite.com/elastic/elastic-agent/builds/8793#018f58b8-f8c9-4c73-9b40-6f2da4a73974

It is from a backport PR: #4709, I'll try re-running it.

[rdner]

I moved all the failures that we can actually recover from to #4794

Since we have not had new errors for a while now and there is nothing new to report here, I'm closing this issue in favor of the new one.