[Flaky Test]: TestStartCollector/all-components.yml – Condition never satisfied

[pchila]

Failing test case

TestStartCollector

Error message

Condition never satisfied - It seems that the collector doesn't start within 2 seconds

Build

https://buildkite.com/elastic/elastic-agent/builds/9164#018faa59-b090-4648-ac4e-b87006b789fb

OS

Mac

Stacktrace and notes

2024-05-24 14:07:32 CEST	=== FAIL: internal/pkg/otel TestStartCollector/all-components.yml (3.73s)
2024-05-24 14:07:32 CEST	2024-05-24T04:57:10.308-0700	info	[email protected]/service.go:102	Setting up own telemetry...
2024-05-24 14:07:32 CEST	2024-05-24T04:57:10.308-0700	info	[email protected]/telemetry.go:103	Serving metrics	{"address": ":8888", "level": "Normal"}
2024-05-24 14:07:32 CEST	2024-05-24T04:57:10.309-0700	info	[email protected]/extension.go:167	Development component. May change in the future.	{"kind": "extension", "name": "memory_limiter"}
2024-05-24 14:07:32 CEST	2024-05-24T04:57:10.309-0700	info	memorylimiter/memorylimiter.go:77	Memory limiter configured	{"kind": "extension", "name": "memory_limiter", "limit_mib": 100, "spike_limit_mib": 20, "check_interval": 5}
2024-05-24 14:07:32 CEST	2024-05-24T04:57:10.311-0700	info	[email protected]/exporter.go:275	Development component. May change in the future.	{"kind": "exporter", "data_type": "metrics", "name": "debug"}
2024-05-24 14:07:32 CEST	2024-05-24T04:57:10.316-0700	info	[email protected]/metrics.go:98	Metric filter configured	{"kind": "processor", "name": "filter", "pipeline": "metrics", "include match_type": "", "include expressions": [], "include metric names": [], "include metrics with resource attributes": null, "exclude match_type": "", "exclude expressions": [], "exclude metric names": [], "exclude metrics with resource attributes": null}
2024-05-24 14:07:32 CEST	2024-05-24T04:57:10.324-0700	info	[email protected]/exporter.go:275	Development component. May change in the future.	{"kind": "exporter", "data_type": "logs", "name": "debug"}
2024-05-24 14:07:32 CEST	2024-05-24T04:57:10.331-0700	info	[email protected]/exporter.go:275	Development component. May change in the future.	{"kind": "exporter", "data_type": "traces", "name": "debug"}
2024-05-24 14:07:32 CEST	2024-05-24T04:57:10.336-0700	info	[email protected]/traces.go:72	Span filter configured	{"kind": "processor", "name": "filter", "pipeline": "traces", "[Include] match_type": "[None]", "[Exclude] match_type": "[None]"}
2024-05-24 14:07:32 CEST	2024-05-24T04:57:10.338-0700	info	[email protected]/service.go:169	Starting /tmp/go-build3073199665/b1720/otel.test...	{"Version": "test", "NumCPU": 3}
2024-05-24 14:07:32 CEST	2024-05-24T04:57:10.338-0700	info	extensions/extensions.go:34	Starting extensions...
2024-05-24 14:07:32 CEST	2024-05-24T04:57:10.338-0700	info	extensions/extensions.go:37	Extension is starting...	{"kind": "extension", "name": "memory_limiter"}
2024-05-24 14:07:32 CEST	2024-05-24T04:57:10.338-0700	info	extensions/extensions.go:52	Extension started.	{"kind": "extension", "name": "memory_limiter"}
2024-05-24 14:07:32 CEST	    run_test.go:50:
2024-05-24 14:07:32 CEST	        	Error Trace:	/Users/admin/builds/bk-agent-prod-orka-1716549982697568545/elastic/elastic-agent/internal/pkg/otel/run_test.go:50
2024-05-24 14:07:32 CEST	        	Error:      	Condition never satisfied
2024-05-24 14:07:32 CEST	        	Test:       	TestStartCollector/all-components.yml
2024-05-24 14:07:32 CEST	2024-05-24T04:57:13.695-0700	warn	[email protected]/warning.go:42	Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks. Enable the feature gate to change the default and remove this warning.	{"kind": "receiver", "name": "otlp", "data_type": "metrics", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks", "feature gate ID": "component.UseLocalHostAsDefaultHost"}
2024-05-24 14:07:32 CEST	2024-05-24T04:57:13.701-0700	info	[email protected]/otlp.go:102	Starting GRPC server	{"kind": "receiver", "name": "otlp", "data_type": "metrics", "endpoint": "0.0.0.0:4317"}
2024-05-24 14:07:32 CEST	2024-05-24T04:57:13.766-0700	warn	[email protected]/warning.go:42	Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks. Enable the feature gate to change the default and remove this warning.	{"kind": "receiver", "name": "otlp", "data_type": "metrics", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks", "feature gate ID": "component.UseLocalHostAsDefaultHost"}
2024-05-24 14:07:32 CEST	2024-05-24T04:57:13.769-0700	info	[email protected]/otlp.go:152	Starting HTTP server	{"kind": "receiver", "name": "otlp", "data_type": "metrics", "endpoint": "0.0.0.0:4318"}
2024-05-24 14:07:32 CEST	2024-05-24T04:57:13.936-0700	info	adapter/receiver.go:46	Starting stanza receiver	{"kind": "receiver", "name": "filelog", "data_type": "logs"}
2024-05-24 14:07:32 CEST	2024-05-24T04:57:13.939-0700	warn	fileconsumer/file.go:43	finding files: no files match the configured criteria	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-05-24 14:07:32 CEST	2024-05-24T04:57:13.942-0700	info	[email protected]/service.go:195	Everything is ready. Begin running and processing data.
2024-05-24 14:07:32 CEST	2024-05-24T04:57:13.943-0700	warn	localhostgate/featuregate.go:63	The default endpoints for all servers in components will change to use localhost instead of 0.0.0.0 in a future version. Use the feature gate to preview the new default.	{"feature gate ID": "component.UseLocalHostAsDefaultHost"}
2024-05-24 14:07:32 CEST	2024-05-24T04:57:13.946-0700	info	[email protected]/collector.go:289	Received shutdown request
2024-05-24 14:07:32 CEST	2024-05-24T04:57:13.948-0700	info	[email protected]/service.go:232	Starting shutdown...
2024-05-24 14:07:32 CEST	2024-05-24T04:57:13.954-0700	info	adapter/receiver.go:141	Stopping stanza receiver	{"kind": "receiver", "name": "filelog", "data_type": "logs"}
2024-05-24 14:07:32 CEST	2024-05-24T04:57:13.991-0700	info	extensions/extensions.go:59	Stopping extensions...
2024-05-24 14:07:32 CEST	2024-05-24T04:57:13.992-0700	info	[email protected]/service.go:246	Shutdown complete.
2024-05-24 14:07:32 CEST	
2024-05-24 14:07:32 CEST	=== FAIL: internal/pkg/otel TestStartCollector (3.85s)

It seems that on a slow MacOS agent the otel collector takes more than 2 seconds to start. It should be enough to increase the timeout enough for a slow machine to start the collector reliably

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[andrzej-stencel]

Thanks @pchila for raising this. Here's the timeout increase, hopefully this will do it: #4815.