Fix flaky `TestFQDN` #3097

ycombinator · 2023-07-18T23:10:18Z

What does this PR do?

This PR tries to fix the flaky TestFQDN end-to-end test. It also adds documentation on using test namespaces to avoid flaky tests.

Why is it important?

So we don't have flaky tests.

mergify · 2023-07-18T23:10:54Z

This pull request does not have a backport label. Could you fix it @ycombinator? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v./d./d./d is the label to automatically backport to the 8./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

elasticmachine · 2023-07-18T23:32:43Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-07-20T00:39:19.528+0000
Duration: 32 min 34 sec

Test stats 🧪

Test	Results
Failed	0
Passed	5939
Skipped	27
Total	5966

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages.
run integration tests : Run the Elastic Agent Integration tests.
run end-to-end tests : Generate the packages and run the E2E Tests.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2023-07-18T23:32:52Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	98.718% (`77/78`)	👍
Files	67.416% (`180/267`)	👍
Classes	66.329% (`327/493`)	👍
Methods	53.642% (`1031/1922`)	👍
Lines	39.907% (`11779/29516`)	👎 -0.061
Conditionals	100.0% (`0/0`)	💚

ycombinator · 2023-07-19T00:00:53Z

Buildkite CI (build 1840) failed on TestEnrollAndLog but TestFQDN passed (1 pass out of 1 attempt). Re-running...

ycombinator · 2023-07-19T00:02:16Z

buildkite test this

ycombinator · 2023-07-19T00:54:47Z

Again, buildkite CI (build 1844) failed on TestEnrollAndLog but TestFQDN passed (2 passes out of 2 attempts). Re-running...

ycombinator · 2023-07-19T00:54:56Z

buildkite test this

ycombinator · 2023-07-19T01:40:36Z

This time buildkite CI (build 1845) failed on TestFQDN but the failure seems ephemeral, having to do with Agent enrollment:

=== RUN   TestFQDN
    fqdn_test.go:75: Set FQDN on host to luxbjo.baz.io
    fqdn_test.go:79: Enroll agent in Fleet with a test policy
    fqdn_test.go:95: Creating enrollment API key...
    fqdn_test.go:95: Unpacking and installing Elastic Agent
    fetcher.go:90: Using existing artifact elastic-agent-8.10.0-SNAPSHOT-linux-x86_64.tar.gz
    fixture.go:198: Extracting artifact elastic-agent-8.10.0-SNAPSHOT-linux-x86_64.tar.gz to /tmp/TestFQDN1512827491/001
    fixture.go:211: Completed extraction of artifact elastic-agent-8.10.0-SNAPSHOT-linux-x86_64.tar.gz to /tmp/TestFQDN1512827491/001
    fixture.go:425: Components were not modified from the fetched artifact
    fixture.go:342: >> running agent with: [/tmp/TestFQDN1512827491/001/elastic-agent-8.10.0-SNAPSHOT-linux-x86_64/elastic-agent install --force --non-interactive --url https://0d2825bf0abb421cb776abdeaf733296.fleet.us-central1.gcp.qa.cld.elstc.co:443 --enrollment-token MWt1N2E0a0JXVHJUV3pxOFZ4LUg6TGpTRTJWZWZReldzS2NOZlRzcFJQZw==]
    fqdn_test.go:95: Installing in non-interactive mode.{"log.level":"info","@timestamp":"2023-07-19T01:20:23.936Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":478},"message":"Starting enrollment to URL: https://0d2825bf0abb421cb776abdeaf733296.fleet.us-central1.gcp.qa.cld.elstc.co:443/","ecs.version":"1.6.0"}
        Error: fail to enroll: could not save enrollment information: failed to ensure key: could not get agent key: cipher: message authentication failed
        For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.10/fleet-troubleshooting.html
        Error: enroll command failed with exit code: 1
        For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.10/fleet-troubleshooting.html
        
    fqdn_test.go:96: 
        	Error Trace:	/home/ubuntu/agent/testing/integration/fqdn_test.go:96
        	Error:      	Received unexpected error:
        	            	unable to enroll Elastic Agent: error running agent install command: exit status 1
        	Test:       	TestFQDN
    fqdn_test.go:58: Un-enrolling Elastic Agent...
    fqdn_test.go:61: Restoring hostname...
    fqdn_test.go:65: Restoring original /etc/hosts...
--- FAIL: TestFQDN (48.29s)

This is not how this test has failed in the past when being flaky. It has failed on verifying the hostname in logs-* and metrics-* indices. So I don't think we should count this failure here as the flaky test failure this PR is trying to fix. Re-running...

ycombinator · 2023-07-19T01:41:42Z

buildkite test this

ycombinator · 2023-07-19T05:35:35Z

Buildkite CI (build 1847) passed. Which means TestFQDN passed (3 passes out of 3 attempts). Running again...

ycombinator · 2023-07-19T05:35:40Z

buildkite test this

ycombinator · 2023-07-19T06:35:48Z

Buildkite CI (build 1848) passed again. Which means TestFQDN passed (4 passes out of 4 attempts). Running again...

ycombinator · 2023-07-19T06:36:04Z

buildkite test this

ycombinator · 2023-07-19T13:06:09Z

Buildkite CI (build 1852) passed again. Which means TestFQDN passed (5 passes out of 5 attempts). Running again...

ycombinator · 2023-07-19T13:06:16Z

buildkite test this

ycombinator · 2023-07-19T14:11:24Z

Buildkite CI (build 1859) passed again. Which means TestFQDN passed (6 passes out of 6 attempts). Running again...

ycombinator · 2023-07-19T14:11:31Z

buildkite test this

ycombinator · 2023-07-19T18:32:24Z

Buildkite CI (build 1866) passed again. Which means TestFQDN passed (7 passes out of 7 attempts). Running again...

ycombinator · 2023-07-19T18:32:29Z

buildkite test this

ycombinator · 2023-07-19T23:13:15Z

Buildkite CI (build 1873) passed again. Which means TestFQDN passed (8 passes out of 8 attempts). Running again...

Aiming for 10 attempts total, in case people are wondering how many episodes are in this saga.

ycombinator · 2023-07-19T23:13:23Z

/test

ycombinator · 2023-07-20T00:38:59Z

Buildkite CI (build 1882) passed again. Which means TestFQDN passed (9 passes out of 9 attempts). Running again... hopefully one final time 🤞.

ycombinator · 2023-07-20T00:39:03Z

/test

ycombinator · 2023-07-20T03:58:06Z

Buildkite CI (build 1883) failed on TestEnrollAndLog but TestFQDN passed (10 passes out of 10 attempts).

I think the fix in this PR is good to remove the flakiness in TestFQDN. Putting PR into review.

elasticmachine · 2023-07-20T03:58:12Z

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

pierrehilbert · 2023-07-20T05:41:04Z

Thanks @ycombinator for the time you spent on this.
Let's merge it when the last integration run will be successful

* Pass namespace to FQDN ES query * Add documentation on namespace usage * Add warning (cherry picked from commit d2162bb) # Conflicts: # docs/test-framework-dev-guide.md

* [WIP] Fix flaky `TestFQDN` (#3097) * Pass namespace to FQDN ES query * Add documentation on namespace usage * Add warning (cherry picked from commit d2162bb) # Conflicts: # docs/test-framework-dev-guide.md * Update test-framework-dev-guide.md --------- Co-authored-by: Shaunak Kashyap <[email protected]> Co-authored-by: Pierre HILBERT <[email protected]>

Pass namespace to FQDN ES query

e6dda34

mergify bot assigned ycombinator Jul 18, 2023

mergify bot added the backport-skip label Jul 18, 2023

ycombinator added skip-changelog backport-v8.9.0 Automated backport with mergify Testing and removed backport-skip labels Jul 19, 2023

ycombinator added 2 commits July 19, 2023 16:49

Add documentation on namespace usage

59495fb

Add warning

e303015

ycombinator added the Team:Elastic-Agent Label for the Agent team label Jul 19, 2023

ycombinator marked this pull request as ready for review July 20, 2023 03:58

ycombinator requested a review from a team as a code owner July 20, 2023 03:58

ycombinator requested review from blakerouse and pchila July 20, 2023 03:58

jlind23 approved these changes Jul 20, 2023

View reviewed changes

jlind23 enabled auto-merge (squash) July 20, 2023 05:33

pierrehilbert approved these changes Jul 20, 2023

View reviewed changes

pierrehilbert changed the title ~~[WIP] Fix flaky TestFQDN~~ Fix flaky TestFQDN Jul 20, 2023

pierrehilbert mentioned this pull request Jul 20, 2023

[Upgrade Watcher] Fix error messages #3093

Merged

jlind23 merged commit d2162bb into elastic:main Jul 20, 2023
10 checks passed

mergify bot pushed a commit that referenced this pull request Jul 20, 2023

[WIP] Fix flaky TestFQDN (#3097)

0770d72

* Pass namespace to FQDN ES query * Add documentation on namespace usage * Add warning (cherry picked from commit d2162bb) # Conflicts: # docs/test-framework-dev-guide.md

mergify bot mentioned this pull request Jul 20, 2023

[8.9](backport #3097) Fix flaky TestFQDN #3105

Merged

ycombinator deleted the fix-it-test-fqdn branch July 20, 2023 11:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix flaky `TestFQDN` #3097

Fix flaky `TestFQDN` #3097

ycombinator commented Jul 18, 2023 •

edited

Loading

mergify bot commented Jul 18, 2023

elasticmachine commented Jul 18, 2023 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented Jul 18, 2023 •

edited

Loading

ycombinator commented Jul 19, 2023 •

edited

Loading

ycombinator commented Jul 19, 2023

ycombinator commented Jul 19, 2023 •

edited

Loading

ycombinator commented Jul 19, 2023

ycombinator commented Jul 19, 2023 •

edited

Loading

ycombinator commented Jul 19, 2023

ycombinator commented Jul 19, 2023 •

edited

Loading

ycombinator commented Jul 19, 2023

ycombinator commented Jul 19, 2023 •

edited

Loading

ycombinator commented Jul 19, 2023

ycombinator commented Jul 19, 2023 •

edited

Loading

ycombinator commented Jul 19, 2023

ycombinator commented Jul 19, 2023 •

edited

Loading

ycombinator commented Jul 19, 2023

ycombinator commented Jul 19, 2023

ycombinator commented Jul 19, 2023

ycombinator commented Jul 19, 2023 •

edited

Loading

ycombinator commented Jul 19, 2023

ycombinator commented Jul 20, 2023

ycombinator commented Jul 20, 2023

ycombinator commented Jul 20, 2023

elasticmachine commented Jul 20, 2023

pierrehilbert commented Jul 20, 2023

Fix flaky TestFQDN #3097

Fix flaky TestFQDN #3097

Conversation

ycombinator commented Jul 18, 2023 • edited Loading

What does this PR do?

Why is it important?

mergify bot commented Jul 18, 2023

elasticmachine commented Jul 18, 2023 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

elasticmachine commented Jul 18, 2023 • edited Loading

🌐 Coverage report

ycombinator commented Jul 19, 2023 • edited Loading

ycombinator commented Jul 19, 2023

ycombinator commented Jul 19, 2023 • edited Loading

ycombinator commented Jul 19, 2023

ycombinator commented Jul 19, 2023 • edited Loading

ycombinator commented Jul 19, 2023

ycombinator commented Jul 19, 2023 • edited Loading

ycombinator commented Jul 19, 2023

ycombinator commented Jul 19, 2023 • edited Loading

ycombinator commented Jul 19, 2023

ycombinator commented Jul 19, 2023 • edited Loading

ycombinator commented Jul 19, 2023

ycombinator commented Jul 19, 2023 • edited Loading

ycombinator commented Jul 19, 2023

ycombinator commented Jul 19, 2023

ycombinator commented Jul 19, 2023

ycombinator commented Jul 19, 2023 • edited Loading

ycombinator commented Jul 19, 2023

ycombinator commented Jul 20, 2023

ycombinator commented Jul 20, 2023

ycombinator commented Jul 20, 2023

elasticmachine commented Jul 20, 2023

pierrehilbert commented Jul 20, 2023

Fix flaky `TestFQDN` #3097

Fix flaky `TestFQDN` #3097

ycombinator commented Jul 18, 2023 •

edited

Loading

elasticmachine commented Jul 18, 2023 •

edited

Loading

elasticmachine commented Jul 18, 2023 •

edited

Loading

ycombinator commented Jul 19, 2023 •

edited

Loading

ycombinator commented Jul 19, 2023 •

edited

Loading

ycombinator commented Jul 19, 2023 •

edited

Loading

ycombinator commented Jul 19, 2023 •

edited

Loading

ycombinator commented Jul 19, 2023 •

edited

Loading

ycombinator commented Jul 19, 2023 •

edited

Loading

ycombinator commented Jul 19, 2023 •

edited

Loading

ycombinator commented Jul 19, 2023 •

edited

Loading