Add error handling/retry logic for search source alert tests

[lukasolson]

Summary

Resolves #193842.

Adds error handling & retry logic for search source alerts that are causing failures on MKI.

Checklist

• Flaky Test Runner was used on any tests changed

[elasticmachine]

🤖 Jobs for this PR can be triggered through checkboxes. 🚧

ℹ️ To trigger the CI, please tick the checkbox below 👇

• Click to trigger kibana-pull-request for this PR!
• Click to trigger kibana-deploy-project-from-pr for this PR!

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 62a04a2

Metrics [docs]

✅ unchanged

History

• 💔 Build #242871 failed 4a0eefa

cc @lukasolson

[kibanamachine]

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#7157

[✅] x-pack/test/functional_with_es_ssl/apps/discover_ml_uptime/config.ts: 25/25 tests passed.
[✅] x-pack/test_serverless/functional/test_suites/observability/common_configs/config.group6.ts: 25/25 tests passed.
[✅] x-pack/test_serverless/functional/test_suites/search/common_configs/config.group6.ts: 25/25 tests passed.
[✅] x-pack/test_serverless/functional/test_suites/security/common_configs/config.group6.ts: 25/25 tests passed.

see run history

[lukasolson]

@wayneseymour Do you mind taking a quick look at this and see if it's along the lines of what you were thinking?

[wayneseymour]

@lukasolson I would strongly recommend running the serverless tests through MKI, from your local

One major reason is to ensure the default timeout durations encoded in x-pack/test_serverless/functional/test_suites/common/discover_ml_uptime/discover/search_source_alert.ts, actually work for real serverless as well.
PR CI only runs fake-serverless.
Running against real serverless (mki) will give you useful feedback.

• network access is typically slower, etc

Ran From my Local

Ran the stateful test, no issues.
Ran the serverless test against a real serverless oblt project (mki), I am seeing errors:

└-: Discover alerting
  └-> "before all" hook: beforeTestSuite.trigger in "Discover alerting"
  └-> "before all" hook in "Discover alerting"
  └-: Search source Alert
    └-> "before all" hook: beforeTestSuite.trigger for "should create an alert when there is no data view"
    └-> "before all" hook for "should create an alert when there is no data view"
            │ERROR timeout of 60000ms exceeded
            │      Waiting 15000 ms before the next attempt
    └-> should create an alert when there is no data view
      └-> "before each" hook: global before each for "should create an alert when there is no data view"
      └- ✓ pass  (23.1s)
    └-> should show time field validation error
      └-> "before each" hook: global before each for "should show time field validation error"
      └- ✓ pass  (14.8s)
    └-> should navigate to alert results via view in app link
      └-> "before each" hook: global before each for "should navigate to alert results via view in app link"
      └- ✓ pass  (40.0s)
    └-> should navigate to alert results via link provided in notification
      └-> "before each" hook: global before each for "should navigate to alert results via link provided in notification"
      └- ✓ pass  (37.8s)
    └-> should display prev rule state after params update on clicking prev generated link
      └-> "before each" hook: global before each for "should display prev rule state after params update on clicking prev generated link"
      └- ✖ fail: Discover alerting Search source Alert should display prev rule state after params update on clicking prev generated link
      │      Error: retry.try reached timeout 120000 ms
      │ TimeoutError: Waiting for element to be located By(css selector, [data-test-subj="saveEditedRuleButton"])
      │ Wait timed out after 10054ms
      │     at /Users/trezworkbox/dev/main/node_modules/selenium-webdriver/lib/webdriver.js:923:22
      │     at processTicksAndRejections (node:internal/process/task_queues:95:5)
      │       at onFailure (retry_for_success.ts:18:9)
      │       at retryForSuccess (retry_for_success.ts:85:7)
      │       at Proxy.try (retry.ts:52:12)
      │       at Proxy.clickByCssSelector (find.ts:420:5)
      │       at Proxy.click (test_subjects.ts:169:5)
      │       at Context.<anonymous> (search_source_alert.ts:509:7)
      │       at Object.apply (wrap_function.js:74:16)
      │
      │
    └-> should display actual state after rule params update on clicking viewInApp link
      └-> "before each" hook: global before each for "should display actual state after rule params update on clicking viewInApp link"
      └- ✖ fail: Discover alerting Search source Alert should display actual state after rule params update on clicking viewInApp link
      │      Error: expected '' to equal 'message:msg-1'
      │       at Assertion.assert (expect.js:100:11)
      │       at Assertion.apply (expect.js:227:8)
      │       at Function.equal (expect.js:531:15)
      │       at checkUpdatedRuleParamsState (search_source_alert.ts:343:31)
      │       at processTicksAndRejections (node:internal/process/task_queues:95:5)
      │       at Context.<anonymous> (search_source_alert.ts:522:7)
      │       at Object.apply (wrap_function.js:74:16)
      │
      │
    └-> should display prev data view state after update on clicking prev generated link
      └-> "before each" hook: global before each for "should display prev data view state after update on clicking prev generated link"
     
     ....
     

I recommend running it against search and security projects as well.