[Detection Engine][FTRs] Break down long running FTR groups and fix threshold preview bug #197368

yctercero · 2024-10-23T06:54:15Z

Summary

This PR breaks down long running FTR groups into smaller chunks that now run in <~15 min.

Addresses [Serverless][Kibana QA QG] exception_operators_date_numeric_types:essentials:qa:serverless:release tests optimization #192109
There is no existing ticket but rule_execution group tests are taking ~55m to run and will soon be a bottle neck for us.
No edits made to any existing tests.
Purely just a reshuffle of the tests.

Flakey test runner does not run MKI environment. I ran that manually on my local.

Important

When isolating the tests into different groups, found a bug where running rule preview on a Threshold rule prior
to the .preview-alerts index being created fails. Because there is no execution history, threshold rule attempts
to recreate it by searching alerts index (which is not yet created). Spoke with @marshallmain who pointed me to
the offending code and suggested a fix of adding ignore_unavailable.

See #198209 for details on bug.

Running flakey test runner after fix should see all pass.

kibanamachine · 2024-10-25T06:42:24Z

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#7258

[✅] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/exceptions/operators_data_types/date_types/basic_license_essentials_tier/configs/ess.config.ts: 25/25 tests passed.
[✅] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/exceptions/operators_data_types/date_types/basic_license_essentials_tier/configs/serverless.config.ts: 25/25 tests passed.
[✅] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/exceptions/operators_data_types/double/basic_license_essentials_tier/configs/ess.config.ts: 25/25 tests passed.
[✅] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/exceptions/operators_data_types/double/basic_license_essentials_tier/configs/serverless.config.ts: 25/25 tests passed.
[✅] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/exceptions/operators_data_types/float/basic_license_essentials_tier/configs/ess.config.ts: 25/25 tests passed.
[✅] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/exceptions/operators_data_types/float/basic_license_essentials_tier/configs/serverless.config.ts: 25/25 tests passed.
[✅] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/exceptions/operators_data_types/integer/basic_license_essentials_tier/configs/ess.config.ts: 25/25 tests passed.
[❌] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/exceptions/operators_data_types/integer/basic_license_essentials_tier/configs/serverless.config.ts: 0/25 tests passed.

see run history

kibanamachine · 2024-10-25T18:29:59Z

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#7269

[✅] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/exceptions/operators_data_types/integer/basic_license_essentials_tier/configs/ess.config.ts: 25/25 tests passed.
[✅] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/exceptions/operators_data_types/integer/basic_license_essentials_tier/configs/serverless.config.ts: 25/25 tests passed.

see run history

kibanamachine · 2024-10-25T19:15:22Z

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#7268

[✅] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/eql/trial_license_complete_tier/configs/ess.config.ts: 10/10 tests passed.
[✅] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/eql/trial_license_complete_tier/configs/serverless.config.ts: 10/10 tests passed.
[✅] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/esql/trial_license_complete_tier/configs/ess.config.ts: 10/10 tests passed.
[✅] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/esql/trial_license_complete_tier/configs/serverless.config.ts: 10/10 tests passed.
[✅] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/general_logic/trial_license_complete_tier/configs/ess.config.ts: 10/10 tests passed.
[✅] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/general_logic/trial_license_complete_tier/configs/serverless.config.ts: 10/10 tests passed.
[✅] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/indicator_match/trial_license_complete_tier/configs/ess.config.ts: 10/10 tests passed.
[✅] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/indicator_match/trial_license_complete_tier/configs/serverless.config.ts: 10/10 tests passed.
[✅] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/machine_learning/trial_license_complete_tier/configs/ess.config.ts: 10/10 tests passed.
[✅] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/machine_learning/trial_license_complete_tier/configs/serverless.config.ts: 10/10 tests passed.
[✅] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/new_terms/trial_license_complete_tier/configs/ess.config.ts: 10/10 tests passed.
[✅] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/new_terms/trial_license_complete_tier/configs/serverless.config.ts: 10/10 tests passed.
[✅] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/query/trial_license_complete_tier/configs/ess.config.ts: 10/10 tests passed.
[❌] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/query/trial_license_complete_tier/configs/serverless.config.ts: 1/10 tests passed.
[❌] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/threshold/trial_license_complete_tier/configs/ess.config.ts: 0/10 tests passed.
[❌] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/threshold/trial_license_complete_tier/configs/serverless.config.ts: 0/10 tests passed.

see run history

…iew fails when trying to rebuild history using preview alerts index - thanks marshall for helping fix

elasticmachine · 2024-10-29T21:34:00Z

Pinging @elastic/security-solution (Team: SecuritySolution)

elasticmachine · 2024-10-29T21:34:01Z

Pinging @elastic/security-detection-engine (Team:Detection Engine)

…-fix'

kibanamachine · 2024-10-29T22:24:35Z

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#7299

[✅] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/threshold/trial_license_complete_tier/configs/ess.config.ts: 25/25 tests passed.
[✅] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/threshold/trial_license_complete_tier/configs/serverless.config.ts: 25/25 tests passed.

see run history

elasticmachine · 2024-10-29T23:39:22Z

💚 Build Succeeded

Metrics [docs]

✅ unchanged

cc @yctercero

dhurley14 · 2024-10-30T16:12:35Z

...ty_solution/server/lib/detection_engine/rule_types/threshold/get_threshold_signal_history.ts

@@ -46,6 +46,9 @@ export const getThresholdSignalHistory = async ({
  const response = await esClient.search({
    ...request,
    index: indexPattern,
+    // If alerts index is not yet created,
+    // do not throw a 404
+    ignore_unavailable: true,


kibanamachine · 2024-10-30T17:44:26Z

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/11598766940

…hreshold preview bug (elastic#197368) ## Summary This PR breaks down long running FTR groups into smaller chunks that now run in <~15 min. - Addresses elastic#192109 - There is no existing ticket but rule_execution group tests are taking ~55m to run and will soon be a bottle neck for us. - No edits made to any existing tests. - Purely just a reshuffle of the tests. See elastic#198209 for details on bug. (cherry picked from commit cd1fafe)

kibanamachine · 2024-10-30T17:49:00Z

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

breaking down long running FTR groups

4168bea

This comment was marked as outdated.

Sign in to view

yctercero added 2 commits October 24, 2024 21:09

update refs for execution tests

d2d6680

Merge branch 'main' of github.com:elastic/kibana into breakdown_ftrs

0573aeb

This comment was marked as outdated.

Sign in to view

yctercero changed the title ~~breaking down long running FTR groups~~ [Detection Engine][FTRs] Break down long running FTR groups Oct 25, 2024

yctercero self-assigned this Oct 25, 2024

yctercero added v9.0.0 Team:Detection Engine Security Solution Detection Engine Area v8.17.0 Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. release_note:skip Skip the PR/issue when compiling release notes labels Oct 25, 2024

This comment was marked as outdated.

Sign in to view

yctercero added 2 commits October 29, 2024 11:57

Merge branch 'main' of github.com:elastic/kibana into breakdown_ftrs

5922ec3

found a bug where if preview index is not yet created, threshold prev…

1ceef86

…iew fails when trying to rebuild history using preview alerts index - thanks marshall for helping fix

yctercero marked this pull request as ready for review October 29, 2024 21:33

yctercero requested review from a team as code owners October 29, 2024 21:33

yctercero requested a review from rylnd October 29, 2024 21:34

yctercero added the backport:version Backport to applied version labels label Oct 29, 2024

Ikuni17 approved these changes Oct 29, 2024

View reviewed changes

[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…

1c02ec3

…-fix'

yctercero changed the title ~~[Detection Engine][FTRs] Break down long running FTR groups~~ [Detection Engine][FTRs] Break down long running FTR groups and fix threshold preview bug Oct 30, 2024

dhurley14 approved these changes Oct 30, 2024

View reviewed changes

yctercero merged commit cd1fafe into elastic:main Oct 30, 2024
45 checks passed

kibanamachine mentioned this pull request Oct 30, 2024

[8.x] [Detection Engine][FTRs] Break down long running FTR groups and fix threshold preview bug (#197368) #198419

Open

This was referenced Oct 30, 2024

[Serverless][Kibana QA QG] exception_operators_date_numeric_types:essentials:qa:serverless:release tests optimization #192109

Closed

[Detection Engine][Preview] Threshold preview fails when no alerts index exists #198209

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Detection Engine][FTRs] Break down long running FTR groups and fix threshold preview bug #197368

[Detection Engine][FTRs] Break down long running FTR groups and fix threshold preview bug #197368

yctercero commented Oct 23, 2024 •

edited by kibanamachine

Loading

This comment was marked as outdated.

This comment was marked as outdated.

kibanamachine commented Oct 25, 2024

kibanamachine commented Oct 25, 2024

kibanamachine commented Oct 25, 2024

This comment was marked as outdated.

elasticmachine commented Oct 29, 2024

elasticmachine commented Oct 29, 2024

kibanamachine commented Oct 29, 2024

elasticmachine commented Oct 29, 2024

dhurley14 Oct 30, 2024

kibanamachine commented Oct 30, 2024

kibanamachine commented Oct 30, 2024

[Detection Engine][FTRs] Break down long running FTR groups and fix threshold preview bug #197368

[Detection Engine][FTRs] Break down long running FTR groups and fix threshold preview bug #197368

Conversation

yctercero commented Oct 23, 2024 • edited by kibanamachine Loading

Summary

This comment was marked as outdated.

This comment was marked as outdated.

kibanamachine commented Oct 25, 2024

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#7258

kibanamachine commented Oct 25, 2024

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#7269

kibanamachine commented Oct 25, 2024

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#7268

This comment was marked as outdated.

elasticmachine commented Oct 29, 2024

elasticmachine commented Oct 29, 2024

kibanamachine commented Oct 29, 2024

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#7299

elasticmachine commented Oct 29, 2024

💚 Build Succeeded

Metrics [docs]

dhurley14 Oct 30, 2024

Choose a reason for hiding this comment

kibanamachine commented Oct 30, 2024

kibanamachine commented Oct 30, 2024

💚 All backports created successfully

Questions ?

yctercero commented Oct 23, 2024 •

edited by kibanamachine

Loading