[EDR Workflows] Verify flaky osquery tests #169749

tomsonpl · 2023-10-25T06:55:54Z

Add video recording functionality to Osquery Cypress tests (only failed specs recording will be saved and uploaded as artifacts). Currently the video compressions is limited, because on full scale it was coming up to 100% of CPU usage.
remove usage of react-cypress-selector from packs_integration.cy.ts

Closes:

Failing test: Osquery Cypress.x-pack/plugins/osquery/cypress/e2e/all/add_integration·cy·ts - ALL - Add Integration Add and upgrade integration "after each" hook for "should add the old integration and be able to upgrade it" "after each" hook for "should add the old integration and be able to upgrade it" #169702
Failing test: Osquery Cypress.x-pack/plugins/osquery/cypress/e2e/all/add_integration·cy·ts - ALL - Add Integration Add and upgrade integration should add the old integration and be able to upgrade it should add the old integration and be able to upgrade it #169701
Failing test: Osquery Cypress.x-pack/plugins/osquery/cypress/e2e/all/alerts_automated_action_results·cy·ts - Alert Flyout Automated Action Results "before each" hook for "can add to timeline from response action results" "before each" hook for "can add to timeline from response action results" #169727
Failing test: Osquery Cypress.x-pack/plugins/osquery/cypress/e2e/all/cases·cy·ts - Add to Cases security "before all" hook for "should add result a case and have add to timeline in result" "before all" hook for "should add result a case and have add to timeline in result" #169747
Failing test: Osquery Cypress.x-pack/plugins/osquery/cypress/e2e/all/saved_queries·cy·ts - ALL - Saved queries Saved queries Complex Test should create a new query and verify: - hidden columns, full screen and sorting - pagination - query can viewed (status), edited and deleted should create a new query and verify: - hidden columns, full screen and sorting - pagination - query can viewed (status), edited and deleted #169786
Failing test: Osquery Cypress.x-pack/plugins/osquery/cypress/e2e/all/saved_queries·cy·ts - ALL - Saved queries prebuilt user can run prebuilt saved query and add to case user can run prebuilt saved query and add to case #169787
Failing test: Osquery Cypress.x-pack/plugins/osquery/cypress/e2e/all/alerts_response_actions_form·cy·ts - Alert Event Details - Response Actions Form adds response actions with osquery with proper validation and form values adds response actions with osquery with proper validation and form values #169785
Failing test: Osquery Cypress.x-pack/plugins/osquery/cypress/e2e/all/live_query·cy·ts - ALL - Live Query should run multiline query should run multiline query #169725
Failing test: Osquery Cypress.x-pack/plugins/osquery/cypress/e2e/all/packs_integration·cy·ts - ALL - Packs Global packs add proper shard to policies packs config #169688
Failing test: Osquery Cypress.x-pack/plugins/osquery/cypress/e2e/all/metrics·cy·ts - ALL - Inventory should be able to run the previously saved query should be able to run the previously saved query #169370
Failing test: Osquery Cypress.x-pack/plugins/osquery/cypress/e2e/all/metrics·cy·ts - ALL - Inventory should be able to run the query should be able to run the query #169369
Failing test: Osquery Cypress.x-pack/plugins/osquery/cypress/e2e/all/alerts_multiple_agents·cy·ts - Alert Event Details - dynamic params "before each" hook for "should substitute parameters in investigation guide" "before each" hook for "should substitute parameters in investigation guide" #170157
Failing test: Osquery Cypress.x-pack/plugins/osquery/cypress/e2e/all/metrics·cy·ts - ALL - Inventory should be able to run the previously saved query should be able to run the previously saved query #169575

…-fix'

…-alerts-loading

…-fix'

…-alerts-loading

…-ref HEAD~1..HEAD --fix'

patrykkopycinski · 2023-10-31T10:57:57Z

x-pack/plugins/osquery/cypress/e2e/all/packs_integration.cy.ts

        cy.getBySel('policyIdsComboBox').should('exist');
        cy.getBySel('osqueryPackTypeGlobal').click();
        cy.getBySel('policyIdsComboBox').should('not.exist');

-        findAndClickButton('Save pack');
+        cy.get('button').contains('Save pack').click();


can we add data-test-subj instead?

I think we can, let me try

patrykkopycinski · 2023-10-31T10:58:06Z

x-pack/plugins/osquery/cypress/e2e/all/saved_queries.cy.ts

-      cy.react('CustomItemAction', {
-        props: { index: 1, item: { id: 'users_elastic' } },
-      }).click();
+      cy.get(`[aria-label="Edit users_elastic"]`).click();


patrykkopycinski · 2023-10-31T10:58:33Z

x-pack/plugins/osquery/cypress/support/e2e.ts

@@ -80,9 +81,13 @@ Cypress.Commands.add(

 Cypress.Commands.add('login', (role) => {
  if (isServerless) {
+    cy.log('commands1');


is it needed?

no, thanks 👍

patrykkopycinski · 2023-10-31T10:58:50Z

x-pack/plugins/osquery/cypress/support/filter_videos.ts

+import fs from 'fs';
+
+// makes sure we save videos just for failed specs
+export const filterCypressVideos = (spec: Cypress.Spec, results: CypressCommandLine.RunResult) => {


can we add more descriptive variable name?

filteredFailedSpecVideos?

getFailedSpecVideos ?

patrykkopycinski · 2023-10-31T10:59:30Z

x-pack/plugins/osquery/cypress/tasks/inventory.ts

@@ -6,10 +6,12 @@
 */

 export const triggerLoadData = () => {
+  // @ts-expect-error update types for multiple true
+  const nodeContainers = cy.getBySel('nodeContainer', { multiple: true });


I don't think it's a good practice to store the results of cy.getBySel in variable and used it later on

patrykkopycinski · 2023-10-31T11:00:30Z

x-pack/test/osquery_cypress/runner.ts

+  const hostname = agent.substring(0, 12);
+  await waitForHostToEnroll(kbnClient, hostname);
+  const agentTwo = await new AgentManager(policyEnrollmentKeyTwo, port, log).setup();
+  const hostnameTwo = agentTwo.substring(0, 12);


can we return hostname from AgentManager, so we don't need to hardcode (0, 12)?

patrykkopycinski · 2023-10-31T13:16:55Z

x-pack/plugins/osquery/cypress/e2e/all/alerts_response_actions_form.cy.ts


+    // cy.contains('Please fix issues listed below').should('not.exist');


can we remove it?

patrykkopycinski · 2023-10-31T13:18:30Z

x-pack/plugins/osquery/cypress/e2e/all/live_query.cy.ts

@@ -87,6 +86,6 @@ describe.skip('ALL - Live Query', { tags: ['@ess', '@serverless'] }, () => {

    inputQuery('{selectall}{backspace}{selectall}{backspace}');
    // not sure if this is how it used to work when I implemented the functionality, but let's leave it like this for now
-    cy.get(LIVE_QUERY_EDITOR).invoke('height').should('be.gt', 200).and('be.lt', 380);
+    cy.get(LIVE_QUERY_EDITOR).invoke('height').should('be.gt', 200).and('be.lt', 400);


instead of hardcoding values, could we write it in a more dynamic way? checking if the values have changed?

Yeah not sure, sometimes these numbers change, but in general it is good to know if it is in some scope, and not changed as from 300px to 3000px. I find this to be a quite easy way to get it tested, now just making the expectations more loose.

patrykkopycinski · 2023-10-31T13:19:08Z

x-pack/plugins/osquery/cypress/e2e/all/packs_integration.cy.ts

-        findAndClickButton('Save pack');
-
+        navigateToWithoutWaitForReact('app/osquery/packs');
+        cy.get('span').contains('Add pack').click();


I don't think span are clickable

How do you think this works then? :P

I'll use data-test-subj 👍

patrykkopycinski · 2023-10-31T13:19:17Z

x-pack/plugins/osquery/cypress/e2e/all/packs_integration.cy.ts

+        cy.get('span').contains('Add pack').click();
+        cy.get('input[name="name"]').type(`${REMOVING_PACK}{downArrow}{enter}`);
+        cy.getBySel('policyIdsComboBox').type(`${AGENT_POLICY_NAME}{downArrow}{enter}`);
+        // cy.get('button').contains('Save pack').click();


remove commented out line

patrykkopycinski · 2023-10-31T13:20:25Z

x-pack/test/osquery_cypress/runner.ts

+  const port = config.get('servers.fleetserver.port');
+
+  const agent = await new AgentManager(policyEnrollmentKey, port, log).setup();
+  await waitForHostToEnroll(kbnClient, agent);


maybe we can move waitForHostToEnroll into AgentManager?

I think we can, I had a different approach in place for firstly waitForHostToEnroll and if failed to create a new agent, but I do not do it now so this can be easily moved to agentManager if you think it makes sense

…-alerts-loading

jbudz

.buildkite LGTM

@jbudz Hey, I added a video recording functionality to our cypress tests (saved only for failed specs), can you take a look from kibana-ops perspective, if this is fine ? Thanks!

ffmpeg is relatively cpu heavy. I see a few CPU spikes to 100%, but mostly in the 85% range. I'm okay with trying things out, but it's something to keep in mind if we start seeing timeouts.

kibana-ci · 2023-10-31T18:22:04Z

💚 Build Succeeded

Buildkite Build
Commit: 80c962b

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`osquery`	1.0MB	1.0MB	+103.0B

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @tomsonpl

kibanamachine · 2023-11-01T06:14:50Z

💔 All backports failed

Status	Branch	Result
❌	8.11	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 169749

Questions ?

Please refer to the Backport tool documentation

ashokaditya · 2023-11-07T13:31:59Z

💚 All backports created successfully

Status	Branch	Result
✅	8.11

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

(cherry picked from commit a15a948) # Conflicts: # .buildkite/scripts/lifecycle/post_command.sh # x-pack/plugins/osquery/cypress/e2e/all/alerts_automated_action_results.cy.ts # x-pack/plugins/osquery/cypress/e2e/all/alerts_response_actions_form.cy.ts # x-pack/plugins/osquery/cypress/e2e/all/packs_integration.cy.ts # x-pack/plugins/osquery/cypress/support/e2e.ts

tomsonpl · 2023-11-09T10:37:10Z

💚 All backports created successfully

Status	Branch	Result
✅	8.11

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

test flakiness

f2d6929

tomsonpl added the release_note:skip Skip the PR/issue when compiling release notes label Oct 25, 2023

tomsonpl self-assigned this Oct 25, 2023

tomsonpl and others added 27 commits October 25, 2023 10:01

test flakiness

18bc8b0

test flakiness

69b847c

fix cases

b0e4530

actions_form

bdeecef

check login

91ca659

brokeninserverless one test

5853004

change login

7bd8231

Merge branch 'main' into flaky-alerts-loading

f40ba89

add check for agent status

a6fa2ae

test

4abf5d3

test

01d1c39

clickoutside and add cy.log

cb282ca

[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…

cc11fbc

…-fix'

recursively set up agent

bffa366

Merge remote-tracking branch 'origin/flaky-alerts-loading' into flaky…

cbf6f8f

…-alerts-loading

add more time

969d114

more time

c313e04

test video

835e39a

[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…

dcbcd05

…-fix'

test video

dc55d9c

Merge remote-tracking branch 'origin/flaky-alerts-loading' into flaky…

0d9e7d8

…-alerts-loading

fix test

e825588

fix test

68b9640

remove global before to login

2da2515

limit specs

fc350db

clean up

f719820

[CI] Auto-commit changed files from 'node scripts/precommit_hook.js -…

ef9cc6f

…-ref HEAD~1..HEAD --fix'

patrykkopycinski reviewed Oct 31, 2023

View reviewed changes

tomsonpl added 3 commits October 31, 2023 13:47

fixes

9105d07

pass containerName

e188c9c

fix

9429418

tomsonpl requested a review from patrykkopycinski October 31, 2023 13:12

patrykkopycinski reviewed Oct 31, 2023

View reviewed changes

fix

11479e3

patrykkopycinski reviewed Oct 31, 2023

View reviewed changes

tomsonpl added 2 commits October 31, 2023 14:51

fix

ed51ade

fix

97b8865

tomsonpl requested a review from patrykkopycinski October 31, 2023 14:21

tomsonpl and others added 4 commits October 31, 2023 15:31

fix

76d1e87

Merge branch 'main' into flaky-alerts-loading

212cc0a

remove video compressions

03be3ef

Merge remote-tracking branch 'origin/flaky-alerts-loading' into flaky…

89c5342

…-alerts-loading

jbudz approved these changes Oct 31, 2023

View reviewed changes

change compression

80c962b

joeypoon approved these changes Nov 1, 2023

View reviewed changes

tomsonpl merged commit a15a948 into elastic:main Nov 1, 2023
26 checks passed

delanni pushed a commit to delanni/kibana that referenced this pull request Nov 6, 2023

[EDR Workflows] Verify flaky osquery tests (elastic#169749)

a9e06b2

kibanamachine mentioned this pull request Nov 6, 2023

[Security Solution][Endpoint][Cypress] Update endpoint cypress config to include videos #170499

Merged

ashokaditya mentioned this pull request Nov 7, 2023

[8.11] [EDR Workflows] Verify flaky osquery tests (#169749) #170736

Closed

tomsonpl mentioned this pull request Nov 9, 2023

[8.11] [EDR Workflows] Verify flaky osquery tests (#169749) #170930

Merged

tomsonpl added a commit that referenced this pull request Nov 14, 2023

[8.11] [EDR Workflows] Verify flaky osquery tests (#169749) (#170930)

b8d0002

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[EDR Workflows] Verify flaky osquery tests #169749

[EDR Workflows] Verify flaky osquery tests #169749

tomsonpl commented Oct 25, 2023 •

edited

Loading

patrykkopycinski Oct 31, 2023

tomsonpl Oct 31, 2023

patrykkopycinski Oct 31, 2023

patrykkopycinski Oct 31, 2023

tomsonpl Oct 31, 2023

patrykkopycinski Oct 31, 2023

tomsonpl Oct 31, 2023

tomsonpl Oct 31, 2023

patrykkopycinski Oct 31, 2023

patrykkopycinski Oct 31, 2023

patrykkopycinski Oct 31, 2023

patrykkopycinski Oct 31, 2023

tomsonpl Oct 31, 2023

patrykkopycinski Oct 31, 2023

tomsonpl Oct 31, 2023

tomsonpl Oct 31, 2023

patrykkopycinski Oct 31, 2023

patrykkopycinski Oct 31, 2023

tomsonpl Oct 31, 2023

jbudz left a comment

kibana-ci commented Oct 31, 2023

kibanamachine commented Nov 1, 2023

ashokaditya commented Nov 7, 2023

tomsonpl commented Nov 9, 2023


		// cy.contains('Please fix issues listed below').should('not.exist');

[EDR Workflows] Verify flaky osquery tests #169749

[EDR Workflows] Verify flaky osquery tests #169749

Conversation

tomsonpl commented Oct 25, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jbudz left a comment

Choose a reason for hiding this comment

kibana-ci commented Oct 31, 2023

💚 Build Succeeded

Metrics [docs]

Async chunks

History

kibanamachine commented Nov 1, 2023

💔 All backports failed

Manual backport

Questions ?

ashokaditya commented Nov 7, 2023

💚 All backports created successfully

Questions ?

tomsonpl commented Nov 9, 2023

💚 All backports created successfully

Questions ?

tomsonpl commented Oct 25, 2023 •

edited

Loading