Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Security Solution][Detection Engine] ES|QL rule type generates less alerts than expected with MV_EXPAND #197130

Open
marshallmain opened this issue Oct 21, 2024 · 2 comments
Assignees
Labels
9.1 candidate Feature:Detection Rules Anything related to Security Solution's Detection Rules Team:Detection Engine Security Solution Detection Engine Area

Comments

@marshallmain
Copy link
Contributor

The MV_EXPAND command is in preview and was added to ES|QL for 8.11 (https://www.elastic.co/guide/en/elasticsearch/reference/8.15/esql-commands.html#esql-mv_expand). This command takes a multi-valued column and produces a separate row in the results for each value. As a consequence, the results can have multiple rows with the same _id and _index values but the DE rule logic will only create an alert for one of these rows due to deduplication. We should investigate what the correct handling is in this scenario.

@marshallmain marshallmain added Feature:Detection Rules Anything related to Security Solution's Detection Rules Team:Detection Engine Security Solution Detection Engine Area labels Oct 21, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detection-engine (Team:Detection Engine)

@vitaliidm
Copy link
Contributor

Possible way to do this is to extend what properties we include in alert id.

Currently, for non-agg queries we include id, index, version, ruleId, spaceId

By using utilities from @kbn/esql-ast package, I believe we would be able extract mv_expand field and retrieve its value from processed result row. Then we can include it key-value object into id generator. In this case, this would allow to generate unique alert id per each "expanded" value.
One corner case, we should keep in mind, user might drop or rename "expanded" field. In this case we might resort to remove all identical values from result rows and use the rest values in id generation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
9.1 candidate Feature:Detection Rules Anything related to Security Solution's Detection Rules Team:Detection Engine Security Solution Detection Engine Area
Projects
None yet
Development

No branches or pull requests

4 participants