Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expand ML Job/Rule documentation to include Related Integrations and Setup information #2974

Open
spong opened this issue Feb 14, 2023 · 6 comments
Assignees
Labels
Docset: ESS Issues that apply to docs in the Stack release Docset: Serverless Issues for Serverless Security Effort: Large Issues that require significant planning, research, writing, and testing enhancement New feature or request Feature: Machine Learning Feature: Prebuilt rules Priority: Medium Issues that have relevance, but aren't urgent Team: Detections/Response Detections and Response Team: Security Solution v8.7.0

Comments

@spong
Copy link
Member

spong commented Feb 14, 2023

Description

From an internal discussion, it was asked if there was any chance to document better what exact integrations or configurations need to be fulfilled so that the ML jobs will have data as this is not really obvious from the existing documentation alone.

The docs in question include the security prebuilt ml jobs (https://www.elastic.co/guide/en/security/8.6/prebuilt-ml-jobs.html), and potentially the ML docs as well for the OOTB ML Jobs (https://www.elastic.co/guide/en/machine-learning/8.6/ootb-ml-jobs-uptime.html).

Note: This documentation update can be done in parallel with exposing this information within the Rule Details by means of adding it directly to the rule. Detection-rules issue for track: elastic/detection-rules#2548

Acceptance Test Criteria

As a user, I should be able to see what integrations or prior setup instructions may be necessary to successfully use our prebuilt ML Jobs/Rules.

Notes

Please see the @elastic/security-ml folks for the actual Related Integration and Setup content to be added here.

@ajosh0504
Copy link
Contributor

@jmikell821 Let us know by when you want to start work on this so we can prioritize documenting the list of related integrations for you to use accordingly.

@jmikell821 jmikell821 added v8.7.0 Team: Detections/Response Detections and Response labels Feb 22, 2023
@jmikell821
Copy link
Contributor

@jmikell821 Let us know by when you want to start work on this so we can prioritize documenting the list of related integrations for you to use accordingly.

Hi @ajosh0504 - it depends on the urgency. We can start working on it now, though it may get in a little post 8.7. Let me know when you have the list of integrations ready, then we'll take it from there. Thanks!

@joepeeples joepeeples added Priority: Medium Issues that have relevance, but aren't urgent Effort: Large Issues that require significant planning, research, writing, and testing labels Jun 23, 2023
@jmikell821
Copy link
Contributor

Hey there @spong and @ajosh0504. The following PRs are merged:

Can you review when you get a chance and let us know if anything is missing for this issue that we need to add or modify? Thanks!

@spong
Copy link
Member Author

spong commented Aug 17, 2023

Thanks for the ping @jmikell821! 🙂

So the added docs looks great, but I'm not sure they answer the original linked question from slack as to what data integrations are needed for these jobs/rules:

Would there be any chance to document better what exact integrations or configurations need to be fulfilled so that these ML learning jobs will have data. This is not really obvious from in the documentation only.

The above two PR's seem to add general documentation around the ML Integration itself, but I'm not seeing anywhere where it details what data integrations are necessary for these jobs/rules? Or does the data collection happen as part of the installed integration itself now?

e.g., for the Lateral Movement Detection (LMD) integration it states that it detect lateral movement based on file transfer activity and Windows RDP events. What integrations should the user install to get the necessary file transfer activity and Windows RDP events?

@ajosh0504
Copy link
Contributor

@jmikell821 Sorry it took a while to get to this. And agree with @spong that the above PRs are not related to this work. Related integrations were added to pre-built ML rules in this PR and should show up in docs automatically. This however does not include our Advanced Analytics jobs which were added to the docs in the PRs you linked above. There is no automated process to add related integrations for these. That said, I now have a catalog of ALL our jobs with related integrations in this spreadsheet. I'll leave it up to you to decide what the best way to add them to the existing docs is.

@joepeeples joepeeples added Docset: Serverless Issues for Serverless Security Docset: ESS Issues that apply to docs in the Stack release labels Dec 1, 2023
@joepeeples joepeeples removed their assignment Mar 21, 2024
@susan-shu-c
Copy link
Member

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Docset: ESS Issues that apply to docs in the Stack release Docset: Serverless Issues for Serverless Security Effort: Large Issues that require significant planning, research, writing, and testing enhancement New feature or request Feature: Machine Learning Feature: Prebuilt rules Priority: Medium Issues that have relevance, but aren't urgent Team: Detections/Response Detections and Response Team: Security Solution v8.7.0
Projects
None yet
Development

No branches or pull requests

5 participants