Expand ML Job/Rule documentation to include Related Integrations and Setup information

[spong]

Description

From an internal discussion, it was asked if there was any chance to document better what exact integrations or configurations need to be fulfilled so that the ML jobs will have data as this is not really obvious from the existing documentation alone.

The docs in question include the security prebuilt ml jobs (https://www.elastic.co/guide/en/security/8.6/prebuilt-ml-jobs.html), and potentially the ML docs as well for the OOTB ML Jobs (https://www.elastic.co/guide/en/machine-learning/8.6/ootb-ml-jobs-uptime.html).

Note: This documentation update can be done in parallel with exposing this information within the Rule Details by means of adding it directly to the rule. Detection-rules issue for track: elastic/detection-rules#2548

Acceptance Test Criteria

As a user, I should be able to see what integrations or prior setup instructions may be necessary to successfully use our prebuilt ML Jobs/Rules.

Notes

Please see the @elastic/security-ml folks for the actual Related Integration and Setup content to be added here.

[ajosh0504]

@jmikell821 Let us know by when you want to start work on this so we can prioritize documenting the list of related integrations for you to use accordingly.

[jmikell821]

@jmikell821 Let us know by when you want to start work on this so we can prioritize documenting the list of related integrations for you to use accordingly.

Hi @ajosh0504 - it depends on the urgency. We can start working on it now, though it may get in a little post 8.7. Let me know when you have the list of integrations ready, then we'll take it from there. Thanks!

[jmikell821]

Hey there @spong and @ajosh0504. The following PRs are merged:

• Update ML page with anomaly detection jobs from Elastic integrations #3648
• Add package ml jobs stack-docs#2496

Can you review when you get a chance and let us know if anything is missing for this issue that we need to add or modify? Thanks!

[spong]

Thanks for the ping @jmikell821! 🙂

So the added docs looks great, but I'm not sure they answer the original linked question from slack as to what data integrations are needed for these jobs/rules:

Would there be any chance to document better what exact integrations or configurations need to be fulfilled so that these ML learning jobs will have data. This is not really obvious from in the documentation only.

The above two PR's seem to add general documentation around the ML Integration itself, but I'm not seeing anywhere where it details what data integrations are necessary for these jobs/rules? Or does the data collection happen as part of the installed integration itself now?

e.g., for the Lateral Movement Detection (LMD) integration it states that it detect lateral movement based on file transfer activity and Windows RDP events. What integrations should the user install to get the necessary file transfer activity and Windows RDP events?

[ajosh0504]

@jmikell821 Sorry it took a while to get to this. And agree with @spong that the above PRs are not related to this work. Related integrations were added to pre-built ML rules in this PR and should show up in docs automatically. This however does not include our Advanced Analytics jobs which were added to the docs in the PRs you linked above. There is no automated process to add related integrations for these. That said, I now have a catalog of ALL our jobs with related integrations in this spreadsheet. I'll leave it up to you to decide what the best way to add them to the existing docs is.

[susan-shu-c]

Related to: https://github.com/elastic/security-team/issues/8897