Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add nginx ML module tutorial #3878

Merged
merged 6 commits into from
May 30, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 70 additions & 0 deletions docs/en/observability/monitor-nginx-ml.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
[discrete]
[[monitor-nginx-ml]]
== Part 5: Find anomalies in your nginx access logs

Use the {integrations-docs}/nginx[nginx Elastic integration] machine learning (ML) module to help find unusual activity in your nginx access logs.
Monitoring anomalies in your access logs helps you detect:

* security threats
* network issues
* system performance issues
* operational efficiency issues

[discrete]
[[monitor-nginx-ml-jobs]]
=== nginx anomaly detection jobs

The nginx ML module provides the following anomaly detection jobs:

[[horizontal]]
Low request rates (`low_request_rate_nginx`):: Uses the {ml-docs}/ml-count-functions.html#ml-count[`low_count`] function to detect abnormally low request rates. Abnormally low request rates might indicate that network issues or other issues are preventing requests from reaching the server.
Unusual source IPs - high request rates (`source_ip_request_rate_nginx`):: Uses the {ml-docs}/ml-count-functions.html#ml-count[`hight_count`] function to detect abnormally high request rates from individual IP addresses. Many requests from a single IP or small group of IPs might indicate something malicious like a distributed denial of service (DDoS) attack where a large number of requests are sent to overwhelm the server and make it unavailable to users.
Unusual source IPs - high distinct count of URLs (`source_ip_url_count_nginx`):: Uses the {ml-docs}/ml-count-functions.html#ml-distinct-count[`high_distinct_count`] function to detect individual IP addresses accessing abnormally high numbers of unique URLs. A single IP accessing many unique URLs might indicate something malicious like web scraping or an attempt to find sensitive data or vulnerabilities.
Unusual status code rates (`status_code_rate_nginx`):: Uses the {ml-docs}/ml-count-functions.html#ml-count[`count`] function to detect abnormal error status code rates. A high rate of status codes could indicate problems with broken links, bad URLs, or unauthorized access attempts. A high rate of status codes could also point to server issues like limited resources or bugs in your code.
Unusual visitor rates (`visitor_rate_nginx`):: Uses the {ml-docs}/ml-count-functions.html#ml-nonzero-count[`non_zero_count`] function to detect abnormal visitor rates. High visitor rates could indicate something malicious like a DDoS attack.
Low visitor rates could indicate issues with access to the server.

NOTE: These anomaly detection jobs are available when you have data that matches the query specified in the ML module manifest. Users not following this tutorial can refer to {integrations-docs}/nginx#ml-modules[nginx integration ML modules] for more about the ML module manifest.

[discrete]
[[monitor-nginx-ml-prereqs]]
=== Before you begin

Verify that your environment is set up properly to use the {ml-features}.
If {es} {security-features} are enabled, you need a user with permissions to manage {anomaly-jobs}.
Refer to {ml-docs}/setup.html[Set up ML features].

[discrete]
[[monitor-nginx-ml-add-jobs]]
=== Add nginx ML jobs

Add the nginx ML jobs from the nginx integration to start using anomaly detection:

. From the main {kib} menu, go to *Machine Learning*. Under *Anomaly Detection*, select *Jobs*.
. Select *Create job*.
. In the search bar, enter *nginx* and select *Nginx access logs [Logs Nginx]*.
. Under *Use preconfigured jobs*, select the *Nginx access logs* card.
. Select *Create jobs*.

Back on the *Anomaly Detection Jobs* page, you should see the nginx anomaly detection jobs—`low_request_rate_nginx`, `source_ip_request_rate_nginx`, `source_ip_url_count_nginx`, `status_code_rate_nginx`, and `visitor_rate_nginx`.

[discrete]
[[monitor-nginx-ml-explore]]
=== Explore your anomaly detection job results

View your anomaly detection job results using the Anomaly Explorer or Single Metric Viewer found under *Anomaly Detection* in the Machine Learning menu.
The Anomaly Explorer shows the results from all or any combination of your nginx ML jobs.
The Single Metric Viewer focuses on a specific job.
These tools offer a comprehensive view of anomalies and help find patterns and irregularities across data points and time intervals.

Refer to {ml-docs}/ml-ad-view-results.html[View anomaly detection job results] for more on viewing and understanding your anomaly detection job results.

[discrete]
[[monitor-nginx-ml-alert]]
=== Set up alerts

With the nginx ML jobs detecting anomalies, you can set rules to generate alerts when your jobs meet specific conditions.
For example, you could set up a rule on the `low_request_rate_nginx` job to alert when low request rates hit a specific severity threshold.
When you get alerted, you can make sure your server isn't experiencing issues.

Refer to {ml-docs}/ml-configuring-alerts.html[Generating alerts for anomaly detection jobs] for more on setting these rules and generating alerts.
10 changes: 6 additions & 4 deletions docs/en/observability/monitor-nginx.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
Return to this tutorial after you've learned the basics.
****

Use the https://docs.elastic.co/integrations/nginx[Nginx Elastic integration] and the {agent} to collect valuable metrics and logs from your nginx instances. Then, use built-in dashboards and tools like Logs Explorer in {kib} allow you to visualize and monitor your nginx data from one place. This data provides valuable insight into your nginx instances—for example:
Use the {integrations-docs}/nginx[nginx Elastic integration] and the {agent} to collect valuable metrics and logs from your nginx instances. Then, use built-in dashboards and tools like Logs Explorer in {kib} to visualize and monitor your nginx data from one place. This data provides valuable insight into your nginx instances—for example:

* A spike in error logs for a certain resource may mean you have a deleted resource that is still needed.
* Access logs can show when a service's peak times are, and, from this, when it might be best to perform things like maintenance.
Expand Down Expand Up @@ -199,7 +199,7 @@ To open the nginx dashboard:
The *Metrics Nginx overview* shows visual representations of total requests, processed requests, heartbeat/up, active connections, reading/writing/waiting rates, request rate, accepts and handled rates, and drops rate.

[role="screenshot"]
image::images/nginx-metrics-dashboard.png[Nginx metrics dashboard, 75%]
image::images/nginx-metrics-dashboard.png[nginx metrics dashboard, 75%]

[discrete]
[[monitor-nginx-explore-logs]]
Expand Down Expand Up @@ -229,7 +229,7 @@ image::images/nginx-logs-explorer.png[Logs Explorer showing nginx error logs]

[discrete]
[[monitor-nginx-logs-dashboard]]
==== Nginx logs dashboards
==== nginx logs dashboards

The nginx integration has built-in dashboards that show the full picture of your nginx logs in one place.
To open the nginx dashboards:
Expand All @@ -246,4 +246,6 @@ image::images/nginx-logs-overview-dashboard.png[nginx logs overview dashboard, 7
The *Nginx access and error logs* dashboard shows your access logs over time, and lists your access and error logs.

[role="screenshot"]
image::images/nginx-logs-access-error-dashboard.png[nginx access and error logs dashboard, 75%]
image::images/nginx-logs-access-error-dashboard.png[nginx access and error logs dashboard, 75%]

include::monitor-nginx-ml.asciidoc[]