From 3ba100b2777f9a4f43fa04bc91e3e33f1041ac69 Mon Sep 17 00:00:00 2001 From: mdbirnstiehl Date: Thu, 9 May 2024 10:30:11 -0500 Subject: [PATCH 1/6] add nginx ML module tutorial --- .../observability/monitor-nginx-ml.asciidoc | 70 +++++++++++++++++++ docs/en/observability/monitor-nginx.asciidoc | 6 +- 2 files changed, 74 insertions(+), 2 deletions(-) create mode 100644 docs/en/observability/monitor-nginx-ml.asciidoc diff --git a/docs/en/observability/monitor-nginx-ml.asciidoc b/docs/en/observability/monitor-nginx-ml.asciidoc new file mode 100644 index 0000000000..14ef493658 --- /dev/null +++ b/docs/en/observability/monitor-nginx-ml.asciidoc @@ -0,0 +1,70 @@ +[discrete] +[[monitor-nginx-ml]] +== Part 5: Find anomalies in your nginx access logs + +Use the nginx machine learning (ML) module from the {integrations-docs}/nginx[Nginx Elastic integration] provides built in ML jobs to help find unusual activity in your nginx access logs. +Monitoring anomalies in your access logs helps you detect: + +* security threats +* network issues +* system performance issues +* operational efficiency issues + +[discrete] +[[monitor-nginx-ml-jobs]] +=== nginx anomaly detection jobs + +The nginx ML module provides the following jobs for detecting anomalies in your access logs: + +[[horizontal]] +Low request rates (low_request_rate_nginx):: Uses the {ml-docs}/ml-count-functions.html#ml-count[`low_count`] function to detect abnormally low request rates. Abnormally low request rates might indicate network issues or other issues are preventing requests from reaching the server. +Unusual source IPs - high request rates (`source_ip_request_rate_nginx`):: Uses the {ml-docs}/ml-count-functions.html#ml-count[`hight_count`] function to detect abnormally high request rates from individual IP addresses. Many requests from a single IP or small group of IPs might indicate something malicious like a DDoS attack where many requests are sent to overwhelm the server and make it unavailable to users. +Unusual source IPs - high distinct count of URLs (`source_ip_url_count_nginx`):: Uses the {ml-docs}/ml-count-functions.html#ml-distinct-count[`high_distinct_count`] function to detect individual IP addresses accessing abnormally high numbers of unique URLs. A single IP accessing many unique URLs might indicate something malicious like web scraping or an attempt to find sensitive data or vulnerabilities. +Unusual status code rates (`status_code_rate_nginx`):: Uses the {ml-docs}/ml-count-functions.html#ml-count[`count`] function to detect abnormal status code rates. A high rate of status codes could indicate problems with broken links, bad URLs, or unauthorized access attempts. A high rate of status codes could also point to server issues like limited resources or bugs in your code. +Unusual visitor rates (`visitor_rate_nginx`):: Uses the {ml-docs}/ml-count-functions.html#ml-nonzero-count[`non_zero_count`] function to detect abnormal visitor rates. High visitor rates could indicate something malicious like a DDoS attack. +Low visitor rates could indicate issues with access to the server. + +Refer to {integrations-docs}/nginx#ml-modules[Nginx integration ML modules] for more on the jobs and the ML module manifest. + +[discrete] +[[monitor-nginx-ml-prereqs]] +=== Before you begin + +Verify that your environment is set up properly to use the {ml-features}. +If the {es} {security-features} are enabled, you need a user with permissions to manage {anomaly-jobs}. +Refer to {ml-docs}/setup.html[Set up ML features]. + +[discrete] +[[monitor-nginx-ml-add-jobs]] +=== Add nginx ML jobs + +Add the nginx ML jobs from the nginx integration to start using anomaly detection: + +. From the main {kib} menu, go to *Machine Learning* → *Jobs*. +. Select *Create job*. +. In the search bar, enter *nginx* and select *Nginx access logs [Logs Nginx]*. +. Under *Use preconfigured jobs*, select the *Nginx access logs* card. +. Select *Create jobs*. + +Back on the *Jobs* page, you should see the nginx anomaly detection jobs—`low_request_rate_nginx`, `source_ip_request_rate_nginx`, s`ource_ip_url_count_nginx`, `status_code_rate_nginx`, and `visitor_rate_nginx`. + +[discrete] +[[monitor-nginx-ml-explore]] +=== Explore your anomaly detection job results + +View your anomaly detection job results using the Anomaly Explorer or Single Metric Viewer found under *Anomaly Detection* in the Machine Learning menu. +The Anomaly Explorer shows the results from all or any combination of your nginx ML jobs. +The Single Metric Viewer focuses on a specific job. +These tools offer a comprehensive view of anomalies and help find patterns and irregularities across data points and time intervals. + +Refer to {ml-docs}/ml-ad-view-results.html[View anomaly detection job results] for more on viewing and understanding your anomaly detection job results. + +[discrete] +[[monitor-nginx-ml-alert]] +=== Set up alerts + +With the nginx ML jobs detecting anomalies, you can set rules to generate alerts when your jobs meet specific conditions. +For example, you could set up a rule on the `low_request_rate_nginx` job to alert when low request rates hit a specific severity threshold. +When you get alerted, you can make sure your server isn't experiencing issues. + +Refer to {ml-docs}/ml-configuring-alerts.html[Generating alerts for anomaly detection jobs] for more on setting these rules and generating alerts. \ No newline at end of file diff --git a/docs/en/observability/monitor-nginx.asciidoc b/docs/en/observability/monitor-nginx.asciidoc index b9225d3e78..2f10a16ed3 100644 --- a/docs/en/observability/monitor-nginx.asciidoc +++ b/docs/en/observability/monitor-nginx.asciidoc @@ -9,7 +9,7 @@ Return to this tutorial after you've learned the basics. **** -Use the https://docs.elastic.co/integrations/nginx[Nginx Elastic integration] and the {agent} to collect valuable metrics and logs from your nginx instances. Then, use built-in dashboards and tools like Logs Explorer in {kib} allow you to visualize and monitor your nginx data from one place. This data provides valuable insight into your nginx instances—for example: +Use the {integrations-docs}/nginx[Nginx Elastic integration] and the {agent} to collect valuable metrics and logs from your nginx instances. Then, use built-in dashboards and tools like Logs Explorer in {kib} allow you to visualize and monitor your nginx data from one place. This data provides valuable insight into your nginx instances—for example: * A spike in error logs for a certain resource may mean you have a deleted resource that is still needed. * Access logs can show when a service's peak times are, and, from this, when it might be best to perform things like maintenance. @@ -246,4 +246,6 @@ image::images/nginx-logs-overview-dashboard.png[nginx logs overview dashboard, 7 The *Nginx access and error logs* dashboard shows your access logs over time, and lists your access and error logs. [role="screenshot"] -image::images/nginx-logs-access-error-dashboard.png[nginx access and error logs dashboard, 75%] \ No newline at end of file +image::images/nginx-logs-access-error-dashboard.png[nginx access and error logs dashboard, 75%] + +include::monitor-nginx-ml.asciidoc[] \ No newline at end of file From 1ad6b330a326c0149aea2f14abfaef9e50a9ed2d Mon Sep 17 00:00:00 2001 From: mdbirnstiehl Date: Fri, 10 May 2024 11:36:05 -0500 Subject: [PATCH 2/6] fix intro --- docs/en/observability/monitor-nginx-ml.asciidoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/en/observability/monitor-nginx-ml.asciidoc b/docs/en/observability/monitor-nginx-ml.asciidoc index 14ef493658..9f7ce399d5 100644 --- a/docs/en/observability/monitor-nginx-ml.asciidoc +++ b/docs/en/observability/monitor-nginx-ml.asciidoc @@ -2,7 +2,7 @@ [[monitor-nginx-ml]] == Part 5: Find anomalies in your nginx access logs -Use the nginx machine learning (ML) module from the {integrations-docs}/nginx[Nginx Elastic integration] provides built in ML jobs to help find unusual activity in your nginx access logs. +The {integrations-docs}/nginx[Nginx Elastic integration] provides built in machine learning (ML) jobs to help find unusual activity in your nginx access logs. Monitoring anomalies in your access logs helps you detect: * security threats @@ -14,7 +14,7 @@ Monitoring anomalies in your access logs helps you detect: [[monitor-nginx-ml-jobs]] === nginx anomaly detection jobs -The nginx ML module provides the following jobs for detecting anomalies in your access logs: +The nginx ML module provides the following anomaly detection jobs: [[horizontal]] Low request rates (low_request_rate_nginx):: Uses the {ml-docs}/ml-count-functions.html#ml-count[`low_count`] function to detect abnormally low request rates. Abnormally low request rates might indicate network issues or other issues are preventing requests from reaching the server. From 1119e46cace57ad80fcde0071d7efaab0dfbba8f Mon Sep 17 00:00:00 2001 From: mdbirnstiehl Date: Fri, 10 May 2024 12:07:01 -0500 Subject: [PATCH 3/6] review updates --- docs/en/observability/monitor-nginx-ml.asciidoc | 10 +++++----- docs/en/observability/monitor-nginx.asciidoc | 6 +++--- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/en/observability/monitor-nginx-ml.asciidoc b/docs/en/observability/monitor-nginx-ml.asciidoc index 9f7ce399d5..87dd7a7652 100644 --- a/docs/en/observability/monitor-nginx-ml.asciidoc +++ b/docs/en/observability/monitor-nginx-ml.asciidoc @@ -2,7 +2,7 @@ [[monitor-nginx-ml]] == Part 5: Find anomalies in your nginx access logs -The {integrations-docs}/nginx[Nginx Elastic integration] provides built in machine learning (ML) jobs to help find unusual activity in your nginx access logs. +The {integrations-docs}/nginx[Nginx Elastic integration] to help find unusual activity in your nginx access logs. Monitoring anomalies in your access logs helps you detect: * security threats @@ -17,21 +17,21 @@ Monitoring anomalies in your access logs helps you detect: The nginx ML module provides the following anomaly detection jobs: [[horizontal]] -Low request rates (low_request_rate_nginx):: Uses the {ml-docs}/ml-count-functions.html#ml-count[`low_count`] function to detect abnormally low request rates. Abnormally low request rates might indicate network issues or other issues are preventing requests from reaching the server. -Unusual source IPs - high request rates (`source_ip_request_rate_nginx`):: Uses the {ml-docs}/ml-count-functions.html#ml-count[`hight_count`] function to detect abnormally high request rates from individual IP addresses. Many requests from a single IP or small group of IPs might indicate something malicious like a DDoS attack where many requests are sent to overwhelm the server and make it unavailable to users. +Low request rates (`low_request_rate_nginx`):: Uses the {ml-docs}/ml-count-functions.html#ml-count[`low_count`] function to detect abnormally low request rates. Abnormally low request rates might indicate that network issues or other issues are preventing requests from reaching the server. +Unusual source IPs - high request rates (`source_ip_request_rate_nginx`):: Uses the {ml-docs}/ml-count-functions.html#ml-count[`hight_count`] function to detect abnormally high request rates from individual IP addresses. Many requests from a single IP or small group of IPs might indicate something malicious like a DDoS attack where a large number of requests are sent to overwhelm the server and make it unavailable to users. Unusual source IPs - high distinct count of URLs (`source_ip_url_count_nginx`):: Uses the {ml-docs}/ml-count-functions.html#ml-distinct-count[`high_distinct_count`] function to detect individual IP addresses accessing abnormally high numbers of unique URLs. A single IP accessing many unique URLs might indicate something malicious like web scraping or an attempt to find sensitive data or vulnerabilities. Unusual status code rates (`status_code_rate_nginx`):: Uses the {ml-docs}/ml-count-functions.html#ml-count[`count`] function to detect abnormal status code rates. A high rate of status codes could indicate problems with broken links, bad URLs, or unauthorized access attempts. A high rate of status codes could also point to server issues like limited resources or bugs in your code. Unusual visitor rates (`visitor_rate_nginx`):: Uses the {ml-docs}/ml-count-functions.html#ml-nonzero-count[`non_zero_count`] function to detect abnormal visitor rates. High visitor rates could indicate something malicious like a DDoS attack. Low visitor rates could indicate issues with access to the server. -Refer to {integrations-docs}/nginx#ml-modules[Nginx integration ML modules] for more on the jobs and the ML module manifest. +Refer to {integrations-docs}/nginx#ml-modules[nginx integration ML modules] for more on the jobs and the ML module manifest. [discrete] [[monitor-nginx-ml-prereqs]] === Before you begin Verify that your environment is set up properly to use the {ml-features}. -If the {es} {security-features} are enabled, you need a user with permissions to manage {anomaly-jobs}. +If {es} {security-features} are enabled, you need a user with permissions to manage {anomaly-jobs}. Refer to {ml-docs}/setup.html[Set up ML features]. [discrete] diff --git a/docs/en/observability/monitor-nginx.asciidoc b/docs/en/observability/monitor-nginx.asciidoc index 2f10a16ed3..e4d7a4dedc 100644 --- a/docs/en/observability/monitor-nginx.asciidoc +++ b/docs/en/observability/monitor-nginx.asciidoc @@ -9,7 +9,7 @@ Return to this tutorial after you've learned the basics. **** -Use the {integrations-docs}/nginx[Nginx Elastic integration] and the {agent} to collect valuable metrics and logs from your nginx instances. Then, use built-in dashboards and tools like Logs Explorer in {kib} allow you to visualize and monitor your nginx data from one place. This data provides valuable insight into your nginx instances—for example: +Use the {integrations-docs}/nginx[nginx Elastic integration] and the {agent} to collect valuable metrics and logs from your nginx instances. Then, use built-in dashboards and tools like Logs Explorer in {kib} to visualize and monitor your nginx data from one place. This data provides valuable insight into your nginx instances—for example: * A spike in error logs for a certain resource may mean you have a deleted resource that is still needed. * Access logs can show when a service's peak times are, and, from this, when it might be best to perform things like maintenance. @@ -199,7 +199,7 @@ To open the nginx dashboard: The *Metrics Nginx overview* shows visual representations of total requests, processed requests, heartbeat/up, active connections, reading/writing/waiting rates, request rate, accepts and handled rates, and drops rate. [role="screenshot"] -image::images/nginx-metrics-dashboard.png[Nginx metrics dashboard, 75%] +image::images/nginx-metrics-dashboard.png[nginx metrics dashboard, 75%] [discrete] [[monitor-nginx-explore-logs]] @@ -229,7 +229,7 @@ image::images/nginx-logs-explorer.png[Logs Explorer showing nginx error logs] [discrete] [[monitor-nginx-logs-dashboard]] -==== Nginx logs dashboards +==== nginx logs dashboards The nginx integration has built-in dashboards that show the full picture of your nginx logs in one place. To open the nginx dashboards: From 11da181991c8d6438bab428bb0a73b0808e81ef8 Mon Sep 17 00:00:00 2001 From: mdbirnstiehl Date: Mon, 13 May 2024 09:39:44 -0500 Subject: [PATCH 4/6] review updates --- docs/en/observability/monitor-nginx-ml.asciidoc | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/en/observability/monitor-nginx-ml.asciidoc b/docs/en/observability/monitor-nginx-ml.asciidoc index 87dd7a7652..aefc973fad 100644 --- a/docs/en/observability/monitor-nginx-ml.asciidoc +++ b/docs/en/observability/monitor-nginx-ml.asciidoc @@ -18,9 +18,9 @@ The nginx ML module provides the following anomaly detection jobs: [[horizontal]] Low request rates (`low_request_rate_nginx`):: Uses the {ml-docs}/ml-count-functions.html#ml-count[`low_count`] function to detect abnormally low request rates. Abnormally low request rates might indicate that network issues or other issues are preventing requests from reaching the server. -Unusual source IPs - high request rates (`source_ip_request_rate_nginx`):: Uses the {ml-docs}/ml-count-functions.html#ml-count[`hight_count`] function to detect abnormally high request rates from individual IP addresses. Many requests from a single IP or small group of IPs might indicate something malicious like a DDoS attack where a large number of requests are sent to overwhelm the server and make it unavailable to users. +Unusual source IPs - high request rates (`source_ip_request_rate_nginx`):: Uses the {ml-docs}/ml-count-functions.html#ml-count[`hight_count`] function to detect abnormally high request rates from individual IP addresses. Many requests from a single IP or small group of IPs might indicate something malicious like a distributed denial of service (DDoS) attack where a large number of requests are sent to overwhelm the server and make it unavailable to users. Unusual source IPs - high distinct count of URLs (`source_ip_url_count_nginx`):: Uses the {ml-docs}/ml-count-functions.html#ml-distinct-count[`high_distinct_count`] function to detect individual IP addresses accessing abnormally high numbers of unique URLs. A single IP accessing many unique URLs might indicate something malicious like web scraping or an attempt to find sensitive data or vulnerabilities. -Unusual status code rates (`status_code_rate_nginx`):: Uses the {ml-docs}/ml-count-functions.html#ml-count[`count`] function to detect abnormal status code rates. A high rate of status codes could indicate problems with broken links, bad URLs, or unauthorized access attempts. A high rate of status codes could also point to server issues like limited resources or bugs in your code. +Unusual status code rates (`status_code_rate_nginx`):: Uses the {ml-docs}/ml-count-functions.html#ml-count[`count`] function to detect abnormal error status code rates. A high rate of status codes could indicate problems with broken links, bad URLs, or unauthorized access attempts. A high rate of status codes could also point to server issues like limited resources or bugs in your code. Unusual visitor rates (`visitor_rate_nginx`):: Uses the {ml-docs}/ml-count-functions.html#ml-nonzero-count[`non_zero_count`] function to detect abnormal visitor rates. High visitor rates could indicate something malicious like a DDoS attack. Low visitor rates could indicate issues with access to the server. @@ -46,7 +46,7 @@ Add the nginx ML jobs from the nginx integration to start using anomaly detectio . Under *Use preconfigured jobs*, select the *Nginx access logs* card. . Select *Create jobs*. -Back on the *Jobs* page, you should see the nginx anomaly detection jobs—`low_request_rate_nginx`, `source_ip_request_rate_nginx`, s`ource_ip_url_count_nginx`, `status_code_rate_nginx`, and `visitor_rate_nginx`. +Back on the *Jobs* page, you should see the nginx anomaly detection jobs—`low_request_rate_nginx`, `source_ip_request_rate_nginx`, `source_ip_url_count_nginx`, `status_code_rate_nginx`, and `visitor_rate_nginx`. [discrete] [[monitor-nginx-ml-explore]] From 606ec165f3b77ee7d83987ee27932adafc35ff99 Mon Sep 17 00:00:00 2001 From: mdbirnstiehl Date: Thu, 30 May 2024 10:00:01 -0500 Subject: [PATCH 5/6] update kibana tasks --- docs/en/observability/monitor-nginx-ml.asciidoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/en/observability/monitor-nginx-ml.asciidoc b/docs/en/observability/monitor-nginx-ml.asciidoc index aefc973fad..735bf727b3 100644 --- a/docs/en/observability/monitor-nginx-ml.asciidoc +++ b/docs/en/observability/monitor-nginx-ml.asciidoc @@ -40,13 +40,13 @@ Refer to {ml-docs}/setup.html[Set up ML features]. Add the nginx ML jobs from the nginx integration to start using anomaly detection: -. From the main {kib} menu, go to *Machine Learning* → *Jobs*. +. From the main {kib} menu, go to *Machine Learning*. Under *Anomaly Detection*, select *Jobs*. . Select *Create job*. . In the search bar, enter *nginx* and select *Nginx access logs [Logs Nginx]*. . Under *Use preconfigured jobs*, select the *Nginx access logs* card. . Select *Create jobs*. -Back on the *Jobs* page, you should see the nginx anomaly detection jobs—`low_request_rate_nginx`, `source_ip_request_rate_nginx`, `source_ip_url_count_nginx`, `status_code_rate_nginx`, and `visitor_rate_nginx`. +Back on the *Anomaly Detection Jobs* page, you should see the nginx anomaly detection jobs—`low_request_rate_nginx`, `source_ip_request_rate_nginx`, `source_ip_url_count_nginx`, `status_code_rate_nginx`, and `visitor_rate_nginx`. [discrete] [[monitor-nginx-ml-explore]] From a3479a83feec5d5d84e5600ef7761b0c6904bf09 Mon Sep 17 00:00:00 2001 From: mdbirnstiehl Date: Thu, 30 May 2024 15:17:19 -0500 Subject: [PATCH 6/6] review updates --- docs/en/observability/monitor-nginx-ml.asciidoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/en/observability/monitor-nginx-ml.asciidoc b/docs/en/observability/monitor-nginx-ml.asciidoc index 735bf727b3..e07e24808d 100644 --- a/docs/en/observability/monitor-nginx-ml.asciidoc +++ b/docs/en/observability/monitor-nginx-ml.asciidoc @@ -2,7 +2,7 @@ [[monitor-nginx-ml]] == Part 5: Find anomalies in your nginx access logs -The {integrations-docs}/nginx[Nginx Elastic integration] to help find unusual activity in your nginx access logs. +Use the {integrations-docs}/nginx[nginx Elastic integration] machine learning (ML) module to help find unusual activity in your nginx access logs. Monitoring anomalies in your access logs helps you detect: * security threats @@ -24,7 +24,7 @@ Unusual status code rates (`status_code_rate_nginx`):: Uses the {ml-docs}/ml-cou Unusual visitor rates (`visitor_rate_nginx`):: Uses the {ml-docs}/ml-count-functions.html#ml-nonzero-count[`non_zero_count`] function to detect abnormal visitor rates. High visitor rates could indicate something malicious like a DDoS attack. Low visitor rates could indicate issues with access to the server. -Refer to {integrations-docs}/nginx#ml-modules[nginx integration ML modules] for more on the jobs and the ML module manifest. +NOTE: These anomaly detection jobs are available when you have data that matches the query specified in the ML module manifest. Users not following this tutorial can refer to {integrations-docs}/nginx#ml-modules[nginx integration ML modules] for more about the ML module manifest. [discrete] [[monitor-nginx-ml-prereqs]]