From 34aeadfeaf0ffa410a78e1fab2c45c5ca4b9450f Mon Sep 17 00:00:00 2001 From: Kaise Cheng Date: Thu, 17 Oct 2024 17:30:00 +0100 Subject: [PATCH 1/2] remove leftover azure file --- docs/static/azure-module.asciidoc | 532 ------------------------------ 1 file changed, 532 deletions(-) delete mode 100644 docs/static/azure-module.asciidoc diff --git a/docs/static/azure-module.asciidoc b/docs/static/azure-module.asciidoc deleted file mode 100644 index 14c9caea3a8..00000000000 --- a/docs/static/azure-module.asciidoc +++ /dev/null @@ -1,532 +0,0 @@ -[role="xpack"] -[[azure-module]] -=== Azure Module -experimental[] - -++++ -Azure Module (deprecated) -++++ - -deprecated[7.8.0, "We recommend using the Azure modules in {filebeat-ref}/filebeat-module-azure.html[{Filebeat}] and {metricbeat-ref}/metricbeat-module-azure.html[{metricbeat}], which are compliant with the {ecs-ref}/index.html[Elastic Common Schema (ECS)]"] - -The https://azure.microsoft.com/en-us/overview/what-is-azure/[Microsoft Azure] -module in Logstash helps you easily integrate your Azure activity logs and SQL -diagnostic logs with the Elastic Stack. - -image::static/images/azure-flow.png["Azure Work Flow",width="80%"] - -You can monitor your Azure cloud environments and SQL DB deployments with -deep operational insights across multiple Azure subscriptions. You can explore -the health of your infrastructure in real-time, accelerating root cause analysis -and decreasing overall time to resolution. The Azure module helps you: - -* Analyze infrastructure changes and authorization activity -* Identify suspicious behaviors and potential malicious actors -* Perform root-cause analysis by investigating user activity -* Monitor and optimize your SQL DB deployments. - -The Azure module uses the -{logstash-ref}/plugins-inputs-azure_event_hubs.html[Logstash Azure Event Hubs -input plugin] to consume data from Azure Event Hubs. The module taps directly into the -Azure dashboard, parses and indexes events into Elasticsearch, and installs a -suite of {kib} dashboards to help you start exploring your data immediately. - -[[azure-dashboards]] -==== Dashboards - -These {kib} dashboards are available and ready for you to use. You can use them as they are, or tailor them to meet your needs. - -===== Infrastructure activity monitoring - -* *Overview*. Top-level view into your Azure operations, including info about users, resource groups, service health, access, activities, and alerts. - -* *Alerts*. Alert info, including activity, alert status, and alerts heatmap - -* *User Activity*. Info about system users, their activity, and requests. - -===== SQL Database monitoring - -* *SQL DB Overview*. Top-level view into your SQL Databases, including counts for databases, servers, resource groups, and subscriptions. - -* *SQL DB Database View*. Detailed info about each SQL Database, including wait time, errors, DTU and storage utilization, size, and read and write input/output. - -* *SQL DB Queries*. Info about SQL Database queries and performance. - -[[azure-module-prereqs]] -==== Prerequisites - -Azure Monitor enabled with Azure Event Hubs and the Elastic Stack are required -for this module. - -[[azure-elk-prereqs]] -===== Elastic prerequisites - -The instructions below assume that you have {ls}, {es}, and {kib} running locally. -You can also run {ls}, {es}, and {kib} on separate hosts. - -The Elastic Stack version 6.4 (or later) is required for this module. - -The Azure module uses the `azure_event_hubs` input plugin to consume logs and -metrics from your Azure environment. It is installed by default with {ls} 6.4 -(or later). Basic understanding of the plugin and options is helpful when you -set up the Azure module. -See the {logstash-ref}/plugins-inputs-azure_event_hubs.html[azure_event_hubs input -plugin documentation] for more information. - -Elastic products are https://www.elastic.co/downloads[available to download] and -easy to install. - -[[azure-prereqs]] -===== Azure prerequisites - -Azure Monitor should be configured to stream logs to one or more Event Hubs. -Logstash will need to access these Event Hubs instances to consume your Azure logs and metrics. -See <> at the end of this topic for links to Microsoft Azure documentation. - -[[configuring-azure]] -==== Configure the module - -Specify <> for the Logstash Azure module in the -`logstash.yml` configuration file. - -* *Basic configuration.* You can use the `logstash.yml` file to configure inputs from multiple Event Hubs that share the same configuration. -Basic configuration is recommended for most use cases. - -* *Advanced configuration.* The advanced configuration is available for deployments where different Event Hubs -require different configurations. The `logstash.yml` file holds your settings. Advanced configuration is not necessary or -recommended for most use cases. - -See the {logstash-ref}/plugins-inputs-azure_event_hubs.html[azure_event_hubs input plugin -documentation] for more information about basic and advanced configuration -models. - -[[basic-config-sample]] -===== Basic configuration sample - -The configuration in the `logstash.yml` file is shared between Event Hubs. -Basic configuration is recommended for most use cases - -["source","shell",subs="attributes"] ------ -modules: - - name: azure - var.elasticsearch.hosts: localhost:9200 - var.kibana.host: localhost:5601 - var.input.azure_event_hubs.consumer_group: "logstash" <1> - var.input.azure_event_hubs.storage_connection: "DefaultEndpointsProtocol=https;AccountName=instance1..." <2> - var.input.azure_event_hubs.threads: 9 <3> - var.input.azure_event_hubs.event_hub_connections: - - "Endpoint=sb://...EntityPath=insights-operational-logs" <4> - - "Endpoint=sb://...EntityPath=insights-metrics-pt1m" <5> - - "Endpoint=sb://...EntityPath=insights-logs-blocks" - - "Endpoint=sb://...EntityPath=insights-logs-databasewaitstatistics" - - "Endpoint=sb://...EntityPath=insights-logs-errors" - - "Endpoint=sb://...EntityPath=insights-logs-querystoreruntimestatistics" - - "Endpoint=sb://...EntityPath=insights-logs-querystorewaitstatistics" - - "Endpoint=sb://...EntityPath=insights-logs-timeouts" ------ -<1> The `consumer_group` (optional) is highly recommended. See <>. -<2> The `storage_connection` (optional) sets the Azure Blob Storage connection for tracking processing state for Event Hubs when scaling out a deployment with multiple Logstash instances. See <> for additional details. -<3> See <> for guidelines on choosing an appropriate number of threads. -<4> This connection sets up the consumption of Activity Logs. By default, Azure Monitor uses the `insights-operational-logs` Event Hub name. Make sure this matches the name of the Event Hub specified for Activity Logs. -<5> This connection and the ones below set up the consumption of SQL DB diagnostic logs and metrics. By default, Azure Monitor uses all these different Event Hub names. - -The basic configuration requires the `var.input.azure_event_hubs.` prefix -before a configuration option. -Notice the notation for the `threads` option. - -[[adv-config-sample]] -===== Advanced configuration sample - -Advanced configuration in the `logstash.yml` file supports Event Hub specific -options. Advanced configuration is available for more granular tuning of -threading and Blob Storage usage across multiple Event Hubs. Advanced -configuration is not necessary or recommended for most use cases. Use it only if -it is required for your deployment scenario. - -You must define the `header` array with `name` in the first position. You can -define other options in any order. The per Event Hub configuration takes -precedence. Any values not defined per Event Hub use the global config value. - -In this example `threads`, `consumer_group`, and `storage_connection` will be -applied to each of the configured Event Hubs. Note that `decorate_events` is -defined in both the global and per Event Hub configuration. The per Event Hub -configuration takes precedence, and the global configuration is effectively -ignored when the per Event Hub setting is present. - -["source","shell",subs="attributes"] ------ -modules: - - name: azure - var.elasticsearch.hosts: localhost:9200 - var.kibana.host: localhost:5601 - var.input.azure_event_hubs.decorate_events: true <1> - var.input.azure_event_hubs.threads: 9 <2> - var.input.azure_event_hubs.consumer_group: "logstash" - var.input.azure_event_hubs.storage_connection: "DefaultEndpointsProtocol=https;AccountName=instance1..." - var.input.azure_event_hubs.event_hubs: - - ["name", "initial_position", "storage_container", "decorate_events", "event_hub_connection"] <3> - - ["insights-operational-logs", "TAIL", "activity-logs1", "true", "Endpoint=sb://...EntityPath=insights-operational-logs"] - - ["insights-operational-logs", "TAIL", "activity_logs2", "true", "Endpoint=sb://...EntityPath=insights-operational-logs"] <4> - - ["insights-metrics-pt1m", "TAIL", "dbmetrics", "true", "Endpoint=sb://...EntityPath=insights-metrics-pt1m"] - - ["insights-logs-blocks", "TAIL", "dbblocks", "true", "Endpoint=sb://...EntityPath=insights-logs-blocks"] - - ["insights-logs-databasewaitstatistics", "TAIL", "dbwaitstats", "false", "Endpoint=sb://...EntityPath=insights-logs-databasewaitstatistics"] - - ["insights-logs-errors", "HEAD", "dberrors", "true", "Endpoint=sb://...EntityPath=insights-logs-errors" - - ["insights-logs-querystoreruntimestatistics", "TAIL", "dbstoreruntime", "true", "Endpoint=sb://...EntityPath=insights-logs-querystoreruntimestatistics"] - - ["insights-logs-querystorewaitstatistics", "TAIL", "dbstorewaitstats", "true", "Endpoint=sb://...EntityPath=insights-logs-querystorewaitstatistics"] - - ["insights-logs-timeouts", "TAIL", "dbtimeouts", "true", "Endpoint=sb://...EntityPath=insights-logs-timeouts"] ------ -<1> You can specify global Event Hub options. They will be overridden by any configurations specified in the event_hubs option. -<2> See <> for guidelines on choosing an appropriate number of threads. -<3> The header array must be defined with name in the first position. Other options can be defined in any order. The per Event Hub configuration takes precedence. Any values not defined per Event Hub use the global config value. -<4> This enables consuming from a second Activity Logs Event Hub that uses a different Blob Storage container. This is necessary to avoid the offsets from the first insights-operational-logs from overwriting the offsets for the second insights-operational-logs. - -The advanced configuration doesn't require a prefix before a per Event Hub -configuration option. Notice the notation for the `initial_position` option. - -[[scaling-blob]] -===== Scale Event Hub consumption - -An https://azure.microsoft.com/en-us/services/storage/blobs[Azure Blob Storage -account] is an essential part of Azure-to-Logstash configuration. -It is required for users who want to scale out multiple {ls} instances to consume from Event Hubs. - -A Blob Storage account is a central location that enables multiple instances of -{ls} to work together to process events. It records the -offset (location) of processed events. On restart, {ls} resumes processing -exactly where it left off. - -Configuration notes: - -* A Blob Storage account is highly recommended for use with this module, and is -likely required for production servers. -* The `storage_connection` option passes the blob storage connection string. -* Configure all {ls} instances to use the same `storage_connection` to get the -benefits of shared processing. - -Sample Blob Storage connection string: - -[source,text] ----- -DefaultEndpointsProtocol=https;AccountName=logstash;AccountKey=ETOPnkd/hDAWidkEpPZDiXffQPku/SZdXhPSLnfqdRTalssdEuPkZwIcouzXjCLb/xPZjzhmHfwRCGo0SBSw==;EndpointSuffix=core.windows.net ----- - -Find the connection string to Blob Storage here: -https://portal.azure.com[Azure Portal]`-> Blob Storage account -> Access keys`. - -[[azure_best_practices]] -===== Best practices - -Here are some guidelines to help you achieve a successful deployment, and avoid -data conflicts that can cause lost events. - -* <> -* <> -* <> - -[[azure-bp-group]] -====== Create a {ls} consumer group - -Create a new consumer group specifically for {ls}. Do not use the $default or -any other consumer group that might already be in use. Reusing consumer groups -among non-related consumers can cause unexpected behavior and possibly lost -events. All {ls} instances should use the same consumer group so that they can -work together for processing events. - -[[azure-bp-multihub]] -====== Avoid overwriting offset with multiple Event Hubs - -The offsets (position) of the Event Hubs are stored in the configured Azure Blob -store. The Azure Blob store uses paths like a file system to store the offsets. -If the paths between multiple Event Hubs overlap, then the offsets may be stored -incorrectly. - -To avoid duplicate file paths, use the advanced configuration model and make -sure that at least one of these options is different per Event Hub: - -* storage_connection -* storage_container (defaults to Event Hub name if not defined) -* consumer_group - -[[azure-bp-threads]] -====== Set number of threads correctly - -By default, the number of threads used to service all event hubs is `16`. And -while this may be sufficient for most use cases, throughput may be improved by -refining this number. When servicing a large number of partitions across one or -more event hubs, setting a higher value may result in improved performance. The -maximum number of threads is not strictly bound by the total number of -partitions being serviced, but setting the value much higher than that may mean -that some threads are idle. - -NOTE: The number of threads *must* be greater than or equal to the number of Event -hubs plus one. - -NOTE: Threads are currently available only as a global setting across all event hubs -in a single `azure_event_hubs` input definition. However if your configuration -includes multiple `azure_event_hubs` inputs, the threads setting applies -independently to each. - -**Sample scenarios:** - -* Event Hubs = 4. Partitions on each Event Hub = 3. -Minimum threads is 5 (4 Event Hubs plus one). Maximum threads is 13 (4 Event -Hubs times 3 partitions plus one). -* If you're collecting activity logs from only one specified event hub instance, -then only 2 threads (1 Event Hub plus one) are required. - -[[azure-module-setup]] -==== Set up and run the module - -Be sure that the `logstash.yml` file is <>. - -===== First time setup - -Run this command from the Logstash directory: - -["source","shell",subs="attributes"] ------ -bin/logstash --setup ------ - -The `--modules azure` option starts a Logstash pipeline for ingestion from Azure -Event Hubs. The `--setup` option creates an `azure-*` index pattern in -Elasticsearch and imports Kibana dashboards and visualizations. - -===== Subsequent starts - -Run this command from the Logstash directory: - -["source","shell",subs="attributes"] ------ -bin/logstash ------ - -NOTE: The `--setup` option is intended only for first-time setup. If you include -`--setup` on subsequent runs, your existing Kibana dashboards will be -overwritten. - -[[exploring-data-azure]] -==== Explore your data -When the Logstash Azure module starts receiving events, you can begin using the -packaged Kibana dashboards to explore and visualize your data. - -To explore your data with Kibana: - -. Open a browser to http://localhost:5601[http://localhost:5601] (username: - "elastic"; password: "{pwd}") -. Click *Dashboard*. -. Click *[Azure Monitor] Overview*. - -[[azure_config_options]] -==== Configuration options - -NOTE: All Event Hubs options are common to both basic and advanced -configurations, with the following exceptions. The basic configuration uses -`event_hub_connections` to support multiple connections. The advanced -configuration uses `event_hubs` and `event_hub_connection` (singular). - -[[azure_event_hubs]] -===== `event_hubs` -* Value type is <> -* No default value -* Ignored for basic and command line configuration -* Required for advanced configuration - -Defines the per Event Hubs configuration for the <>. - -The advanced configuration uses `event_hub_connection` instead of `event_hub_connections`. -The `event_hub_connection` option is defined per Event Hub. - -[[azure_event_hub_connections]] -===== `event_hub_connections` -* Value type is <> -* No default value -* Required for basic and command line configuration -* Ignored for advanced configuration - -List of connection strings that identifies the Event Hubs to be read. Connection -strings include the EntityPath for the Event Hub. - - -[[azure_checkpoint_interval]] -===== `checkpoint_interval` -* Value type is <> -* Default value is `5` seconds -* Set to `0` to disable. - -Interval in seconds to write checkpoints during batch processing. Checkpoints -tell {ls} where to resume processing after a restart. Checkpoints are -automatically written at the end of each batch, regardless of this setting. - -Writing checkpoints too frequently can slow down processing unnecessarily. - - -[[azure_consumer_group]] -===== `consumer_group` -* Value type is <> -* Default value is `$Default` - -Consumer group used to read the Event Hub(s). Create a consumer group -specifically for Logstash. Then ensure that all instances of Logstash use that -consumer group so that they can work together properly. - - -[[azure_decorate_events]] -===== `decorate_events` - -* Value type is <> -* Default value is `false` - -Adds metadata about the Event Hub, including `Event Hub name`, `consumer_group`, -`processor_host`, `partition`, `offset`, `sequence`, `timestamp`, and `event_size`. - - -[[azure_initial_position]] -===== `initial_position` -* Value type is <> -* Valid arguments are `beginning`, `end`, `look_back` -* Default value is `beginning` - -When first reading from an Event Hub, start from this position: - -* `beginning` reads all pre-existing events in the Event Hub -* `end` does not read any pre-existing events in the Event Hub -* `look_back` reads `end` minus a number of seconds worth of pre-existing events. -You control the number of seconds using the `initial_position_look_back` option. - -If `storage_connection` is set, the `initial_position` value is used only -the first time Logstash reads from the Event Hub. - - -[[azure_initial_position_look_back]] -===== `initial_position_look_back` -* Value type is <> -* Default value is `86400` -* Used only if `initial_position` is set to `look-back` - -Number of seconds to look back to find the initial position for pre-existing -events. This option is used only if `initial_position` is set to `look_back`. If -`storage_connection` is set, this configuration applies only the first time {ls} -reads from the Event Hub. - - -[[azure_max_batch_size]] -===== `max_batch_size` - -* Value type is <> -* Default value is `125` - -Maximum number of events retrieved and processed together. A checkpoint is -created after each batch. Increasing this value may help with performance, but -requires more memory. - - -[[azure_storage_connection]] -===== `storage_connection` -* Value type is <> -* No default value - -Connection string for blob account storage. Blob account storage persists the -offsets between restarts, and ensures that multiple instances of Logstash -process different partitions. -When this value is set, restarts resume where processing left off. -When this value is not set, the `initial_position` value is used on every restart. - -We strongly recommend that you define this value for production environments. - - -[[azure_storage_container]] -===== `storage_container` -* Value type is <> -* Defaults to the Event Hub name if not defined - -Name of the storage container used to persist offsets and allow multiple instances of {ls} -to work together. - -To avoid overwriting offsets, you can use different storage containers. This is -particularly important if you are monitoring two Event Hubs with the same name. -You can use the advanced configuration model to configure different storage -containers. - - -[[azure_threads]] -===== `threads` -* Value type is <> -* Minimum value is `2` -* Default value is `16` - -Total number of threads used to process events. The value you set here applies -to all Event Hubs. Even with advanced configuration, this value is a global -setting, and can't be set per event hub. - -The number of threads should be the number of Event Hubs plus one or more. -See <> for more information. - - -include::shared-module-options.asciidoc[] - -==== Azure module schema - -This module reads data from the Azure Event Hub and adds some additional structure to the data for Activity Logs and SQL Diagnostics. The original data is always preserved and any data added or parsed will be namespaced under 'azure'. For example, 'azure.subscription' may have been parsed from a longer more complex URN. - -[cols="<,<,<",options="header",] -|======================================================================= -|Name |Description|Notes - -|azure.subscription |Azure subscription from which this data originates. |Some Activity Log events may not be associated with a subscription. -|azure.group |Primary type of data. |Current values are either 'activity_log' or 'sql_diagnostics' -|azure.category* |Secondary type of data specific to group from which the data originated | -|azure.provider |Azure provider | -|azure.resource_group |Azure resource group | -|azure.resource_type |Azure resource type | -|azure.resource_name |Azure resource name | -|azure.database |Azure database name, for display purposes |SQL Diagnostics only -|azure.db_unique_id |Azure database name that is guaranteed to be unique |SQL Diagnostics only -|azure.server |Azure server for the database |SQL Diagnostics only -|azure.server_and_database |Azure server and database combined |SQL Diagnostics only -|azure.metadata |Any @metadata added by the plugins, for example var.input.azure_event_hubs.decorate_events: true -|======================================================================= - -Notes: - -* Activity Logs can have the following categories: Administrative, ServiceHealth, Alert, Autoscale, Security -* SQL Diagnostics can have the following categories: Metric, Blocks, Errors, Timeouts, QueryStoreRuntimeStatistics, QueryStoreWaitStatistics, DatabaseWaitStatistics, SQLInsights - -Microsoft documents Activity log schema -https://docs.microsoft.com/en-us/azure/monitoring-and-diagnostics/monitoring-activity-log-schema[here]. -The SQL Diagnostics data is documented -https://docs.microsoft.com/en-us/azure/sql-database/sql-database-metrics-diag-logging[here]. -Elastic does not own these data models, and as such, cannot make any -assurances of information accuracy or passivity. - -===== Special note - Properties field - -Many of the logs contain a `properties` top level field. This is often where the -most interesting data lives. There is not a fixed schema between log types for -properties fields coming from different sources. - -For example, one log may have -`properties.type` where one log sets this a String type and another sets this an -Integer type. To avoid mapping errors, the original properties field is moved to -`__properties.`. -For example -`properties.type` may end up as `sql_diagnostics_Errors_properties.type` or -`activity_log_Security_properties.type` depending on the group/category where -the event originated. - -[[azure-production]] -==== Deploying the module in production - -Use security best practices to secure your configuration. -See {ref}/secure-cluster.html[Secure a cluster] for details and recommendations. - -[[azure-resources]] -==== Microsoft Azure resources - -Microsoft is the best source for the most up-to-date Azure information. - -* https://docs.microsoft.com/en-us/azure/monitoring-and-diagnostics/monitoring-overview-azure-monitor[Overview of Azure Monitor] -* https://docs.microsoft.com/en-us/azure/sql-database/sql-database-metrics-diag-logging[Azure SQL Database metrics and diagnostics logging] -* https://docs.microsoft.com/en-us/azure/monitoring-and-diagnostics/monitoring-stream-activity-logs-event-hubs[Stream the Azure Activity Log to Event Hubs] - From 7b5ace7db3443117ebaf1d28ed79b492a0ff25d6 Mon Sep 17 00:00:00 2001 From: Kaise Cheng Date: Thu, 17 Oct 2024 17:30:30 +0100 Subject: [PATCH 2/2] update module example --- docs/static/modules.asciidoc | 24 +++--------------------- 1 file changed, 3 insertions(+), 21 deletions(-) diff --git a/docs/static/modules.asciidoc b/docs/static/modules.asciidoc index 8b596e96a0f..67e255af6de 100644 --- a/docs/static/modules.asciidoc +++ b/docs/static/modules.asciidoc @@ -52,23 +52,6 @@ dashboards and visualizations. Running `--setup` is a one-time setup step. Omit this option for subsequent runs of the module to avoid overwriting existing Kibana dashboards. -For example, the following command runs the Netflow module with the default -settings, and sets up the netflow index pattern and dashboards: - -[source,shell] ----- -bin/logstash --modules netflow --setup ----- - -The following command runs the Netflow module and overrides the Elasticsearch -`host` setting. Here it's assumed that you've already run the setup step. - -[source,shell] ----- -bin/logstash --modules netflow -M "netflow.var.elasticsearch.host=es.mycloud.com" ----- - - [discrete] [[configuring-logstash-modules]] === Configuring modules @@ -91,14 +74,13 @@ settings. For example: [source,shell] ---- modules: -- name: netflow +- name: arcsight var.elasticsearch.hosts: "es.mycloud.com" var.elasticsearch.username: "foo" var.elasticsearch.password: "password" var.kibana.host: "kb.mycloud.com" var.kibana.username: "foo" var.kibana.password: "password" - var.input.tcp.port: 5606 ---- For a list of available module settings, see the documentation for the module. @@ -150,7 +132,7 @@ Elasticsearch `host` setting and the `udp.port` setting: [source,shell] ---- -bin/logstash --modules netflow -M "netflow.var.input.udp.port=3555" -M "netflow.var.elasticsearch.hosts=my-es-cloud" +bin/logstash --modules arcsight -M "arcsight.var.elasticsearch.hosts=my-es-cloud" ---- Any settings defined in the command line are ephemeral and will not persist across @@ -200,7 +182,7 @@ These settings can be also specified at the command line, like this: ["source","sh",subs="attributes,callouts"] ---- -bin/logstash --modules netflow -M "netflow.var.input.udp.port=3555" --cloud.id --cloud.auth +bin/logstash --modules arcsight --cloud.id --cloud.auth ---- NOTE: When working with modules, use the dot notation to specify cloud.id and