Merge branch 'main' into cloudwatch-firehose

.github/CODEOWNERS

@@ -3,3 +3,4 @@
 # For more info, see https://help.github.com/articles/about-codeowners/
 
 * @elastic/obs-docs
+/.github/workflows/co-docs-builder.yml @elastic/docs-engineering

.github/workflows/co-docs-builder.yml

@@ -0,0 +1,31 @@
+name: Elastic docs
+
+on:
+  pull_request_target:
+    # The paths property can be omitted entirely if the repo is mainly used for docs. Leaving it in can result in builds that 
+    # have branch protection checks in place lose the ability to merge because the workflow is not starting. If this property 
+    # is included, please ensure that branch protection checks are disabled for the repo. 
+    paths:
+      # Preface with your docs dir if you need further specificity (optional)
+      - 'docs/en/serverless/**.mdx'
+      - 'docs/en/serverless/**.docnav.json'
+      - 'docs/en/serverless/**.docapi.json'
+      - 'docs/en/serverless/**.devdocs.json'
+      - 'docs/en/serverless/**.jpg'
+      - 'docs/en/serverless/**.jpeg'
+      - 'docs/en/serverless/**.svg'
+      - 'docs/en/serverless/**.png'
+      - 'docs/en/serverless/**.gif'
+    types: [closed, opened, synchronize, labeled]
+
+jobs:
+  publish:
+    if: contains(github.event.pull_request.labels.*.name, 'ci:doc-build')
+    uses: elastic/workflows/.github/workflows/docs-elastic-co-publish.yml@main
+    with:
+      subdirectory: 'docs/en/serverless/'
+    secrets:
+      VERCEL_GITHUB_TOKEN: ${{ secrets.VERCEL_GITHUB_TOKEN_PUBLIC }}
+      VERCEL_TOKEN: ${{ secrets.VERCEL_TOKEN_PUBLIC }}
+      VERCEL_ORG_ID: ${{ secrets.VERCEL_ORG_ID_PUBLIC }}
+      VERCEL_PROJECT_ID_DOCS_CO: ${{ secrets.VERCEL_PROJECT_ID_DOCS_CO_PUBLIC }}

README.md

@@ -8,32 +8,70 @@ Within this repo, the `/docs/en/` directory is structured as follows:
 
 | Directory             | Description |
 | --------------------- | ----------- |
-| __integrations__      | Contains the source files for the [Integrations Developer Guide](https://www.elastic.co/guide/en/integrations-developer/current/index.html)
+| __integrations__      | Contains the source files for the [Integrations Developer Guide](https://www.elastic.co/guide/en/integrations-developer/current/index.html).
 | __observability__     | Contains the source files for the [Observability Guide](https://www.elastic.co/guide/en/observability/current/index.html), which includes content for APM, Logs, Metrics, Synthetics, User experience, and Uptime.|
+| __serverless__ | Contains the source files for the [Elastic Observability Serverless docs](https://docs.elastic.co/serverless/observability/what-is-observability-serverless).
 | __shared__ | Contains the source files for shared Observability content.|
 | __templates__ | Contains content templates.|
 
-## Reviews
-
-All documentation pull requests automatically add the **[@obs-docs](https://github.com/orgs/elastic/teams/obs-docs)** team as a reviewer.
+## Contributing
 
-## Backporting
+If you find any bugs in our documentation, or want to request an enhancement, then you can open an issue using our template. We also welcome contributions in the form of PRs. Before you submit a PR, make sure that you have signed our [Contributor License Agreement](https://www.elastic.co/contributor-agreement/).
 
-Pull requests should be tagged with the target version of the Elastic Stack along with any relevant backport labels. In general, we only backport documentation changes to [live stack versions](https://github.com/elastic/docs/blob/master/conf.yaml#L74). For manual backports, we recommend using the [backport tool](https://github.com/sqren/backport) to easily open backport PRs. If you need help, ping **[@obs-docs](https://github.com/orgs/elastic/teams/obs-docs)** and we'd be happy to handle the backport process for you.
+Contributing directly to the docs works differently across the doc sets in this repo.
 
+### Observability Guide
 
-## Build
+The source files for the Observability Guide are written in [AsciiDoc](https://docs.asciidoctor.org/asciidoc/latest/) and are built using [elastic/docs](https://github.com/elastic/docs).
 
-To build the docs:
+To build the docs locally:
 
 1. Check out the `elastic/docs` repository, along with any repositories that contain source files.
-
 2. Run the `build_docs` script, passing in the path to the `index.asciidoc` and resource paths to other repos that contain source files. For example, to build the Observability Guide and open it in the browser, run:
+    ```
+    ../docs/build_docs --doc ./docs/en/observability/index.asciidoc --chunk 3 --resource ../apm-server --resource ../ingest-docs/docs --open
+    ```
 
-```
-../docs/build_docs --doc ./docs/en/observability/index.asciidoc --chunk 3 --resource ../apm-server --resource ../ingest-docs/docs --open
-```
+The above command assumes that this repo, [elastic/docs](https://github.com/elastic/docs), [elastic/ingest-docs](https://github.com/elastic/ingest-docs), and [elastic/apm-server](https://github.com/elastic/apm-server) are checked out into the same parent directory.
+
+If you prefer to use aliases, you can load the [elastic/docs/doc_build_aliases.sh file](https://github.com/elastic/docs/blob/master/doc_build_aliases.sh), which has the resources defined for you.
+
+### Elastic Observability Serverless docs
+
+The Elastic Observability Serverless docs use a custom syntax written in [MDX](https://mdxjs.com/). In many cases, you only need to know plain Markdown to contribute. We'll add a public component reference and additional contribution guidelines in future. Elasticians can refer to our [internal syntax reference](https://docs.elastic.dev/docsmobile/syntax).
+
+### Integrations Developer Guide
+
+The source files for the Integrations Developer Guide are written in [AsciiDoc](https://docs.asciidoctor.org/asciidoc/latest/) and are built using [elastic/docs](https://github.com/elastic/docs).
+
+To build the docs locally:
+
+1. Check out the `elastic/docs` repository, along with any repositories that contain source files.
+2. Run the `build_docs` script, passing in the path to the `index.asciidoc` and resource paths to other repos that contain source files. For example, to build the Observability Guide and open it in the browser, run:
+    ```
+    ../docs/build_docs --doc ./docs/en/integrations/index.asciidoc --resource=../package-spec/versions --chunk 1 --open
+    ```
 
-The above command assumes that [elastic/docs](https://github.com/elastic/docs), [elastic/ingest-docs](https://github.com/elastic/ingest-docs), and [elastic/apm-server](https://github.com/elastic/apm-server) are checked out into the same parent directory.
+The above command assumes that this repo, [elastic/docs](https://github.com/elastic/docs), and [elastic/package-spec](https://github.com/elastic/package-spec), are checked out into the same parent directory.
 
 If you prefer to use aliases, you can load the [elastic/docs/doc_build_aliases.sh file](https://github.com/elastic/docs/blob/master/doc_build_aliases.sh), which has the resources defined for you.
+
+## Backporting
+
+Backporting works differently across the doc sets in this repo.
+
+### Observability Guide
+
+Pull requests should be tagged with the target version of the Elastic Stack along with any relevant backport labels. In general, we only backport documentation changes to [live stack versions](https://github.com/elastic/docs/blob/master/conf.yaml#L74). For manual backports, we recommend using the [backport tool](https://github.com/sqren/backport) to easily open backport PRs. If you need help, ping **[@obs-docs](https://github.com/orgs/elastic/teams/obs-docs)** and we'd be happy to handle the backport process for you.
+
+### Elastic Observability Serverless docs
+
+Serverless docs are not versioned, and should never be backported. All changes should be made to the `main` branch.
+
+### Integrations Developer Guide
+
+The Integrations Developer Guide is not versioned, and should never be backported.  All changes should be made to the `main` branch.
+
+## Reviews
+
+All documentation pull requests automatically add the **[@obs-docs](https://github.com/orgs/elastic/teams/obs-docs)** team as a reviewer.

docs/en/observability/apm/data-model.asciidoc

@@ -357,9 +357,9 @@ You can filter and group by these dimensions (some of which are optional, for ex
 * `host.name`: The user-defined name of the host or the detected hostname of the service that served the transaction
 * `host.os.platform`: The platform name of the service that served the transaction, for example `linux`
 * `kubernetes.pod.name`: The name of the Kubernetes pod running the service that served the transaction
-* `labels`: Key-value object containing string labels set globally by the APM agents.
+* `labels`: Key-value object containing string labels set globally by the APM agents. This dimension is not present for RUM agents.
 * `metricset.interval`: A string with the aggregation interval the metricset represents.
-* `numeric_labels`: Key-value object containing numeric labels set globally by the APM agents.
+* `numeric_labels`: Key-value object containing numeric labels set globally by the APM agents. This dimension is not present for RUM agents.
 * `service.environment`: The environment of the service that served the transaction
 * `service.language.name`: The language name of the service that served the transaction, for example `Go`
 * `service.language.version`: The language version of the service that served the transaction
@@ -411,9 +411,9 @@ These metric documents can be identified by searching for `metricset.name: servi
 You can filter and group by these dimensions:
 
 * `agent.name`: The name of the {apm-agent} that instrumented the operation, for example `java`
-* `labels`: Key-value object containing string labels set globally by the APM agents.
+* `labels`: Key-value object containing string labels set globally by the APM agents. This dimension is not present for RUM agents.
 * `metricset.interval`: A string with the aggregation interval the metricset represents.
-* `numeric_labels`: Key-value object containing numeric labels set globally by the APM agents.
+* `numeric_labels`: Key-value object containing numeric labels set globally by the APM agents. This dimension is not present for RUM agents.
 * `service.environment`: The environment of the service that made the request
 * `service.language.name`: The language name of the service that served the transaction, for example `Go`
 * `service.name`: The name of the service that made the request
@@ -456,9 +456,9 @@ You can filter and group by these dimensions:
 
 * `agent.name`: The name of the {apm-agent} that instrumented the operation, for example `java`
 * `event.outcome`: The outcome of the operation, for example `success`
-* `labels`: Key-value object containing string labels set globally by the APM agents.
+* `labels`: Key-value object containing string labels set globally by the APM agents. This dimension is not present for RUM agents.
 * `metricset.interval`: A string with the aggregation interval the metricset represents.
-* `numeric_labels`: Key-value object containing numeric labels set globally by the APM agents.
+* `numeric_labels`: Key-value object containing numeric labels set globally by the APM agents. This dimension is not present for RUM agents.
 * `service.environment`: The environment of the service that made the request
 * `service.language.name`: The language name of the service that served the transaction, for example `Go`
 * `service.name`: The name of the service that made the request
@@ -496,9 +496,9 @@ These metric documents can be identified by searching for `metricset.name: servi
 You can filter and group by these dimensions:
 
 * `agent.name`: The name of the {apm-agent} that instrumented the operation, for example `java`
-* `labels`: Key-value object containing string labels set globally by the APM agents.
+* `labels`: Key-value object containing string labels set globally by the APM agents. This dimension is not present for RUM agents.
 * `metricset.interval`: A string with the aggregation interval the metricset represents.
-* `numeric_labels`: Key-value object containing numeric labels set globally by the APM agents.
+* `numeric_labels`: Key-value object containing numeric labels set globally by the APM agents. This dimension is not present for RUM agents.
 * `service.environment`: The environment of the service that made the request
 * `service.language.name`: The language name of the service that served the transaction, for example `Go`
 * `service.name`: The name of the service that made the request
@@ -574,7 +574,9 @@ Once this limit is reached, any new combinations of `transaction.name`, `transac
 
 This issue can be resolved by increasing memory available to APM Server, or by ensuring that the dimensions do not use values
 that are based on parameters that can change. For example, user ids, product ids, order numbers, query parameters, etc.,
-should be stripped away from the dimensions.
+should be stripped away from the dimensions. For the same reason, avoid high cardinality global labels (`labels.\*` and `numeric_labels.*`).
+
+Aggregated metrics do not consider global labels from RUM agents in order to protect APM server from using too much memory.
 
 // This heading is linked to from the APM UI section in Kibana
 [[apm-data-model-metadata]]

docs/en/observability/cloud-monitoring/aws/monitor-amazon-intro.asciidoc

@@ -38,6 +38,8 @@ include::monitor-aws-vpc-flow-logs.asciidoc[leveloffset=+2]
 
 include::monitor-aws-cloudtrail-firehose.asciidoc[leveloffset=+2]
 
+include::monitor-aws-waf-firehose.asciidoc[leveloffset=+2]
+
 include::monitor-aws-cloudwatch-firehose.asciidoc[leveloffset=+2]
 
 include::monitor-aws-firehose-troubleshooting.asciidoc[leveloffset=+2]

docs/en/observability/cloud-monitoring/aws/monitor-aws-waf-firehose.asciidoc

@@ -0,0 +1,128 @@
+[[monitor-aws-waf-firehose]]
+= Monitor Web Application Firewall (WAF) logs
+
+++++
+<titleabbrev>Monitor WAF logs</titleabbrev>
+++++
+
+In this section, you'll learn how to send AWS WAF events from AWS to your {stack} using Amazon Data Firehose.
+
+You will go through the following steps:
+
+- Select a WAF-compatible resource (for example, a CloudFront distribution)
+- Create a delivery stream in Amazon Data Firehose
+- Create a web Access Control List (ACL) to generate WAF logs
+- Set up logging to forward the logs to the {stack} using a Firehose stream
+- Visualize your WAF logs in {kib}
+
+[discrete]
+[[firehose-waf-prerequisites]]
+== Before you begin
+
+We assume that you already have:
+
+- An AWS account with permissions to pull the necessary data from AWS.
+- A deployment using our hosted {ess} on {ess-trial}[{ecloud}]. The deployment includes an {es} cluster for storing and searching your data, and {kib} for visualizing and managing your data. AWS Data Firehose works with Elastic Stack version 7.17 or greater, running on Elastic Cloud only.
+
+IMPORTANT: Make sure the deployment is on AWS, because the Amazon Data Firehose delivery stream connects specifically to an endpoint that needs to be on AWS.
+
+[discrete]
+[[firehose-waf-step-one]]
+== Step 1: Install the AWS integration in {kib}
+
+. In {kib}, navigate to *Management* > *Integrations* and browse the catalog to find the AWS integration.
+
+. Navigate to the *Settings* tab and click *Install AWS assets*.
+
+[discrete]
+[[firehose-waf-step-two]]
+== Step 2: Create a delivery stream in Amazon Data Firehose
+
+. Go to the https://console.aws.amazon.com/[AWS console] and navigate to Amazon Data Firehose.  
+
+. Click *Create Firehose stream* and choose the source and destination of your Firehose stream. Unless you are streaming data from Kinesis Data Streams, set source to `Direct PUT` and destination to `Elastic`. 
+
+. Provide a meaningful *Firehose stream name* that will allow you to identify this delivery stream later. Your Firehose name must start with the prefix `aws-waf-logs-` or it will not show up later.
+
+NOTE: For advanced use cases, source records can be transformed by invoking a custom Lambda function. When using Elastic integrations, this should not be required.
+
+[discrete]
+[[firehose-waf-step-three]]
+== Step 3: Specify the destination settings for your Firehose stream
+
+. From the *Destination settings* panel, specify the following settings:
++
+* *Elastic endpoint URL*: Enter the Elastic endpoint URL of your Elasticsearch cluster. To find the Elasticsearch endpoint, go to the Elastic Cloud console, navigate to the Integrations page, and select *Connection details*. Here is an example of how it looks like: `https://my-deployment.es.us-east-1.aws.elastic-cloud.com`.
++
+* *API key*: Enter the encoded Elastic API key. To create an API key, go to the Elastic Cloud console, navigate to the Integrations page, select *Connection details* and click *Create and manage API keys*. If you are using an API key with *Restrict privileges*, make sure to review the Indices privileges to provide at least "auto_configure" & "write" permissions for the indices you will be using with this delivery stream. 
++
+* *Content encoding*: For a better network efficiency, leave content encoding set to GZIP. 
++
+* *Retry duration*: Determines how long Firehose continues retrying the request in the event of an error. A duration of 60-300s should be suitable for most use cases.
++
+* *es_datastream_name*: `logs-aws.waf-default`
+
+[discrete]
+[[firehose-waf-step-four]]
+== Step 4: Create a web access control list
+
+To create a new web access control list (ACL), follow these steps:
+
+. Go to the https://console.aws.amazon.com/[AWS console] and navigate to the *WAF & Shield* page. 
+
+. Describe web ACL by entering the resource type, region, and name. 
+
+. Associate it to an AWS resource. If you don't have an existing resource, you can create and attach a web ACL to several AWS resources:
++
+- CloudFront distribution
+- Application Load Balancers
+- Amazon API Gateway REST APIs
+- Amazon App Runner services
+- AWS AppSync GraphQL APIs
+- Amazon Cognito user pools
+- AWS Verified Access Instances
+
+. Add a 1 or 2 rules to the *Free rule groups* list from the AWS managed rule groups. Keep all other settings to their default values.
+
+. Set the rule priority by keeping default values.
+
+. Configure metrics by keeping default values.
+
+. Review and create the web ACL.
+
+[discrete]
+[[firehose-waf-step-five]]
+== Step 5: Set up logging
+
+. Go to the web ACL you created in the previous step.
+
+. Open the *Logging and metrics* section and edit the following settings:
++
+- *Logging destination*: select "Amazon Data Firehose stream"
+- *Amazon Data Firehose stream*: select the Firehose stream you created in step 2.
+
+WAF creates the required Identity and Access Management (IAM) role.
+If your Firehose stream name doesn't appear in the list, make sure the name you chose for the stream starts with `aws-waf-logs-`, as prescribed by AWS naming conventions.
+
+[discrete]
+[[firehose-waf-step-six]]
+== Step 6: Visualize your WAF logs in {kib}
+
+You can now log into your {stack} to check if the WAF logs are flowing. To generate logs, you can use cURL to send HTTP requests to your testing CloudFront distribution.
+
+[source,console]
+----
+curl -i https://<your cloudfront distribution>.cloudfront.net
+----
+
+To maintain a steady flow of logs, you can use `watch -n 5` to repeat the command every 5 seconds.
+
+[source,console]
+----
+watch -n 5 curl -i https://<your cloudfront distribution>.cloudfront.net
+----
+
+Navigate to Kibana and visualize the first WAF logs in your {stack}.
+
+[role="screenshot"]
+image::firehose-waf-logs.png[Firehose WAF logs in Kibana]

docs/en/observability/index.asciidoc

@@ -172,6 +172,7 @@ include::create-alerts.asciidoc[leveloffset=+1]
 include::aggregation-options.asciidoc[leveloffset=+2]
 include::view-observability-alerts.asciidoc[leveloffset=+2]
 include::triage-slo-burn-rate-breaches.asciidoc[leveloffset=+3]
+include::triage-threshold-breaches.asciidoc[leveloffset=+3]
 
 //SLOs
 include::slo-overview.asciidoc[leveloffset=+1]