[apm] Update APM docs to reflect changes related to the Elasticsearch…

… apm-data plugin (#4333) (#4427)

* update getting started docs

* update upgrade guide

* fix build

* audit mentions of the apm integration

* audit mentions of data streams

* audit mentions of index templates

* audit mentions of index mappings

* address feedback from @carsonip

* update diagram

* address more feedback from @carsonip

* update ilm guide

* address more feedback from @lahsivjar

* address more feedback from @lahsivjar

* add link

* fix table formatting

* address more feedback from @lahsivjar

(cherry picked from commit 27fc737)

Co-authored-by: Colleen McGinnis <[email protected]>

docs/en/observability/apm/configure/general.asciidoc

@@ -173,7 +173,7 @@ Defaults to `debug/vars`.
 === Data stream namespace
 
 Change the default namespace.
-This setting changes the name of the integration's data stream.
+This setting changes the name of the data stream.
 
 For {fleet}-managed users, the namespace is inherited from the selected {agent} policy.
 

docs/en/observability/apm/configure/outputs/console.asciidoc

@@ -26,13 +26,6 @@ output.console:
   pretty: true
 ------------------------------------------------------------------------------
 
-ifdef::apm-server[]
-[float]
-=== {kib} configuration
-
-include::../../shared-kibana-endpoint.asciidoc[tag=shared-kibana-config]
-endif::[]
-
 [float]
 === Configuration options
 

docs/en/observability/apm/configure/outputs/kafka.asciidoc

@@ -37,11 +37,6 @@ output.kafka:
 
 NOTE: Events bigger than <<apm-kafka-max_message_bytes,`max_message_bytes`>> will be dropped. To avoid this problem, make sure APM Server does not generate events bigger than <<apm-kafka-max_message_bytes,`max_message_bytes`>>.
 
-[float]
-=== {kib} configuration
-
-include::../../shared-kibana-endpoint.asciidoc[tag=shared-kibana-config]
-
 [float]
 [[apm-kafka-compatibility]]
 === Compatibility

docs/en/observability/apm/configure/outputs/logstash.asciidoc

@@ -21,7 +21,6 @@ protocol, which runs over TCP.
 To send events to {ls}, you must:
 
 . <<apm-ls-output-config,Enable the {ls} output in APM Server>>
-. <<apm-ls-kib-config,Enable the {kib} endpoint in APM Server>>
 . <<apm-ls-config-pipeline,Create a {ls} configuration pipeline in {ls}>>
 
 [float]
@@ -43,12 +42,6 @@ output.logstash:
 <1> The `hosts` option specifies the {ls} server and the port (`5044`) where {ls} is configured to listen for incoming
 APM Server connections.
 
-[float]
-[[apm-ls-kib-config]]
-=== {kib} endpoint configuration
-
-include::../../shared-kibana-endpoint.asciidoc[tag=shared-kibana-config]
-
 [float]
 [[apm-ls-config-pipeline]]
 === {ls} configuration pipeline

docs/en/observability/apm/configure/outputs/redis.asciidoc

@@ -30,11 +30,6 @@ output.redis:
   timeout: 5
 ------------------------------------------------------------------------------
 
-[float]
-=== {kib} configuration
-
-include::../../shared-kibana-endpoint.asciidoc[tag=shared-kibana-config]
-
 [float]
 === Compatibility
 

docs/en/observability/apm/getting-started-apm/get-started-with-apm-server-binary.asciidoc

@@ -165,69 +165,6 @@ See <<apm-running-on-docker, Running on Docker>> for deploying Docker containers
 [[apm-server-configuration]]
 == Step 2: Set up and configure
 
-// This content is reused in the upgrading guide
-// tag::why-apm-integration[]
-Starting in version 8.0.0, {fleet} uses the APM integration to set up and manage APM index templates,
-{ilm-init} policies, and ingest pipelines. APM Server will only send data to {es} _after_ the APM integration has been installed.
-// end::why-apm-integration[]
-
-[float]
-=== Install the APM integration
-
-// This content is reused in the upgrading guide
-// tag::install-apm-integration[]
-[%collapsible%open]
-.**If you have an internet connection**
-====
-An internet connection is required to install the APM integration via the {fleet} UI in {kib}.
-
-// lint ignore elastic-agent
-. Open {kib} and select **Add integrations** > **Elastic APM**.
-. Click **APM integration**.
-. Click **Add Elastic APM**.
-. Click **Save and continue**.
-. Click **Add Elastic Agent later**. You do not need to run an {agent} to complete the setup.
-====
-
-// tag::install-apm-integration-no-internet[]
-[%collapsible]
-.**If you don't have an internet connection**
-====
-If your environment has network traffic restrictions, there are other ways to install the APM integration.
-See {fleet-guide}/air-gapped.html[Air-gapped environments] for more information.
-
-Option 1: Update `kibana.yml`::
-+
-Update `kibana.yml` to include the following, then restart {kib}.
-+
-[source,yaml]
-----
-xpack.fleet.packages:
-- name: apm
-  version: latest
-----
-+
-See {kibana-ref}/settings.html[Configure Kibana] to learn more about how to edit the Kibana configuration file.
-
-Option 2: Use the {fleet} API::
-+
-Use the {fleet} API to install the APM integration. To be successful, this needs to be run against the {kib}
-API, not the {es} API.
-+
-["source","yaml",subs="attributes"]
-----
-POST kbn:/api/fleet/epm/packages/apm/{apm_server_version}
-{ "force": true }
-----
-+
-See {kibana-ref}/api.html[Kibana API] to learn more about how to use the Kibana APIs.
-====
-// end::install-apm-integration-no-internet[]
-// end::install-apm-integration[]
-
-[float]
-=== Configure APM
-
 Configure APM by editing the `apm-server.yml` configuration file.
 The location of this file varies by platform--see the <<apm-directory-layout>> for help locating it.
 

docs/en/observability/apm/getting-started-apm/get-started-with-fleet-apm-server.asciidoc

@@ -32,7 +32,7 @@ include::{observability-docs-root}/docs/en/observability/tab-widgets/add-apm-int
 An internet connection is required to install the APM integration via the Fleet UI in Kibana.
 
 --
-include::{observability-docs-root}/docs/en/observability/apm/getting-started-apm/get-started-with-apm-server-binary.asciidoc[leveloffset=+2,tag=install-apm-integration-no-internet]
+include::{observability-docs-root}/docs/en/observability/apm/shared/install-apm-integration-no-internet.asciidoc[]
 --
 ****
 

docs/en/observability/apm/getting-started-apm/index.asciidoc

@@ -12,9 +12,8 @@ The {es} Service is available on AWS, GCP, and Azure.
 *To get started in minutes, follow the steps in <<get-started-with-fleet-apm-server>>.*
 ****
 
-// TODO: MOVE THIS
-IMPORTANT: Starting in version 8.0.0, {fleet} uses the APM integration to set up and manage APM index templates,
-{ilm-init} policies, and ingest pipelines. APM Server will only send data to {es} _after_ the APM integration has been installed.
+IMPORTANT: Starting in version 8.15.0, the {es} apm-data plugin manages APM index templates,
+lifecycle policies, and ingest pipelines.
 
 The APM Server receives performance data from your APM agents,
 validates and processes it, and then transforms the data into {es} documents.

docs/en/observability/apm/manage-storage/data-streams.asciidoc

@@ -20,7 +20,7 @@ See the {fleet-guide}/data-streams.html[{fleet} and {agent} Guide] to learn more
 
 // tag::data-streams[]
 APM data follows the `<type>-<dataset>-<namespace>` naming scheme.
-The `type` and `dataset` are predefined by the APM integration,
+The `type` and `dataset` are predefined by the {es} apm-data plugin,
 but the `namespace` is your opportunity to customize how different types of data are stored in {es}.
 There is no recommendation for what to use as your namespace--it is intentionally flexible.
 For example, you might create namespaces for each of your environments,

docs/en/observability/apm/manage-storage/ilm-how-to.asciidoc

@@ -8,133 +8,133 @@
 :append-legacy:
 // tag::ilm-integration[]
 
-Index lifecycle policies allow you to automate the
-lifecycle of your APM indices as they grow and age.
-A default policy is applied to each APM data stream,
-but can be customized depending on your business needs.
-
-See {ref}/index-lifecycle-management.html[{ilm-init}: Manage the index lifecycle] to learn more.
+Lifecycle policies allow you to automate the lifecycle of your APM indices as they grow and age.
+A default policy is applied to each APM data stream, but can be customized depending on your business needs.
 
 [discrete]
 [id="index-lifecycle-policies-default{append-legacy}"]
 == Default policies
 
-The table below describes the default index lifecycle policy applied to each APM data stream.
-Each policy includes a rollover and delete definition:
+[NOTE]
+====
+// Explain what changed
+// Before
+Before 8.15, clusters used {ref}/index-lifecycle-management.html[index lifecycle management (ILM)] to provide the default data retention settings for APM data.
+// After
+As of 8.15, new clusters use {ref}/data-stream-lifecycle.html[data stream lifecycle (DSL)] to provide default data retention settings, but the customization for lifecycle is still performed using ILM policies, which are now configured to override the default DSL polices.
+
+// What this means for existing clusters that are upgraded
+For _existing_ clusters upgrading to 8.15, the default ILM policies should still work as expected.
+You can can continue using ILM or switch the default to DSL explicitly using the {ref}/data-stream-apis.html[data stream API].
+
+// What this means for new clusters that are created
+For _new_ clusters created in 8.15 or later, if you prefer to continue using ILM,
+follow the steps in this guide to create a custom ILM policy and add it to the `*@custom` component template for each data stream.
+====
 
-* **Rollover**: Using rollover indices prevents a single index from growing too large and optimizes indexing and search performance. Rollover, i.e. writing to a new index, occurs after either an age or size metric is met.
-* **Delete**: The delete phase permanently removes the index after a time threshold is met.
+Each APM data stream has its own default lifecycle policy including a delete definition and a rollover definition.
 
-[cols="1,1,1,1",options="header"]
+The table below describes the delete definition for each APM data stream.
+The delete phase permanently removes the index after a time threshold is met.
+
+[cols="1,1,1",options="header"]
 |===
 |Data stream
-|Rollover after
 |Delete after
 |Notes
 
 | `traces-apm`
-| 30 days / 50 GB
 | 10 days
 | Raw trace event data
 
 | `traces-apm.rum`
-| 30 days / 50 GB
 | 90 days
 | Raw RUM trace event data, used in the UI
 
 | `logs-apm.error`
-| 30 days / 50 GB
 | 10 days
 | Error event data
 
 | `logs-apm.app`
-| 30 days / 50 GB
 | 10 days
 | Logs event data
 
 | `metrics-apm.app`
-| 30 days / 50 GB
 | 90 days
 | Custom application specific metrics
 
 | `metrics-apm.internal`
-| 30 days / 50 GB
 | 90 days
 | Common system metrics and language specific metrics (for example, CPU and memory usage)
 
 | `metrics-apm.service_destination_1m`
-| 7 days / 50GB
 | 90 days
 | Aggregated transaction metrics powering the Applications UI
 
 | `metrics-apm.service_destination_10m`
-| 14 days / 50GB
 | 180 days
 | Aggregated transaction metrics powering the Applications UI
 
 | `metrics-apm.service_destination_60m`
-| 30 days / 50GB
 | 390 days
 | Aggregated transaction metrics powering the Applications UI
 
 | `metrics-apm.service_summary_1m`
-| 7 days / 50GB
 | 90 days
 | Aggregated transaction metrics powering the Applications UI
 
 | `metrics-apm.service_summary_10m`
-| 14 days / 50GB
 | 180 days
 | Aggregated transaction metrics powering the Applications UI
 
 | `metrics-apm.service_summary_60m`
-| 30 days / 50GB
 | 390 days
 | Aggregated transaction metrics powering the Applications UI
 
 | `metrics-apm.service_transaction_1m`
-| 7 days / 50GB
 | 90 days
 | Aggregated transaction metrics powering the Applications UI
 
 | `metrics-apm.service_transaction_10m`
-| 14 days / 50GB
 | 180 days
 | Aggregated transaction metrics powering the Applications UI
 
 | `metrics-apm.service_transaction_60m`
-| 30 days / 50GB
 | 390 days
 | Aggregated transaction metrics powering the Applications UI
 
 | `metrics-apm.transaction_1m`
-| 7 days / 50GB
 | 90 days
 | Aggregated transaction metrics powering the Applications UI
 
 | `metrics-apm.transaction_10m`
-| 14 days / 50GB
 | 180 days
 | Aggregated transaction metrics powering the Applications UI
 
 | `metrics-apm.transaction_60m`
-| 30 days / 50GB
 | 390 days
 | Aggregated transaction metrics powering the Applications UI
 
 |===
 
-The APM index lifecycle policies can be viewed in {kib}.
-Navigate to *{stack-manage-app}* / *Index Lifecycle Management*, and search for `apm`.
+Rollover (writing to a new index) prevents a single index from growing too large and optimizes indexing and search performance.
+Rollover occurs after either an age or size metric is met.
+The default rollover definition for each APM data stream is applied based on {ref}/data-stream-lifecycle-settings.html#cluster-lifecycle-default-rollover[`cluster.lifecycle.default.rollover`].
+
+The APM data stream lifecycle policies can be viewed in {kib}:
+
+. Navigate to *{stack-manage-app}* → *Index Management* → *Component Templates*.
+. Search for `apm`.
+. Look for templates with `@lifecycle` suffix.
 
-TIP: Default {ilm-init} policies can change between minor versions.
-This is not considered a breaking change as index management should continually improve and adapt to new features.
+TIP: Default lifecycle policies can change between minor versions. This is not considered a breaking change as index management should continually improve and adapt to new features.
 
 [discrete]
 [id="apm-data-streams-custom-policy{append-legacy}"]
 == Configure a custom index lifecycle policy
 
-When the APM integration is installed, {fleet} creates a default `*@custom` component template for each data stream.
+Mappings and settings for data streams can be customized through the creation of `*@custom` component templates,
+which are referenced by the index templates created by the {es} apm-data plugin.
 The easiest way to configure a custom index lifecycle policy per data stream is to edit this template.
 
 This tutorial explains how to apply a custom index lifecycle policy to the `traces-apm` data stream.
@@ -143,10 +143,10 @@ This tutorial explains how to apply a custom index lifecycle policy to the `trac
 [id="apm-data-streams-custom-one{append-legacy}"]
 == Step 1: View data streams
 
-The **Data Streams** view in {kib} shows you the data streams,
-index templates, and index lifecycle policies associated with a given integration.
+The **Data Streams** view in {kib} shows you data streams,
+index templates, and lifecycle policies:
 
-. Navigate to **{stack-manage-app}** > **Index Management** > **Data Streams**.
+. Navigate to **{stack-manage-app}** → **Index Management** → **Data Streams**.
 . Search for `traces-apm` to see all data streams associated with APM trace data.
 . In this example, I only have one data stream because I'm only using the `default` namespace.
 You may have more if your setup includes multiple namespaces.
@@ -158,7 +158,7 @@ image::images/data-stream-overview.png[Data streams info]
 [id="apm-data-streams-custom-two{append-legacy}"]
 == Step 2: Create an index lifecycle policy
 
-. Navigate to **{stack-manage-app}** > **Index Lifecycle Policies**.
+. Navigate to **{stack-manage-app}** → **Index Lifecycle Policies**.
 . Click **Create policy**.
 
 Name your new policy; For this tutorial, I've chosen `custom-traces-apm-policy`.
@@ -172,14 +172,15 @@ To apply your new index lifecycle policy to the `traces-apm-*` data stream,
 edit the `<data-stream-name>@custom` component template.
 
 . Click on the **Component Template** tab and search for `traces-apm`.
-. Select the `traces-apm@custom` template and click **Manage** > **Edit**.
+. Select the `traces-apm@custom` template and click **Manage** → **Edit**.
 . Under **Index settings**, set the {ilm-init} policy name created in the previous step:
 +
 [source,json]
 ----
 {
   "lifecycle": {
-    "name": "custom-traces-apm-policy"
+    "name": "custom-traces-apm-policy",
+    "prefer_ilm": true
   }
 }
 ----

docs/en/observability/apm/manage-storage/ingest-pipelines.asciidoc

@@ -19,7 +19,7 @@ The default APM pipelines are defined in {es} apm-data plugin index templates.
 [id="custom-ingest-pipelines{append-legacy}"]
 == Custom ingest pipelines
 
-The Elastic APM integration supports custom ingest pipelines.
+Elastic APM supports custom ingest pipelines.
 A custom pipeline allows you to transform data to better match your specific use case.
 This can be useful, for example, to ensure data security by removing or obfuscating sensitive information.