[Profiling] Improve docs on how to manage storage (#3247)

With this commit we restructure the Universal Profiling documentation
and add a new menu item about managing storage. It contains a new
section that explains how to customize the ILM policy (usually to
customize the retention period or to take advantage of data tiers) and
moves the probabilistic profiling documentation also into this section.

---------

Co-authored-by: Mike Birnstiehl <[email protected]>
Co-authored-by: Christos Kalkanis <[email protected]>

docs/en/observability/index.asciidoc

@@ -141,11 +141,14 @@ include::universal-profiling.asciidoc[leveloffset=+1]
 
 include::profiling-get-started.asciidoc[leveloffset=+2]
 
+include::profiling-manage-storage.asciidoc[leveloffset=+2]
+include::profiling-index-lifecycle-management.asciidoc[leveloffset=+3]
+include::profiling-probabilistic-profiling.asciidoc[leveloffset=+3]
+
 include::profiling-advanced-configuration.asciidoc[leveloffset=+2]
 include::profiling-tag-data-query.asciidoc[leveloffset=+3]
 include::profiling-add-symbols.asciidoc[leveloffset=+3]
 include::profiling-use-a-proxy.asciidoc[leveloffset=+3]
-include::profiling-probabilistic-profiling.asciidoc[leveloffset=+3]
 include::profiling-no-kernel-version-check.asciidoc[leveloffset=+3]
 
 include::profiling-upgrade.asciidoc[leveloffset=+2]

docs/en/observability/profiling-advanced-configuration.asciidoc

@@ -7,8 +7,6 @@ See the following sections for more information:
 * <<profiling-tag-data-query, Tag data for querying>>: Tag data collected by the host-agent into multiple logical groups so they can be queried in Kibana.
 * <<profiling-add-symbols, Add symbols for native frames>>: Push symbols to your cluster so you can see function names and line numbers in traces of applications written in programming languages that compile to native code (C, C++, Rust, Go, etc.).
 * <<profiling-use-a-proxy,Use a proxy>>:  Set up an HTTP proxy if your infrastructure host-agent installation needs one to reach {ecloud}.
-* <<profiling-probabilistic-profiling, Configure probabilistic profiling>>: Configure the Universal Profiling agent to run in probabilistic profiling mode.
-
-* <<profiling-no-kernel-version-check, Override kernel version check >>: Configure the Universal Profiling agent to bypass the kernel version compatibility check. 
+* <<profiling-no-kernel-version-check, Override kernel version check >>: Configure the Universal Profiling agent to bypass the kernel version compatibility check.
 
 

docs/en/observability/profiling-index-lifecycle-management.asciidoc

@@ -0,0 +1,159 @@
+[[profiling-index-lifecycle-management]]
+= Universal Profiling index life cycle management
+
+++++
+<titleabbrev>Index lifecycle management</titleabbrev>
+++++
+
+
+Index lifecycle policies allow you to automate the lifecycle of your profiling indices as they grow and age. A default policy is applied, but you can customize it based on your business needs.
+
+[discrete]
+[[profiling-ilm-default-policy]]
+== Default policy
+
+The default Universal Profiling index lifecycle policy includes the following rollover and delete definitions:
+
+
+* **Rollover**: Rollover prevents a single index from growing too large and optimizes indexing and search performance. After an age or size metric threshold is met, a new index is created and all subsequent updates are written to the new index.
+
+* **Delete**: The delete phase permanently removes the index after a time threshold is met.
+
+The following table lists the default thresholds for rollover and delete:
+
+[cols="1,1,1",options="header"]
+|===
+|Rollover
+|Warm tier
+|Delete
+
+| after 30 days or 50 GB
+| after 30 days
+| after 60 days
+
+
+|===
+
+NOTE: The {ref}/ilm-rollover.html#_rollover_condition_blocks_phase_transition[rollover condition blocks phase transitions] which means that indices are kept 30 days *after* rollover on the hot tier.
+
+To view the Universal Profiling index lifecycle policy in {kib}, navigate to *{stack-manage-app}* → *Index Lifecycle Management*, and search for `profiling`.
+
+
+TIP: Default {ilm-init} policies can change between minor versions. This is not considered a breaking change as index management should continually improve and adapt to new features.
+
+[discrete]
+[[profiling-ilm-custom-policy]]
+== Configure a custom index lifecycle policy
+
+Complete the following steps to configure a custom index lifecycle policy.
+
+[discrete]
+[[profiling-ilm-custom-policy-create-policy]]
+=== Step 1: Create an index lifecycle policy
+
+. Navigate to **{stack-manage-app}** → **Index Lifecycle Policies**.
+
+. Click **Create policy**.
+. Name your new policy, for example `custom-profiling-policy`. 
+. Customize the policy to your liking.
+. Click **Save policy**.
+
+TIP: See {ref}/index-lifecycle-management.html[Manage the index lifecycle] to learn more about {ilm-init} policies.
+
+[discrete]
+[[profiling-ilm-custom-policy-apply-policy]]
+=== Step 2: Apply the index lifecycle policy
+
+To apply your new index lifecycle policy for Universal Profiling, create a new component template named `profiling-ilm@custom`.
+
+NOTE: To apply a custom {ilm-init} policy, you must name the component template `profiling-ilm@custom`. Other names are not supported.
+
+
+. From the *Index Management* page, select the **Component Template** tab and click **Create component template**.
+
+. Enter `profiling-ilm@custom` as the name and click **Next**.
+
+. In **Index settings**, set the {ilm-init} policy name created in the previous step:
++
+[source,json]
+----
+{
+  "lifecycle": {
+    "name": "custom-profiling-policy"
+  }
+}
+----
+. Continue to the **Review** step, and select the *Request* tab. Your request should look similar to the following image.
+
+
+If it does, click **Create component template**.
++
+[role="screenshot"]
+image::images/profiling-create-component-template.png[Create component template]
+
+[discrete]
+[[profiling-ilm-custom-policy-rollover]]
+=== Step 3: Rollover indices
+
+To confirm that Universal Profiling is now using the new index template and {ilm-init} policy, navigate to **{dev-tools-app}** and run the following:
+
+[source,bash]
+----
+GET _ilm/policy/custom-profiling-policy <1>
+----
+<1> The name of the custom {ilm-init} policy chosen in <<profiling-ilm-custom-policy-create-policy,Step 1>>.
+
+
+If the custom policy is already applied, the result should include the following:
+
+[source,json]
+----
+{
+    "in_use_by": {
+      "indices": [
+        ...
+      ],
+      "data_streams": [
+        ...
+        "profiling-events-all",
+        ...
+      ],
+      "composable_templates": [
+        "profiling-stackframes",
+        "profiling-symbols-global",
+        "profiling-metrics",
+        "profiling-stacktraces",
+        "profiling-executables",
+        "profiling-hosts",
+        "profiling-events"
+      ]
+    }
+}
+----
+
+If the result is empty, the custom {ilm-init} policy is not yet in use. New {ilm-init} policies only take effect when new indices are created, so either wait for a rollover to occur (usually after 30 days or when the index size reaches 50 GB), or force a rollover using the {ref}/indices-rollover-index.html[{es} rollover API]:
+
+
+[source,bash]
+----
+POST /profiling-events-5pow01/_rollover/
+POST /profiling-events-5pow02/_rollover/
+POST /profiling-events-5pow03/_rollover/
+POST /profiling-events-5pow04/_rollover/
+POST /profiling-events-5pow05/_rollover/
+POST /profiling-events-5pow06/_rollover/
+POST /profiling-events-5pow07/_rollover/
+POST /profiling-events-5pow08/_rollover/
+POST /profiling-events-5pow09/_rollover/
+POST /profiling-events-5pow10/_rollover/
+POST /profiling-events-5pow11/_rollover/
+POST /profiling-events-all/_rollover/
+POST /profiling-executables/_rollover/
+POST /profiling-hosts/_rollover/
+POST /profiling-metrics/_rollover/
+POST /profiling-stackframes/_rollover/
+POST /profiling-stacktraces/_rollover/
+POST /profiling-symbols-global/_rollover/
+----
+
+After the rollover, the custom {ilm-init} policy will be applied to new indices and data streams.

docs/en/observability/profiling-manage-storage.asciidoc

@@ -0,0 +1,11 @@
+[[profiling-manage-storage]]
+= Manage data storage
+
+
+Universal Profiling provides the following ways to manage how your data is stored. 
+
+* <<profiling-index-lifecycle-management, Index lifecycle management>> automatically manages your indices according to age or size metric thresholds. Universal Profiling ships with a default index lifecycle policy, but you can create a custom policy to meet your requirements. 
+* <<profiling-probabilistic-profiling, Probabilistic profiling>> mode uses representative samples of profiling data to reduce storage needs even further.
+
+
+