diff --git a/docs/en/observability/images/profiling-create-component-template.png b/docs/en/observability/images/profiling-create-component-template.png new file mode 100644 index 0000000000..9f4bd9709c Binary files /dev/null and b/docs/en/observability/images/profiling-create-component-template.png differ diff --git a/docs/en/observability/index.asciidoc b/docs/en/observability/index.asciidoc index df7bb4cd74..d17caceae7 100644 --- a/docs/en/observability/index.asciidoc +++ b/docs/en/observability/index.asciidoc @@ -141,11 +141,14 @@ include::universal-profiling.asciidoc[leveloffset=+1] include::profiling-get-started.asciidoc[leveloffset=+2] +include::profiling-manage-storage.asciidoc[leveloffset=+2] +include::profiling-index-lifecycle-management.asciidoc[leveloffset=+3] +include::profiling-probabilistic-profiling.asciidoc[leveloffset=+3] + include::profiling-advanced-configuration.asciidoc[leveloffset=+2] include::profiling-tag-data-query.asciidoc[leveloffset=+3] include::profiling-add-symbols.asciidoc[leveloffset=+3] include::profiling-use-a-proxy.asciidoc[leveloffset=+3] -include::profiling-probabilistic-profiling.asciidoc[leveloffset=+3] include::profiling-no-kernel-version-check.asciidoc[leveloffset=+3] include::profiling-upgrade.asciidoc[leveloffset=+2] diff --git a/docs/en/observability/profiling-advanced-configuration.asciidoc b/docs/en/observability/profiling-advanced-configuration.asciidoc index 8b7e4ec1c0..d3438dc601 100644 --- a/docs/en/observability/profiling-advanced-configuration.asciidoc +++ b/docs/en/observability/profiling-advanced-configuration.asciidoc @@ -7,8 +7,6 @@ See the following sections for more information: * <>: Tag data collected by the host-agent into multiple logical groups so they can be queried in Kibana. * <>: Push symbols to your cluster so you can see function names and line numbers in traces of applications written in programming languages that compile to native code (C, C++, Rust, Go, etc.). * <>: Set up an HTTP proxy if your infrastructure host-agent installation needs one to reach {ecloud}. -* <>: Configure the Universal Profiling agent to run in probabilistic profiling mode. - -* <>: Configure the Universal Profiling agent to bypass the kernel version compatibility check. +* <>: Configure the Universal Profiling agent to bypass the kernel version compatibility check. diff --git a/docs/en/observability/profiling-index-lifecycle-management.asciidoc b/docs/en/observability/profiling-index-lifecycle-management.asciidoc new file mode 100644 index 0000000000..e7622a1a49 --- /dev/null +++ b/docs/en/observability/profiling-index-lifecycle-management.asciidoc @@ -0,0 +1,159 @@ +[[profiling-index-lifecycle-management]] += Universal Profiling index life cycle management + +++++ +Index lifecycle management +++++ + + +Index lifecycle policies allow you to automate the lifecycle of your profiling indices as they grow and age. A default policy is applied, but you can customize it based on your business needs. + +[discrete] +[[profiling-ilm-default-policy]] +== Default policy + +The default Universal Profiling index lifecycle policy includes the following rollover and delete definitions: + + +* **Rollover**: Rollover prevents a single index from growing too large and optimizes indexing and search performance. After an age or size metric threshold is met, a new index is created and all subsequent updates are written to the new index. + +* **Delete**: The delete phase permanently removes the index after a time threshold is met. + +The following table lists the default thresholds for rollover and delete: + +[cols="1,1,1",options="header"] +|=== +|Rollover +|Warm tier +|Delete + +| after 30 days or 50 GB +| after 30 days +| after 60 days + + +|=== + +NOTE: The {ref}/ilm-rollover.html#_rollover_condition_blocks_phase_transition[rollover condition blocks phase transitions] which means that indices are kept 30 days *after* rollover on the hot tier. + +To view the Universal Profiling index lifecycle policy in {kib}, navigate to *{stack-manage-app}* → *Index Lifecycle Management*, and search for `profiling`. + + +TIP: Default {ilm-init} policies can change between minor versions. This is not considered a breaking change as index management should continually improve and adapt to new features. + +[discrete] +[[profiling-ilm-custom-policy]] +== Configure a custom index lifecycle policy + +Complete the following steps to configure a custom index lifecycle policy. + +[discrete] +[[profiling-ilm-custom-policy-create-policy]] +=== Step 1: Create an index lifecycle policy + +. Navigate to **{stack-manage-app}** → **Index Lifecycle Policies**. + +. Click **Create policy**. +. Name your new policy, for example `custom-profiling-policy`. +. Customize the policy to your liking. +. Click **Save policy**. + +TIP: See {ref}/index-lifecycle-management.html[Manage the index lifecycle] to learn more about {ilm-init} policies. + +[discrete] +[[profiling-ilm-custom-policy-apply-policy]] +=== Step 2: Apply the index lifecycle policy + +To apply your new index lifecycle policy for Universal Profiling, create a new component template named `profiling-ilm@custom`. + +NOTE: To apply a custom {ilm-init} policy, you must name the component template `profiling-ilm@custom`. Other names are not supported. + + +. From the *Index Management* page, select the **Component Template** tab and click **Create component template**. + +. Enter `profiling-ilm@custom` as the name and click **Next**. + +. In **Index settings**, set the {ilm-init} policy name created in the previous step: ++ +[source,json] +---- +{ + "lifecycle": { + "name": "custom-profiling-policy" + } +} +---- +. Continue to the **Review** step, and select the *Request* tab. Your request should look similar to the following image. + + +If it does, click **Create component template**. ++ +[role="screenshot"] +image::images/profiling-create-component-template.png[Create component template] + +[discrete] +[[profiling-ilm-custom-policy-rollover]] +=== Step 3: Rollover indices + +To confirm that Universal Profiling is now using the new index template and {ilm-init} policy, navigate to **{dev-tools-app}** and run the following: + +[source,bash] +---- +GET _ilm/policy/custom-profiling-policy <1> +---- +<1> The name of the custom {ilm-init} policy chosen in <>. + + +If the custom policy is already applied, the result should include the following: + +[source,json] +---- +{ + "in_use_by": { + "indices": [ + ... + ], + "data_streams": [ + ... + "profiling-events-all", + ... + ], + "composable_templates": [ + "profiling-stackframes", + "profiling-symbols-global", + "profiling-metrics", + "profiling-stacktraces", + "profiling-executables", + "profiling-hosts", + "profiling-events" + ] + } +} +---- + +If the result is empty, the custom {ilm-init} policy is not yet in use. New {ilm-init} policies only take effect when new indices are created, so either wait for a rollover to occur (usually after 30 days or when the index size reaches 50 GB), or force a rollover using the {ref}/indices-rollover-index.html[{es} rollover API]: + + +[source,bash] +---- +POST /profiling-events-5pow01/_rollover/ +POST /profiling-events-5pow02/_rollover/ +POST /profiling-events-5pow03/_rollover/ +POST /profiling-events-5pow04/_rollover/ +POST /profiling-events-5pow05/_rollover/ +POST /profiling-events-5pow06/_rollover/ +POST /profiling-events-5pow07/_rollover/ +POST /profiling-events-5pow08/_rollover/ +POST /profiling-events-5pow09/_rollover/ +POST /profiling-events-5pow10/_rollover/ +POST /profiling-events-5pow11/_rollover/ +POST /profiling-events-all/_rollover/ +POST /profiling-executables/_rollover/ +POST /profiling-hosts/_rollover/ +POST /profiling-metrics/_rollover/ +POST /profiling-stackframes/_rollover/ +POST /profiling-stacktraces/_rollover/ +POST /profiling-symbols-global/_rollover/ +---- + +After the rollover, the custom {ilm-init} policy will be applied to new indices and data streams. diff --git a/docs/en/observability/profiling-manage-storage.asciidoc b/docs/en/observability/profiling-manage-storage.asciidoc new file mode 100644 index 0000000000..ff82306abb --- /dev/null +++ b/docs/en/observability/profiling-manage-storage.asciidoc @@ -0,0 +1,11 @@ +[[profiling-manage-storage]] += Manage data storage + + +Universal Profiling provides the following ways to manage how your data is stored. + +* <> automatically manages your indices according to age or size metric thresholds. Universal Profiling ships with a default index lifecycle policy, but you can create a custom policy to meet your requirements. +* <> mode uses representative samples of profiling data to reduce storage needs even further. + + +