From 2f906419d0b975e4e3f4f3ed46b67e2e49b7065c Mon Sep 17 00:00:00 2001 From: "mergify[bot]" <37929162+mergify[bot]@users.noreply.github.com> Date: Thu, 31 Oct 2024 10:03:51 -0500 Subject: [PATCH] APM: add known issue of lazy rollover bug (#4459) (#4460) Signed-off-by: inge4pres Co-authored-by: simitt (cherry picked from commit 66a73ef17b50143e6f6f12bc430bc197e8905b46) Co-authored-by: Francesco Gualazzi --- .../observability/apm/known-issues.asciidoc | 44 +++++++++++++++++++ 1 file changed, 44 insertions(+) diff --git a/docs/en/observability/apm/known-issues.asciidoc b/docs/en/observability/apm/known-issues.asciidoc index 3ed733cd2c..c18ca49a84 100644 --- a/docs/en/observability/apm/known-issues.asciidoc +++ b/docs/en/observability/apm/known-issues.asciidoc @@ -21,6 +21,50 @@ _Versions: XX.XX.XX, YY.YY.YY, ZZ.ZZ.ZZ_ // If applicable, link to fix //// +[discrete] +== Upgrading to v8.15.x may cause ingestion to fail + +_Elastic Stack versions: 8.15.0_ + + +// The conditions in which this issue occurs +The issue only occurs when _upgrading_ the {stack} from <= 8.12.2 directly to any 8.15.x version. +The issue does _not_ occur when creating a _new_ cluster using any 8.15.x version, or when upgrading +from 8.12.2 to 8.13.x or 8.14.x and then to 8.15.x. + +// Describe why it happens +In APM Servers versions prior to 8.13.0, an ingestion pipeline exists to perform a check on the version. +The version check would fail any APM document produced with a different version of APM server compared to the version of the installed APM’s ingest pipeline. +In 8.13.0 the version check in the ingest pipeline was removed. +Due to the combination of an internal change in how apm data management assets are set up from 8.15 onwards and a bug in Elasticsearch, +related to https://github.com/elastic/elasticsearch/issues/112781[lazy rollover of data streams], the ingestion pipeline conducting the version check is not removed on upgrade and prevents the ingestion of data. + +// How to fix it +If the deployment is running 8.15.0, upgrade the deployment to 8.15.1 or above. +A manual rollover of all APM data streams is required to pick up the new index templates and remove the faulty ingest pipeline version check. +Perform the following requests to Elasticsearch (they are assuming the `default` namespace is used, adjust if necessary): ++ +[source,txt] +---- +POST /traces-apm-default/_rollover +POST /traces-apm.rum-default/_rollover +POST /logs-apm.error-default/_rollover +POST /logs-apm.app-default/_rollover +POST /metrics-apm.app-default/_rollover +POST /metrics-apm.internal-default/_rollover +POST /metrics-apm.service_destination.1m-default/_rollover +POST /metrics-apm.service_destination.10m-default/_rollover +POST /metrics-apm.service_destination.60m-default/_rollover +POST /metrics-apm.service_summary.1m-default/_rollover +POST /metrics-apm.service_summary.10m-default/_rollover +POST /metrics-apm.service_summary.60m-default/_rollover +POST /metrics-apm.service_transaction.1m-default/_rollover +POST /metrics-apm.service_transaction.10m-default/_rollover +POST /metrics-apm.service_transaction.60m-default/_rollover +POST /metrics-apm.transaction.1m-default/_rollover +POST /metrics-apm.transaction.10m-default/_rollover +POST /metrics-apm.transaction.60m-default/_rollover +---- + [discrete] == Upgrading to v8.15.0 may cause APM indices to lose their lifecycle policy