From e1ccc3bd2530b771e7015d9405b6ef7e2c45c250 Mon Sep 17 00:00:00 2001 From: inge4pres Date: Thu, 31 Oct 2024 13:10:14 +0100 Subject: [PATCH 1/7] APM: add known issue of lazy rollover bug Signed-off-by: inge4pres --- .../observability/apm/known-issues.asciidoc | 45 +++++++++++++++++++ 1 file changed, 45 insertions(+) diff --git a/docs/en/observability/apm/known-issues.asciidoc b/docs/en/observability/apm/known-issues.asciidoc index 3ed733cd2c..ea8c1f84db 100644 --- a/docs/en/observability/apm/known-issues.asciidoc +++ b/docs/en/observability/apm/known-issues.asciidoc @@ -21,6 +21,51 @@ _Versions: XX.XX.XX, YY.YY.YY, ZZ.ZZ.ZZ_ // If applicable, link to fix //// +[discrete] +== Upgrading to v8.15.0 may cause ingestion to fail + +_Elastic Stack versions: 8.15.0_ + +_Fixed in Elastic Stack version 8.15.2_ + +// The conditions in which this issue occurs +The issue only occurs when _upgrading_ the {stack} from 8.12.2 directly to any 8.15.x version. +The issue does _not_ occur when creating a _new_ cluster using any 8.15.x version, or when upgrading +from 8.12.2 to 8.13.x and then to 8.15.x. + +// Describe why it happens +In APM Servers versions prior to 8.13.0, an ingestion pipeline exists to perform a check on the version. +The version check would fail any APM document produced with a different version of APM server compared to the version of the installed APM’s ingest pipeline. +In 8.13.0 the version check in the ingest pipeline was removed. +However, in 8.15.0 APM Server has a different way of managing data streams, that is not aware of the version check present in the ingest pipeline. +Thus, the leftover version check remains and prevents the ingestion of data. + +// How to fix it +If the deployment is running 8.15.0, upgrade the deployment to 8.15.1 or above. +A manual rollover of all APM data streams is required to pick up the new index templates and remove the faulty ingest pipeline version check. +Perform the following requests (they are assuming the `default` namespace is used, adjust if necessary): ++ +[source,txt] +---- +POST /traces-apm-default/_rollover +POST /traces-apm.rum-default/_rollover +POST /logs-apm.error-default/_rollover +POST /logs-apm.app-default/_rollover +POST /metrics-apm.app-default/_rollover +POST /metrics-apm.internal-default/_rollover +POST /metrics-apm.service_destination.1m-default/_rollover +POST /metrics-apm.service_destination.10m-default/_rollover +POST /metrics-apm.service_destination.60m-default/_rollover +POST /metrics-apm.service_summary.1m-default/_rollover +POST /metrics-apm.service_summary.10m-default/_rollover +POST /metrics-apm.service_summary.60m-default/_rollover +POST /metrics-apm.service_transaction.1m-default/_rollover +POST /metrics-apm.service_transaction.10m-default/_rollover +POST /metrics-apm.service_transaction.60m-default/_rollover +POST /metrics-apm.transaction.1m-default/_rollover +POST /metrics-apm.transaction.10m-default/_rollover +POST /metrics-apm.transaction.60m-default/_rollover +---- + [discrete] == Upgrading to v8.15.0 may cause APM indices to lose their lifecycle policy From 98c4a85ba49ea01ad6c3614de5e5000ac7e2fa8f Mon Sep 17 00:00:00 2001 From: inge4pres Date: Thu, 31 Oct 2024 13:14:00 +0100 Subject: [PATCH 2/7] correct fix version Signed-off-by: inge4pres --- docs/en/observability/apm/known-issues.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/en/observability/apm/known-issues.asciidoc b/docs/en/observability/apm/known-issues.asciidoc index ea8c1f84db..be7e4afb1d 100644 --- a/docs/en/observability/apm/known-issues.asciidoc +++ b/docs/en/observability/apm/known-issues.asciidoc @@ -25,7 +25,7 @@ _Versions: XX.XX.XX, YY.YY.YY, ZZ.ZZ.ZZ_ == Upgrading to v8.15.0 may cause ingestion to fail _Elastic Stack versions: 8.15.0_ + -_Fixed in Elastic Stack version 8.15.2_ +_Fixed in Elastic Stack version 8.15.1_ // The conditions in which this issue occurs The issue only occurs when _upgrading_ the {stack} from 8.12.2 directly to any 8.15.x version. From 9205264b685c43816519b0b28ef9b8a5bd1b4289 Mon Sep 17 00:00:00 2001 From: inge4pres Date: Thu, 31 Oct 2024 13:14:54 +0100 Subject: [PATCH 3/7] clarify resolution steps Signed-off-by: inge4pres --- docs/en/observability/apm/known-issues.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/en/observability/apm/known-issues.asciidoc b/docs/en/observability/apm/known-issues.asciidoc index be7e4afb1d..40ccc46586 100644 --- a/docs/en/observability/apm/known-issues.asciidoc +++ b/docs/en/observability/apm/known-issues.asciidoc @@ -42,7 +42,7 @@ Thus, the leftover version check remains and prevents the ingestion of data. // How to fix it If the deployment is running 8.15.0, upgrade the deployment to 8.15.1 or above. A manual rollover of all APM data streams is required to pick up the new index templates and remove the faulty ingest pipeline version check. -Perform the following requests (they are assuming the `default` namespace is used, adjust if necessary): +Perform the following requests to Elasticsearch (they are assuming the `default` namespace is used, adjust if necessary): + [source,txt] ---- From 2e7fc97708ccd16c42c5bd3caf2a6d2d265c77f2 Mon Sep 17 00:00:00 2001 From: Francesco Gualazzi Date: Thu, 31 Oct 2024 13:47:15 +0100 Subject: [PATCH 4/7] suggestions from code review Co-authored-by: simitt --- docs/en/observability/apm/known-issues.asciidoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/en/observability/apm/known-issues.asciidoc b/docs/en/observability/apm/known-issues.asciidoc index 40ccc46586..88d19acbbc 100644 --- a/docs/en/observability/apm/known-issues.asciidoc +++ b/docs/en/observability/apm/known-issues.asciidoc @@ -28,9 +28,9 @@ _Elastic Stack versions: 8.15.0_ + _Fixed in Elastic Stack version 8.15.1_ // The conditions in which this issue occurs -The issue only occurs when _upgrading_ the {stack} from 8.12.2 directly to any 8.15.x version. +The issue only occurs when _upgrading_ the {stack} from <= 8.12.2 directly to any 8.15.x version. The issue does _not_ occur when creating a _new_ cluster using any 8.15.x version, or when upgrading -from 8.12.2 to 8.13.x and then to 8.15.x. +from 8.12.2 to 8.13 or 8.14 and then to 8.15.x. // Describe why it happens In APM Servers versions prior to 8.13.0, an ingestion pipeline exists to perform a check on the version. From 8205f6aa81679759a816df7eb8f89d86b490b55b Mon Sep 17 00:00:00 2001 From: inge4pres Date: Thu, 31 Oct 2024 13:54:28 +0100 Subject: [PATCH 5/7] clarify source of issue Signed-off-by: inge4pres --- docs/en/observability/apm/known-issues.asciidoc | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/docs/en/observability/apm/known-issues.asciidoc b/docs/en/observability/apm/known-issues.asciidoc index 88d19acbbc..311574553c 100644 --- a/docs/en/observability/apm/known-issues.asciidoc +++ b/docs/en/observability/apm/known-issues.asciidoc @@ -25,8 +25,7 @@ _Versions: XX.XX.XX, YY.YY.YY, ZZ.ZZ.ZZ_ == Upgrading to v8.15.0 may cause ingestion to fail _Elastic Stack versions: 8.15.0_ + -_Fixed in Elastic Stack version 8.15.1_ - + // The conditions in which this issue occurs The issue only occurs when _upgrading_ the {stack} from <= 8.12.2 directly to any 8.15.x version. The issue does _not_ occur when creating a _new_ cluster using any 8.15.x version, or when upgrading @@ -36,8 +35,8 @@ from 8.12.2 to 8.13 or 8.14 and then to 8.15.x. In APM Servers versions prior to 8.13.0, an ingestion pipeline exists to perform a check on the version. The version check would fail any APM document produced with a different version of APM server compared to the version of the installed APM’s ingest pipeline. In 8.13.0 the version check in the ingest pipeline was removed. -However, in 8.15.0 APM Server has a different way of managing data streams, that is not aware of the version check present in the ingest pipeline. -Thus, the leftover version check remains and prevents the ingestion of data. +Due to the combination of an internal change in how apm data management assets are set up from 8.15 onwards and a bug in Elasticsearch, +related to https://github.com/elastic/elasticsearch/issues/112781[lazy rollover of data streams], the ingestion pipeline conducting the version check is not removed on upgrade and prevents the ingestion of data. // How to fix it If the deployment is running 8.15.0, upgrade the deployment to 8.15.1 or above. From c90a271b27c485d9a42086f877b7318e377153ae Mon Sep 17 00:00:00 2001 From: inge4pres Date: Thu, 31 Oct 2024 13:54:51 +0100 Subject: [PATCH 6/7] clarify impact Signed-off-by: inge4pres --- docs/en/observability/apm/known-issues.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/en/observability/apm/known-issues.asciidoc b/docs/en/observability/apm/known-issues.asciidoc index 311574553c..827b5f6009 100644 --- a/docs/en/observability/apm/known-issues.asciidoc +++ b/docs/en/observability/apm/known-issues.asciidoc @@ -22,7 +22,7 @@ _Versions: XX.XX.XX, YY.YY.YY, ZZ.ZZ.ZZ_ //// [discrete] -== Upgrading to v8.15.0 may cause ingestion to fail +== Upgrading to v8.15.x may cause ingestion to fail _Elastic Stack versions: 8.15.0_ + From 5409e0bd44f9585fabda346dfe68f58971245fbe Mon Sep 17 00:00:00 2001 From: inge4pres Date: Thu, 31 Oct 2024 13:55:23 +0100 Subject: [PATCH 7/7] consistent versioning Signed-off-by: inge4pres --- docs/en/observability/apm/known-issues.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/en/observability/apm/known-issues.asciidoc b/docs/en/observability/apm/known-issues.asciidoc index 827b5f6009..c18ca49a84 100644 --- a/docs/en/observability/apm/known-issues.asciidoc +++ b/docs/en/observability/apm/known-issues.asciidoc @@ -29,7 +29,7 @@ _Elastic Stack versions: 8.15.0_ + // The conditions in which this issue occurs The issue only occurs when _upgrading_ the {stack} from <= 8.12.2 directly to any 8.15.x version. The issue does _not_ occur when creating a _new_ cluster using any 8.15.x version, or when upgrading -from 8.12.2 to 8.13 or 8.14 and then to 8.15.x. +from 8.12.2 to 8.13.x or 8.14.x and then to 8.15.x. // Describe why it happens In APM Servers versions prior to 8.13.0, an ingestion pipeline exists to perform a check on the version.