From de083d04fd5c12350eba391d7417cd3a79b9bcd9 Mon Sep 17 00:00:00 2001 From: Carson Ip Date: Thu, 4 Jul 2024 11:10:33 +0100 Subject: [PATCH 1/4] APM known issues: Too many small bulk requests --- .../observability/apm/known-issues.asciidoc | 30 +++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/docs/en/observability/apm/known-issues.asciidoc b/docs/en/observability/apm/known-issues.asciidoc index c7b0937ebf..5fdbc896fa 100644 --- a/docs/en/observability/apm/known-issues.asciidoc +++ b/docs/en/observability/apm/known-issues.asciidoc @@ -258,3 +258,33 @@ Mappings update for metrics-apm.service_destination.10m-default failed due to Re // Link to fix A fix was released in 8.12.2: https://github.com/elastic/apm-server/pull/12219[elastic/apm-server#12219]. + + +*Performance regression: APM issues too many small bulk requests for Elasticsearch output* + +_APM Server versions: >=8.13.0, \<= 8.14.2_ + + +// Describe the conditions in which this issue occurs +If you're on APM server version >=8.13.0, \<= 8.14.2_, using Elasticsearch output, +do not specify any `output.elasticsearch.flush_bytes`, +and do not disable compression explicitly by setting `output.elasticsearch.compression_level` to `0`, +// Describe the behavior of the issue +APM server will issue smaller bulk requests of 24KB size, +and more bulk requests will need to be made to maintain the original throughput. +This causes Elasticsearch to experience higher load, +and APM server may exhibit Elasticsearch backpressure symptoms. + +// Describe why it happens +This happens because a performance regression was introduced, such that the default value of bulk indexer flush bytes +was reduced from 1MB to 24KB. + +It is possible to workaround the issue by configuring Elasticsearch output in APM. + +* For APM Server binary +** In `apm-server.yml`, set `output.elasticsearch.flush_bytes: 1mib` +* For Fleet-managed APM +** In Fleet, open the Settings tab. +** Under Outputs, find the Elasticsearch output that receives from APM, select the edit icon. +** In the Edit output flyout, in the Advanced YAML configuration field, add line `flush_bytes: 1mib`. + +// Link to fix +A fix will be released in 8.15.0: https://github.com/elastic/apm-server/pull/13576[elastic/apm-server#13576]. From 840f3f9bc9be4a619878f98c280637d9224a9130 Mon Sep 17 00:00:00 2001 From: Carson Ip Date: Thu, 4 Jul 2024 13:00:57 +0100 Subject: [PATCH 2/4] Add log --- docs/en/observability/apm/known-issues.asciidoc | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/en/observability/apm/known-issues.asciidoc b/docs/en/observability/apm/known-issues.asciidoc index 5fdbc896fa..69d898c7eb 100644 --- a/docs/en/observability/apm/known-issues.asciidoc +++ b/docs/en/observability/apm/known-issues.asciidoc @@ -277,6 +277,13 @@ and APM server may exhibit Elasticsearch backpressure symptoms. This happens because a performance regression was introduced, such that the default value of bulk indexer flush bytes was reduced from 1MB to 24KB. +Affected APM servers will emit the following log: + +[source,txt] +---- +flush_bytes config value is too small (0) and might be ignored by the indexer, increasing value to 24576 +---- + It is possible to workaround the issue by configuring Elasticsearch output in APM. * For APM Server binary From 1495f44402355c3d2979f88e61ad6e34287d88fc Mon Sep 17 00:00:00 2001 From: Carson Ip Date: Thu, 4 Jul 2024 14:55:18 +0100 Subject: [PATCH 3/4] Add elastic cloud --- docs/en/observability/apm/known-issues.asciidoc | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/en/observability/apm/known-issues.asciidoc b/docs/en/observability/apm/known-issues.asciidoc index 69d898c7eb..a66273e186 100644 --- a/docs/en/observability/apm/known-issues.asciidoc +++ b/docs/en/observability/apm/known-issues.asciidoc @@ -284,14 +284,16 @@ Affected APM servers will emit the following log: flush_bytes config value is too small (0) and might be ignored by the indexer, increasing value to 24576 ---- -It is possible to workaround the issue by configuring Elasticsearch output in APM. +To workaround the issue, modify the Elasticsearch output configuration in APM. * For APM Server binary ** In `apm-server.yml`, set `output.elasticsearch.flush_bytes: 1mib` -* For Fleet-managed APM +* For Fleet-managed APM (non-Elastic Cloud) ** In Fleet, open the Settings tab. -** Under Outputs, find the Elasticsearch output that receives from APM, select the edit icon. -** In the Edit output flyout, in the Advanced YAML configuration field, add line `flush_bytes: 1mib`. +** Under Outputs, identify the Elasticsearch output that receives from APM, select the edit icon. +** In the Edit output flyout, in "Advanced YAML configuration" field, add line `flush_bytes: 1mib`. +* For Elastic Cloud +** It is not possible to edit the Fleet "Elastic Cloud internal output". // Link to fix A fix will be released in 8.15.0: https://github.com/elastic/apm-server/pull/13576[elastic/apm-server#13576]. From 802d4aed2dfe99adb598e2c018ff607fd23ac103 Mon Sep 17 00:00:00 2001 From: Carson Ip Date: Fri, 5 Jul 2024 09:35:16 +0100 Subject: [PATCH 4/4] Pencil in 8.14.3 --- docs/en/observability/apm/known-issues.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/en/observability/apm/known-issues.asciidoc b/docs/en/observability/apm/known-issues.asciidoc index a66273e186..0b87b2df0f 100644 --- a/docs/en/observability/apm/known-issues.asciidoc +++ b/docs/en/observability/apm/known-issues.asciidoc @@ -296,4 +296,4 @@ To workaround the issue, modify the Elasticsearch output configuration in APM. ** It is not possible to edit the Fleet "Elastic Cloud internal output". // Link to fix -A fix will be released in 8.15.0: https://github.com/elastic/apm-server/pull/13576[elastic/apm-server#13576]. +A fix will be released in 8.14.3: https://github.com/elastic/apm-server/pull/13576[elastic/apm-server#13576].