APM known issues: Too many small bulk requests (#4047) (#4082)

* APM known issues: Too many small bulk requests * Add log * Add elastic cloud * Pencil in 8.14.3 (cherry picked from commit 0335c16) Co-authored-by: Carson Ip <[email protected]>
elastic · Jul 23, 2024 · 7fc10a8 · 7fc10a8
1 parent 28126a2
commit 7fc10a8
Showing 1 changed file with 39 additions and 0 deletions.
diff --git a/docs/en/observability/apm/known-issues.asciidoc b/docs/en/observability/apm/known-issues.asciidoc
@@ -258,3 +258,42 @@ Mappings update for metrics-apm.service_destination.10m-default failed due to Re
 
 // Link to fix
 A fix was released in 8.12.2: https://github.com/elastic/apm-server/pull/12219[elastic/apm-server#12219].
+
+
+*Performance regression: APM issues too many small bulk requests for Elasticsearch output* +
+_APM Server versions: >=8.13.0, \<= 8.14.2_ +
+
+// Describe the conditions in which this issue occurs
+If you're on APM server version >=8.13.0, \<= 8.14.2_, using Elasticsearch output,
+do not specify any `output.elasticsearch.flush_bytes`,
+and do not disable compression explicitly by setting `output.elasticsearch.compression_level` to `0`,
+// Describe the behavior of the issue
+APM server will issue smaller bulk requests of 24KB size,
+and more bulk requests will need to be made to maintain the original throughput.
+This causes Elasticsearch to experience higher load,
+and APM server may exhibit Elasticsearch backpressure symptoms.
+
+// Describe why it happens
+This happens because a performance regression was introduced, such that the default value of bulk indexer flush bytes
+was reduced from 1MB to 24KB.
+
+Affected APM servers will emit the following log:
+
+[source,txt]
+----
+flush_bytes config value is too small (0) and might be ignored by the indexer, increasing value to 24576
+----
+
+To workaround the issue, modify the Elasticsearch output configuration in APM.
+
+* For APM Server binary
+** In `apm-server.yml`, set `output.elasticsearch.flush_bytes: 1mib`
+* For Fleet-managed APM (non-Elastic Cloud)
+** In Fleet, open the Settings tab.
+** Under Outputs, identify the Elasticsearch output that receives from APM, select the edit icon.
+** In the Edit output flyout, in "Advanced YAML configuration" field, add line `flush_bytes: 1mib`.
+* For Elastic Cloud
+** It is not possible to edit the Fleet "Elastic Cloud internal output".
+
+// Link to fix
+A fix will be released in 8.14.3: https://github.com/elastic/apm-server/pull/13576[elastic/apm-server#13576].