Skip to content

Commit

Permalink
APM known issues: Too many small bulk requests (#4047)
Browse files Browse the repository at this point in the history
* APM known issues: Too many small bulk requests

* Add log

* Add elastic cloud

* Pencil in 8.14.3

(cherry picked from commit 0335c16)
  • Loading branch information
carsonip authored and mergify[bot] committed Jul 18, 2024
1 parent 455f869 commit 3063788
Showing 1 changed file with 39 additions and 0 deletions.
39 changes: 39 additions & 0 deletions docs/en/observability/apm/known-issues.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -258,3 +258,42 @@ Mappings update for metrics-apm.service_destination.10m-default failed due to Re

// Link to fix
A fix was released in 8.12.2: https://github.com/elastic/apm-server/pull/12219[elastic/apm-server#12219].


*Performance regression: APM issues too many small bulk requests for Elasticsearch output* +
_APM Server versions: >=8.13.0, \<= 8.14.2_ +

// Describe the conditions in which this issue occurs
If you're on APM server version >=8.13.0, \<= 8.14.2_, using Elasticsearch output,
do not specify any `output.elasticsearch.flush_bytes`,
and do not disable compression explicitly by setting `output.elasticsearch.compression_level` to `0`,
// Describe the behavior of the issue
APM server will issue smaller bulk requests of 24KB size,
and more bulk requests will need to be made to maintain the original throughput.
This causes Elasticsearch to experience higher load,
and APM server may exhibit Elasticsearch backpressure symptoms.

// Describe why it happens
This happens because a performance regression was introduced, such that the default value of bulk indexer flush bytes
was reduced from 1MB to 24KB.

Affected APM servers will emit the following log:

[source,txt]
----
flush_bytes config value is too small (0) and might be ignored by the indexer, increasing value to 24576
----

To workaround the issue, modify the Elasticsearch output configuration in APM.

* For APM Server binary
** In `apm-server.yml`, set `output.elasticsearch.flush_bytes: 1mib`
* For Fleet-managed APM (non-Elastic Cloud)
** In Fleet, open the Settings tab.
** Under Outputs, identify the Elasticsearch output that receives from APM, select the edit icon.
** In the Edit output flyout, in "Advanced YAML configuration" field, add line `flush_bytes: 1mib`.
* For Elastic Cloud
** It is not possible to edit the Fleet "Elastic Cloud internal output".

// Link to fix
A fix will be released in 8.14.3: https://github.com/elastic/apm-server/pull/13576[elastic/apm-server#13576].

0 comments on commit 3063788

Please sign in to comment.