Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

APM known issues: Too many small bulk requests #4047

Merged
merged 4 commits into from
Jul 18, 2024
Merged

APM known issues: Too many small bulk requests #4047

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
de083d0
APM known issues: Too many small bulk requests
carsonip Jul 4, 2024
840f3f9
Add log
carsonip Jul 4, 2024
1495f44
Add elastic cloud
carsonip Jul 4, 2024
802d4ae
Pencil in 8.14.3
carsonip Jul 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 39 additions & 0 deletions docs/en/observability/apm/known-issues.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -258,3 +258,42 @@ Mappings update for metrics-apm.service_destination.10m-default failed due to Re

// Link to fix
A fix was released in 8.12.2: https://github.com/elastic/apm-server/pull/12219[elastic/apm-server#12219].


*Performance regression: APM issues too many small bulk requests for Elasticsearch output* +
_APM Server versions: >=8.13.0, \<= 8.14.2_ +

// Describe the conditions in which this issue occurs
If you're on APM server version >=8.13.0, \<= 8.14.2_, using Elasticsearch output,
do not specify any `output.elasticsearch.flush_bytes`,
and do not disable compression explicitly by setting `output.elasticsearch.compression_level` to `0`,
// Describe the behavior of the issue
APM server will issue smaller bulk requests of 24KB size,
and more bulk requests will need to be made to maintain the original throughput.
This causes Elasticsearch to experience higher load,
and APM server may exhibit Elasticsearch backpressure symptoms.

// Describe why it happens
This happens because a performance regression was introduced, such that the default value of bulk indexer flush bytes
was reduced from 1MB to 24KB.

Affected APM servers will emit the following log:

[source,txt]
----
flush_bytes config value is too small (0) and might be ignored by the indexer, increasing value to 24576
----

To workaround the issue, modify the Elasticsearch output configuration in APM.

* For APM Server binary
** In `apm-server.yml`, set `output.elasticsearch.flush_bytes: 1mib`
* For Fleet-managed APM (non-Elastic Cloud)
** In Fleet, open the Settings tab.
** Under Outputs, identify the Elasticsearch output that receives from APM, select the edit icon.
** In the Edit output flyout, in "Advanced YAML configuration" field, add line `flush_bytes: 1mib`.
* For Elastic Cloud
** It is not possible to edit the Fleet "Elastic Cloud internal output".

// Link to fix
A fix will be released in 8.14.3: https://github.com/elastic/apm-server/pull/13576[elastic/apm-server#13576].