Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

APM known issues: Too many small bulk requests #4047

Merged
merged 4 commits into from
Jul 18, 2024
Merged

APM known issues: Too many small bulk requests #4047

Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
de083d0
APM known issues: Too many small bulk requests
carsonip Jul 4, 2024
840f3f9
Add log
carsonip Jul 4, 2024
1495f44
Add elastic cloud
carsonip Jul 4, 2024
802d4ae
Pencil in 8.14.3
carsonip Jul 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions docs/en/observability/apm/known-issues.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -258,3 +258,33 @@ Mappings update for metrics-apm.service_destination.10m-default failed due to Re

// Link to fix
A fix was released in 8.12.2: https://github.com/elastic/apm-server/pull/12219[elastic/apm-server#12219].


*Performance regression: APM issues too many small bulk requests for Elasticsearch output* +
_APM Server versions: >=8.13.0, \<= 8.14.2_ +

// Describe the conditions in which this issue occurs
If you're on APM server version >=8.13.0, \<= 8.14.2_, using Elasticsearch output,
do not specify any `output.elasticsearch.flush_bytes`,
and do not disable compression explicitly by setting `output.elasticsearch.compression_level` to `0`,
// Describe the behavior of the issue
APM server will issue smaller bulk requests of 24KB size,
and more bulk requests will need to be made to maintain the original throughput.
This causes Elasticsearch to experience higher load,
and APM server may exhibit Elasticsearch backpressure symptoms.

// Describe why it happens
This happens because a performance regression was introduced, such that the default value of bulk indexer flush bytes
was reduced from 1MB to 24KB.

It is possible to workaround the issue by configuring Elasticsearch output in APM.

* For APM Server binary
** In `apm-server.yml`, set `output.elasticsearch.flush_bytes: 1mib`
* For Fleet-managed APM
carsonip marked this conversation as resolved.
Show resolved Hide resolved
** In Fleet, open the Settings tab.
** Under Outputs, find the Elasticsearch output that receives from APM, select the edit icon.
** In the Edit output flyout, in the Advanced YAML configuration field, add line `flush_bytes: 1mib`.

// Link to fix
A fix will be released in 8.15.0: https://github.com/elastic/apm-server/pull/13576[elastic/apm-server#13576].
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we keep this open to the possibility of having the fix added to 8.14.3?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Current info points at the earliest availability at 8.15.0. We can always update the docs if anything changes.

Copy link
Member Author

@carsonip carsonip Jul 4, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On a separate note, I'll leave it to @bmorelli25 to suggest a more efficient workflow. Currently I'll update the docs when the bugfix is actually released, but there may be a way to word it so that we don't need to change "A fix will be released..." to "A fix was released..." later.