[Es Archiver][Load Action] High Water Mark to 5000, Concurrency 4; Override-able #167993
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
wayneseymour
added
release_note:skip
wayneseymour
changed the title
|
@elasticmachine merge upstream |
1 similar comment
|
@elasticmachine merge upstream |
wayneseymour
force-pushed
the
esarch-fixed-higher-watermark-impl-minimal-code-chages-without-benchmark-tool
branch
8 times, most recently
from
51c2cd0
to
6bfb9b1
Compare
wayneseymour
force-pushed
the
esarch-fixed-higher-watermark-impl-minimal-code-chages-without-benchmark-tool
branch
6 times, most recently
from
fe67122
to
2b885fb
Compare
wayneseymour
force-pushed
the
esarch-fixed-higher-watermark-impl-minimal-code-chages-without-benchmark-tool
branch
3 times, most recently
from
b9f0d5d
to
336a21d
Compare
wayneseymour
force-pushed
the
esarch-fixed-higher-watermark-impl-minimal-code-chages-without-benchmark-tool
branch
2 times, most recently
from
8940657
to
050e4ab
Compare
This was referenced Nov 10, 2023
|
@elasticmachine merge upstream |
…ode-chages-without-benchmark-tool
|
/ci |
…without-benchmark-tool' of github.com:wayneseymour/kibana into esarch-fixed-higher-watermark-impl-minimal-code-chages-without-benchmark-tool
|
/ci |
|
/ci |
|
/ci |
wayneseymour
changed the title
|
/ci |
wayneseymour
changed the title
|
@elasticmachine merge upstream |
…ode-chages-without-benchmark-tool
|
/ci |
…without-benchmark-tool' of github.com:wayneseymour/kibana into esarch-fixed-higher-watermark-impl-minimal-code-chages-without-benchmark-tool
|
@elasticmachine merge upstream |
…ode-chages-without-benchmark-tool
|
/ci |
💔 Build FailedFailed CI Steps
Test Failures
Metrics [docs]Public APIs missing comments
History
To update your PR or re-run it, just comment with: |
|
Closing in order to have less reviewers, in a new PR. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
NEW CHANGE COMING
Summary
To speed up the load action in general, and in serverless in particular, this pr:
Increases stream batch size from
300to5000.Also, it's now using concurrency from the bulk call api, set to
4.Summary of Performance Characteristics
Using a list of over 70 archives from within the repo, we've benchmarked the speed before changing the performance characteristics, and after.
One champion example is the
x-pack/test/functional/es_archives/getting_started/shakespearearchive.It's load time decreased from ~2.5 minutes, to less than 18 seconds, in a serverless environment.
There are a little over a handful where the time actually got worse, but most are only milliseconds slower.
None are a full second slower.