Fix/slow cmu #35
Merged
Fix/slow cmu #35
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rluetzner
force-pushed
the
fix/slow-cmu-rb
branch
from
93f1b8a to
027f542
Compare
added 3 commits
June 22, 2022 11:40
There's no need to continously stat all object parts on disk if most of them have already been appended. (cherry picked from commit 3d4fb6a)
The backgroundAppend was meant as a performance optimization where uploaded chunks would be aggregated to a complete file in the background while more parts were uploaded at the same time. This might be an optimization on actual filesystem paths, but on Gluster it made the operation a lot slower because many redundant filesystem calls were executed. The CompleteMultipartUpload request which requires the uploaded chunks to be put together has to 1. wait for any ongoing backgroundAppend operations to complete, 2. enumerate all chunks and check if they were put together in the right order (otherwise start from scratch), 3. move the aggregated file. Removing this piece of code started out as an experiment, because I expected the chunks to not be aggregated at all. It turned out that there is a fallback which is also necessary in case the final object should have a different order or not contain some of the uploaded parts. This also ensures that a final aggregate is created. At least on GlusterFS this makes any CMU request run almost twice as fast as when using backgroundAppend. (cherry picked from commit 4ac0b96)
(cherry picked from commit 9137a44)
rluetzner
force-pushed
the
fix/slow-cmu-rb
branch
from
027f542 to
a00b604
Compare
|
We verified that the change works and CompleteMultipartRequests are indeed much faster than before. There are no visible regressions in the unit tests or the Mint test suite. |
aweisser
approved these changes
Jun 23, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
In combination with a GlusterFS backend we realized that the CompleteMultipartUpload request becomes very slow. The larger the uploaded filesize is, the longer the request will take.
This is reapplying #19 . The code hasn't changed since being reviewed, but we previously decided not to include it in our next release.
Motivation and Context
We noticed that for larger filesizes some clients would run into a timeout during CompleteMultipartUpload requests. The default timeout seems to be 1h, which is pretty big. Nevertheless uploading 20 or 50GiB files caused CMU requests that took way longer to complete.
Compared to AWS it is non-intuitive that a CMU request should take so long, but we're willing to accept it, because it's a limitation of the current implementation, which has to put all chunks together into one big file.
This fixes #13 and #11 .
How to test this PR?
Tests can easily be performed manually.
timeis helpful in tracking the total duration on the client side, whereasmc admin trace -v -awill log all requests including their duration.If large files are required
truncate -s 100G filecan help you quickly create a file containing only zeros. Without using compression, this is just as good as creating such a big file with actual content, but waaaaay faster.Types of changes
Checklist:
commit-idorPR #here)