This repository has been archived by the owner on Feb 18, 2025. It is now read-only.
eth,les/downloader: improve the throttle threshold when syncing #272
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
minh-bq
force-pushed
the
feat/improve-sync-throttle
branch
2 times, most recently
from
40d4167 to
cc08041
Compare
minh-bq
force-pushed
the
feat/improve-sync-throttle
branch
from
cc08041 to
977d580
Compare
minh-bq
changed the title
huyngopt1994
approved these changes
Feb 20, 2024
Currently, the new threshold is calculated when fetching the result from result queue and is applied to the next reserve/fetch result. However, the result processing may take too much time when data is large, the threshold is not updated early enough to slow down further reserve/fetch. As a result, the memory can peak so high. In this commit, we re-calculate threshold when handling response. In case the result queue is throttled, we discard the response and queue the fetch task back to queue for later re-fetch. This may increase the cost in network bandwidth but help to regulate the memory usage.
minh-bq
force-pushed
the
feat/improve-sync-throttle
branch
from
977d580 to
b8ed269
Compare
Francesco4203
pushed a commit
to Francesco4203/ronin
that referenced
this pull request
Jun 18, 2024
Currently, the new threshold is calculated when fetching the result from result queue and is applied to the next reserve/fetch result. However, the result processing may take too much time when data is large, the threshold is not updated early enough to slow down further reserve/fetch. As a result, the memory can peak so high. In this commit, we re-calculate threshold when handling response. In case the result queue is throttled, we discard the response and queue the fetch task back to queue for later re-fetch. This may increase the cost in network bandwidth but help to regulate the memory usage.
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently, the new threshold is calculated when fetching the result from result queue and is applied to the next reserve/fetch result. However, the result processing may take too much time when data is large, the threshold is not updated early enough to slow down further reserve/fetch. As a result, the memory can peak so high. In this commit, we re-calculate threshold when handling response. In case the result queue is throttled, we discard the response and queue the fetch task back to queue for later re-fetch. This may increase the cost in network bandwidth but help to regulate the memory usage.