Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a configuration option to limit the number of simultaneous RESTORE threads #125

Open
RKrahl opened this issue Aug 11, 2022 · 0 comments
Labels
enhancement New feature or request
Milestone

Comments

@RKrahl
Copy link
Member

RKrahl commented Aug 11, 2022

It may happen that a large number of RESTORE threads is triggered at once, just consider requesting a restore of an investigation, that will translate into starting a RESTORE thread for each dataset in that investigation. So the number of simultaneous RESTORE threads may easily raise to a few hundreds. RESTORE operations are relative expensive in terms of I/O as they need to write many files. It has been observed that such an event may seriously degrade server performance. Furthermore, RESTORE operations are usually less urgent than WRITE (which makes sure data is saved to archive) or ARCHIVE (which may need to be done quickly to prevent hitting the size limit of main storage). So it makes sense to prioritize the latter.

Therefore I suggest there should be a configuration option that would set a limit to the number of RESTORE threads that are started simultaneously. The exceeding RESTORE requests would remain waiting in the queue and would be carried out as soon as there is free capacity, so this shouldn't hurt much.

@RKrahl RKrahl added the enhancement New feature or request label Aug 11, 2022
@RKrahl RKrahl added this to the 3.0.0 milestone Dec 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant