Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: CLI add rate limit logic to gaia dump and restore #1607

Merged
merged 2 commits into from
Dec 18, 2023

Conversation

pradel
Copy link
Contributor

@pradel pradel commented Dec 16, 2023

Description

Added some sleep logic in the CLI to wait between batches for gaia_dump_bucket and gaia_restore_bucket as some users have been hitting the rate limit when dumping their Gaia bucket as they have many files (> 200).

I did manually test the logic and with the current config not hitting the rate limit, uploading 99 files every mins does hit the rate limit so switched to 99 every 2 mins to get it to work.

Breaking change?

No

Example

Create a bucket with many files and see that you can download/upload all of them without hitting the rate limit.


Checklist

Copy link

vercel bot commented Dec 16, 2023

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
stacksjs-docs ✅ Ready (Inspect) Visit Preview 💬 Add feedback Dec 16, 2023 3:16pm

Copy link

codecov bot commented Dec 16, 2023

Codecov Report

Attention: 27 lines in your changes are missing coverage. Please review.

Comparison is base (25b3994) 66.35% compared to head (e0f970d) 66.28%.

Files Patch % Lines
packages/cli/src/cli.ts 0.00% 27 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1607      +/-   ##
==========================================
- Coverage   66.35%   66.28%   -0.07%     
==========================================
  Files         119      119              
  Lines        8699     8708       +9     
  Branches     1916     1919       +3     
==========================================
  Hits         5772     5772              
- Misses       2802     2811       +9     
  Partials      125      125              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@janniks
Copy link
Collaborator

janniks commented Dec 18, 2023

Thanks, 👏 Looks good

@janniks janniks merged commit 445a420 into hirosystems:main Dec 18, 2023
9 of 10 checks passed
@pradel pradel deleted the feat/sleep-gaia-dump-rate-limit branch December 18, 2023 09:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

3 participants