Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf(s3stream): speedup wal recovery #910

Merged
merged 12 commits into from
Jan 30, 2024
Merged

Conversation

Chillax-0v0
Copy link
Contributor

@Chillax-0v0 Chillax-0v0 commented Jan 30, 2024

resolve AutoMQ/automq#573

On AWS EBS gp3 (3000 IOPS, 125 MB/s):
Recover 65536 messages (16 KiB each, 1 GiB in total) cost 92.7s -> 15.3s
Recover 65536 messages (4 KiB each, 256 MiB in total) cost 65.8s -> 2.5s

@Chillax-0v0 Chillax-0v0 added enhancement New feature or request s3stream wal the WAL of S3 stream labels Jan 30, 2024
@Chillax-0v0 Chillax-0v0 self-assigned this Jan 30, 2024
Copy link

codecov bot commented Jan 30, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (db7af3a) 55.94% compared to head (4f26fba) 55.92%.

Additional details and impacted files
@@             Coverage Diff              @@
##               main     #910      +/-   ##
============================================
- Coverage     55.94%   55.92%   -0.02%     
  Complexity     1268     1268              
============================================
  Files           133      133              
  Lines          8534     8534              
  Branches        789      789              
============================================
- Hits           4774     4773       -1     
  Misses         3327     3327              
- Partials        433      434       +1     
Components Coverage Δ
RocketMQ Broker ∅ <ø> (∅)
RocketMQ Common 5.80% <ø> (ø)
RocketMQ Controller 57.39% <ø> (ø)
RocketMQ Proxy 37.28% <ø> (ø)
RocketMQ Store 73.60% <ø> (-0.05%) ⬇️
RocketMQ Stream ∅ <ø> (∅)
RocketMQ Metadata 35.44% <ø> (ø)

see 1 file with indirect coverage changes

@Chillax-0v0 Chillax-0v0 force-pushed the perf-s3stream-wal-recovery-speedup branch from ad9ba3a to 927ad8e Compare January 30, 2024 12:33
@Chillax-0v0 Chillax-0v0 marked this pull request as draft January 30, 2024 12:35
Signed-off-by: Ning Yu <[email protected]>
@Chillax-0v0 Chillax-0v0 marked this pull request as ready for review January 30, 2024 12:35
@superhx superhx merged commit f980595 into main Jan 30, 2024
7 checks passed
@superhx superhx deleted the perf-s3stream-wal-recovery-speedup branch January 30, 2024 12:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request s3stream wal the WAL of S3 stream
Projects
None yet
Development

Successfully merging this pull request may close these issues.

WAL unclean shutdown recovery need speed up
2 participants