-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
raftstore-v2: support column family based write buffer manager #15453
Conversation
Signed-off-by: SpadeA-Tang <[email protected]>
[REVIEW NOTIFICATION] This pull request has been approved by:
To complete the pull request process, please ask the reviewers in the list to review by filling The full list of commands accepted by this bot can be found here. Reviewer can indicate their review by submitting an approval review. |
Signed-off-by: SpadeA-Tang <[email protected]>
Signed-off-by: SpadeA-Tang <[email protected]>
src/config/mod.rs
Outdated
@@ -1330,6 +1336,7 @@ impl Default for DbConfig { | |||
enable_unordered_write: false, | |||
allow_concurrent_memtable_write: None, | |||
write_buffer_limit: None, | |||
lock_write_buffer_limit: None, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Putting it in CfConfig makes more sense.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
Signed-off-by: SpadeA-Tang <[email protected]>
src/config/mod.rs
Outdated
|| self.defaultcf.write_buffer_limit.is_some() | ||
|| self.raftcf.write_buffer_limit.is_some() | ||
{ | ||
return Err( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can support them all, as long as write_buffer_stall_ratio
is 0.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's better to not support now? Expose this when we have use cases and relevant test.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The code is very simple (one modification in build_cf_resources
) compared to the benefit. When you forbid a configuration it would be hard to work around it when needed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Okay,done
Signed-off-by: SpadeA-Tang <[email protected]>
components/engine_rocks/Cargo.toml
Outdated
@@ -57,7 +57,8 @@ tracker = { workspace = true } | |||
txn_types = { workspace = true } | |||
|
|||
[dependencies.rocksdb] | |||
git = "https://github.com/tikv/rust-rocksdb.git" | |||
git = "https://github.com/SpadeA-Tang/rust-rocksdb.git" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
make sure this is updated.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sure
Signed-off-by: SpadeA-Tang <[email protected]>
/run-test retry=5 |
/merge |
@SpadeA-Tang: It seems you want to merge this PR, I will help you trigger all the tests: /run-all-tests You only need to trigger
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository. |
This pull request has been accepted and is ready to merge. Commit hash: f668a00
|
In response to a cherrypick label: new pull request created to branch |
ref tikv#12842 Signed-off-by: ti-chi-bot <[email protected]>
What is changed and how it works?
Issue Number: Ref #12842
What's Changed:
This PR (include related PRs) enables cf based write buffer manager, so that we can use a
separate write buffer manager to mange lock cf.
Compared with default and write cfs, lock cf normally uses and should use significantly less memory, and making all cfs
use a single write buffer manager actually means lock cf has no limit.
Sometimes especially when there's heavy coprocessor requests with insert request, we wish lock cf memtable flushes and compactions timely to make tombstone be removed quickly which can be done by using a relatively small lock write buffer limit after this PR. YCSB workload e demonstrates good for this senario:
ycsb workload e comparison:
qps increases around 15.2%. Both p99 latency and tombstone met reduced a lot.
metrics listed below:
master:
command per second
coprocessor p99 duration
tombstone
this PR
command per second
coprocessor p99 duration
tombstone
Related changes
tikv/rust-rocksdb#763
tikv/rocksdb#343
Check List
Tests
Release note