Raftstore v2 #389
Merged
Raftstore v2 #389
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ti-chi-bot
bot
added
do-not-merge/work-in-progress
v01dstar
force-pushed
the
raftstore-v2
branch
2 times, most recently
from
4c52cc0
to
d9a0ac2
Compare
* return sequence number of writes Signed-off-by: 5kbpers <[email protected]> * fix compile error Signed-off-by: 5kbpers <[email protected]> Signed-off-by: tabokie <[email protected]> Signed-off-by: Yang Zhang <[email protected]>
* expose seqno for multi-batch-write Signed-off-by: 5kbpers <[email protected]> * format Signed-off-by: 5kbpers <[email protected]> Signed-off-by: 5kbpers <[email protected]> Signed-off-by: Yang Zhang <[email protected]>
A callback that is called after write succeeds and changes have been applied to memtable. Titan change: tikv/titan#270 Signed-off-by: tabokie <[email protected]> Signed-off-by: Yang Zhang <[email protected]>
* fix bug of using post write callback with empty batch Signed-off-by: tabokie <[email protected]> * fix nullptr Signed-off-by: tabokie <[email protected]> Signed-off-by: tabokie <[email protected]> Signed-off-by: Yang Zhang <[email protected]>
Add support to merge multiple DBs that have no overlapping data (tombstone included). Memtables are frozen and then referenced by the target DB. Table files are hard linked with new file numbers into the target DB. After merge, the sequence numbers of memtables and L0 files will appear out-of-order compared to a single DB. But for any given user key, the ordering still holds because there will only be one unique source DB that contains the key and the source DB's ordering is inherited by the target DB. If source and target instances share the same block cache, target instance will be able to reuse cache. This is done by cloning the table readers of source instances to the target instance. Because the cache key is stored in table reader, reads after the merge can still retrieve source instances' blocks via old cache key. Under release build, it takes 8ms to merge a 25GB DB (500 files) into another. Signed-off-by: tabokie <[email protected]> Signed-off-by: Yang Zhang <[email protected]>
…nFlushBegin event (tikv#300) * add largest seqno of memtable Signed-off-by: 5kbpers <[email protected]> * add test Signed-off-by: 5kbpers <[email protected]> * address comment Signed-off-by: 5kbpers <[email protected]> * address comment Signed-off-by: 5kbpers <[email protected]> * format Signed-off-by: 5kbpers <[email protected]> * memtable info Signed-off-by: 5kbpers <[email protected]> Signed-off-by: 5kbpers <[email protected]> Signed-off-by: Yang Zhang <[email protected]>
Summary: Modify existing write buffer manager to support multiple instances. Previously, a flush is triggered before user writes if `ShouldFlush()` returns true. But in the multiple-instance context, this will cause flushing for all DBs that are undergoing writes. In this patch, column families are registered to a shared linked list inside the write buffer manager. When flush condition is triggered, the column family with highest score from this list will be chosen and flushed. The score can be either size or age. The flush condition calculation is also changed to exclude immutable memtables. This is because RocksDB schedules flush every time an immutable memtable is generated. They will eventually be evicted from memory given the flush bandwidth doesn't bottleneck. Test plan: - Unit test cases - Trigger flush of largest/oldest memtable in another DB - Resolve flush condition by destroy CF/DB - Dynamically change flush threshold - Manual test insert, update, read-write workload, [script](https://gist.github.com/tabokie/d38d27dc3843946c7813ab7bafd0f753). Signed-off-by: tabokie <[email protected]> Signed-off-by: Yang Zhang <[email protected]>
* add toggle Signed-off-by: tabokie <[email protected]> * protect underflow Signed-off-by: tabokie <[email protected]> * fix build Signed-off-by: tabokie <[email protected]> * remove deadline and add penalty for l0 files Signed-off-by: tabokie <[email protected]> * fix build Signed-off-by: tabokie <[email protected]> * consider compaction trigger Signed-off-by: tabokie <[email protected]> --------- Signed-off-by: tabokie <[email protected]> Signed-off-by: Yang Zhang <[email protected]>
* hook delete dir in encrypted env Signed-off-by: tabokie <[email protected]> * add a comment Signed-off-by: tabokie <[email protected]> --------- Signed-off-by: tabokie <[email protected]> Signed-off-by: Yang Zhang <[email protected]>
Signed-off-by: Yang Zhang <[email protected]>
Also added a new options to detect whether manual compaction is disabled. In practice we use this to avoid blocking on flushing a tablet that will be destroyed shortly after. --------- Signed-off-by: tabokie <[email protected]> Signed-off-by: Yang Zhang <[email protected]>
…heckpoint (tikv#338) * fix renaming encrypted directory Signed-off-by: tabokie <[email protected]> * fix build Signed-off-by: tabokie <[email protected]> * patch test manager Signed-off-by: tabokie <[email protected]> * fix build Signed-off-by: tabokie <[email protected]> * check compaction paused during checkpoint Signed-off-by: tabokie <[email protected]> * add comment Signed-off-by: tabokie <[email protected]> --------- Signed-off-by: tabokie <[email protected]> Signed-off-by: Yang Zhang <[email protected]>
Signed-off-by: Yang Zhang <[email protected]>
And delay the buffer initialization of writable file to first actual write. --------- Signed-off-by: tabokie <[email protected]> Signed-off-by: Yang Zhang <[email protected]>
Signed-off-by: Spade A <[email protected]> Signed-off-by: Yang Zhang <[email protected]>
Signed-off-by: SpadeA-Tang <[email protected]> Signed-off-by: Yang Zhang <[email protected]>
Signed-off-by: Yang Zhang <[email protected]>
v01dstar
force-pushed
the
raftstore-v2
branch
3 times, most recently
from
383bdfe
to
72f0962
Compare
|
/run-all-tests |
|
/run-all-tests |
Signed-off-by: Yang Zhang <[email protected]>
ti-chi-bot
bot
removed
the
do-not-merge/work-in-progress
Connor1996
approved these changes
Oct 8, 2024
ti-chi-bot
bot
added
needs-1-more-lgtm
LykxSassinator
approved these changes
Oct 8, 2024
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: Connor1996, LykxSassinator The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
ti-chi-bot
bot
added
lgtm
and removed
needs-1-more-lgtm
[LGTM Timeline notifier]Timeline:
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Cherry-picking following PRs