enable cf uses separete write buffer manager #343

SpadeA-Tang · 2023-07-25T08:07:20Z

This PR enables different write buffer managers to manager different cfs.

Signed-off-by: Spade A <[email protected]>

Signed-off-by: SpadeA-Tang <[email protected]>

Signed-off-by: Spade A <[email protected]>

Signed-off-by: SpadeA-Tang <[email protected]>

SpadeA-Tang · 2023-08-16T02:31:16Z

/run-test

Signed-off-by: SpadeA-Tang <[email protected]>

SpadeA-Tang · 2023-08-16T09:30:20Z

/run-test

tabokie

This seems way too complex for such a simple functionality. Why would be the problem if we just put a manager in ColumnFamilyOptions? You will need an additional spinlock to protect the vector holding all managers so that we could access them in PreprocessWrite. But I think in total it should be done in less than 50LOC.

SpadeA-Tang · 2023-08-22T02:09:37Z

PTAL /cc @tabokie @Connor1996

Signed-off-by: SpadeA-Tang <[email protected]>

tabokie · 2023-08-24T02:31:13Z

db/column_family.cc

@@ -1589,7 +1590,7 @@ ColumnFamilySet::ColumnFamilySet(const std::string& dbname,
                                 const ImmutableDBOptions* db_options,
                                 const FileOptions& file_options,
                                 Cache* table_cache,
-                                 WriteBufferManager* _write_buffer_manager,
+                                 WriteBufferManager*, /*non-used parameter*/


To avoid changing the tests, we should make it fallback to old behavior if cf_options.wbm is null. In specific, ColumnFamilySet should still accept and store a wbm, but ColumnFamilyData ctor might take a different wbm depending on the specific cf_options.

Good suggestion and done.

tabokie · 2023-08-24T02:31:54Z

db/db_impl/db_impl.cc

+        }
+      }
+      if (!already_exists) {
+        write_buffer_manager_.push_back(cf_options.write_buffer_manager);


You need a spinlock to modify a vector.

tabokie · 2023-08-24T02:32:58Z

db/db_impl/db_impl.h

@@ -2282,7 +2281,7 @@ class DBImpl : public DB {

  Directories directories_;

-  WriteBufferManager* write_buffer_manager_;
+  std::vector<std::shared_ptr<WriteBufferManager>> write_buffer_manager_;


use autovector<Manager*> and put DB level manager separately from CF level managers.

Good suggestion and done.

tabokie · 2023-08-24T02:36:02Z

db/db_impl/db_impl_write.cc

+        status = Status::Incomplete("Write stall");
+      } else {
+        InstrumentedMutexLock l(&mutex_);
+        WriteBufferManagerStallWrites(i);


For simplicity, we don't need to allow stalling from cf level wbm. For default and write cf, we will still use DB level wbm and reuse the old code path.

Good suggestion and done.

Signed-off-by: SpadeA-Tang <[email protected]>

tabokie · 2023-08-24T05:13:18Z

include/rocksdb/options.h

+  // in DBImpl.
+  //
+  // Default: null
+  std::shared_ptr<WriteBufferManager> cf_write_buffer_manager = nullptr;


Suggested change

std::shared_ptr<WriteBufferManager> cf_write_buffer_manager = nullptr;

std::shared_ptr<WriteBufferManager> write_buffer_manager = nullptr;

Option inherited from DBOptions and ColumnFamilyOptions, so I use different name for it to avoid trouble.

tabokie · 2023-08-24T05:15:12Z

db/db_test_util.h

-                            const Options& options);
+  void CreateColumnFamilies(
+      const std::vector<std::string>& cfs, const Options& options,
+      std::unordered_map<std::string, std::shared_ptr<WriteBufferManager>>


This is not the intended use case of this function. If you have different options for each cf, use ReopenWithColumnFamilies(const std::vector<std::string>& cfs, const std::vector<Options>& options); instead.

tabokie · 2023-08-24T05:17:24Z

db/db_impl/db_impl_write.cc

@@ -2282,6 +2287,12 @@ size_t DBImpl::GetWalPreallocateBlockSize(uint64_t write_buffer_size) const {
      bsize = std::min<size_t>(bsize, buffer_size);
    }
  }
+  for (auto manager : cf_based_write_buffer_manager_) {


It should be min(db_write_buffer_size, sum(cf_wbm_size) + db_wbm_size).

tabokie · 2023-08-24T05:18:22Z

db/db_impl/db_impl.h

+  // Note: std::shared_ptr<WriteBufferManager> is store in ColumnfamilyOptions
+  // which is destroyed before DBimpl, so we use
+  // std::shared_ptr<WriteBufferManager> here in the vector.
+  autovector<std::shared_ptr<WriteBufferManager>>


Comment that this is only modified in Open, and lock is not needed.

tabokie · 2023-08-24T05:19:17Z

db/db_impl/db_impl.h

+  // which is destroyed before DBimpl, so we use
+  // std::shared_ptr<WriteBufferManager> here in the vector.
+  autovector<std::shared_ptr<WriteBufferManager>>
+      cf_based_write_buffer_manager_;


If CreateColumnFamily passed in an option not in the list, return error.

tabokie · 2023-08-24T05:34:54Z

db/db_impl/db_impl.h

+  // For simplicity, CF based write buffer manager does not support stall the
+  // write.
+  // Note: std::shared_ptr<WriteBufferManager> is store in ColumnfamilyOptions
+  // which is destroyed before DBimpl, so we use


Have you had any actual error caused by not using shared_ptr? All code using WriteBufferManager should be destroyed before VersionSet, there shouldn't be any problem.

Yes, UnregisterDB is used after VersionSet is destroyed.

Signed-off-by: SpadeA-Tang <[email protected]>

tabokie

Rest LG

tabokie · 2023-08-24T08:15:50Z

db/column_family.cc

+                           options.cf_write_buffer_manager != nullptr
+                               ? options.cf_write_buffer_manager.get()
+                               : write_buffer_manager_,


Moving this block out looks better.

tabokie · 2023-08-24T08:15:58Z

db/db_impl/db_impl.cc

@@ -685,6 +685,9 @@ Status DBImpl::CloseHelper() {
    delete txn_entry.second;
  }

+  for (auto m : cf_based_write_buffer_manager_) {


Comment the ordering.

tabokie · 2023-08-24T08:21:14Z

db/db_impl/db_impl.h

@@ -419,8 +419,7 @@ class DBImpl : public DB {
  virtual Status GetSortedWalFiles(VectorLogPtr& files) override;
  virtual Status GetCurrentWalFile(
      std::unique_ptr<LogFile>* current_log_file) override;
-  virtual Status GetCreationTimeOfOldestFile(
-      uint64_t* creation_time) override;
+  virtual Status GetCreationTimeOfOldestFile(uint64_t* creation_time) override;


Revert this line, need to commit without running clang-format locally.

Signed-off-by: SpadeA-Tang <[email protected]>

tonyxuqqi · 2023-08-25T05:26:34Z

db/db_impl/db_impl_open.cc

@@ -1625,6 +1625,22 @@ Status DBImpl::Open(const DBOptions& db_options, const std::string& dbname,
  }

  DBImpl* impl = new DBImpl(db_options, dbname, seq_per_batch, batch_per_txn);
+  for (auto cf : column_families) {


The logic of this block of code is quite isolated. I think it's better to wrap a private function for it.

I prefer not, adding a member function has both mental overhead and compilation overhead. Plus this block can't be exactly reused elsewhere.

I think the major benefit of a private function is better readability: people can skip the detail when they don't care the very detail of the block. Other benefits include better testability, and more modular code.
Of course adding a comment for the block serves the same purpose in terms of readability, but old-school programmers typically prefer a function and even chat-gpt also recommends a wrap function.

No, it is always a trade-off to extract a sub-routine. Reading function has a significant context switch overhead, and for non-pure function that modifies states it is mostly needed to read the implementation of the function. In RocksDB codebase it is generally preferred to use inlined code rather than many small functions. Some believe it helps with code quality.

tonyxuqqi · 2023-08-25T05:32:45Z

db/db_impl/db_impl_write.cc

@@ -2278,6 +2283,9 @@ size_t DBImpl::GetWalPreallocateBlockSize(uint64_t write_buffer_size) const {
  if (immutable_db_options_.write_buffer_manager) {
    size_t buffer_size =
        immutable_db_options_.write_buffer_manager->flush_size();
+    for (auto manager : cf_based_write_buffer_manager_) {


Why do we need to update buffer_size for WAL here?

If the flush size of write buffer manager is small compared with write buffer size, we will switch memtable before write bytes up to write buffer size. Use min here can avoid wasty pre allocation.

but here you're using += instead of min<> of cf_based_write_buffer_manager_

This is because all CF will write in the same WAL file. Imaging we have 1M write-buffer-limit for CF1 but 1GB write-buffer-limit for CF2.

SpadeA-Tang · 2023-08-29T08:54:00Z

/merge

Signed-off-by: Spade A <[email protected]>

Signed-off-by: Spade A <[email protected]> Signed-off-by: Yang Zhang <[email protected]>

SpadeA-Tang added 12 commits July 19, 2023 13:31

update

5079707

Signed-off-by: Spade A <[email protected]>

update

e3ada2d

Signed-off-by: Spade A <[email protected]>

update

fadfef3

Signed-off-by: Spade A <[email protected]>

update

6ae46ff

Signed-off-by: Spade A <[email protected]>

update wbm

cd5a294

Signed-off-by: SpadeA-Tang <[email protected]>

update

af33283

Signed-off-by: Spade A <[email protected]>

update

ab43653

Signed-off-by: Spade A <[email protected]>

update

b25567e

Signed-off-by: Spade A <[email protected]>

update

033d847

Signed-off-by: Spade A <[email protected]>

update

199d704

Signed-off-by: Spade A <[email protected]>

can pass original wbm tests

c47e7d8

Signed-off-by: SpadeA-Tang <[email protected]>

update

004be9b

Signed-off-by: SpadeA-Tang <[email protected]>

SpadeA-Tang force-pushed the lock-cf-manager branch from cffafd6 to 004be9b Compare August 14, 2023 06:13

SpadeA-Tang added 8 commits August 14, 2023 17:50

add test

30b7862

Signed-off-by: SpadeA-Tang <[email protected]>

format

41775e2

Signed-off-by: SpadeA-Tang <[email protected]>

fix test

0095aa8

Signed-off-by: SpadeA-Tang <[email protected]>

update

964cf81

Signed-off-by: SpadeA-Tang <[email protected]>

update

64fbf24

Signed-off-by: SpadeA-Tang <[email protected]>

fix test

e373067

Signed-off-by: SpadeA-Tang <[email protected]>

update

4ce4cef

Signed-off-by: SpadeA-Tang <[email protected]>

format

e67df36

Signed-off-by: SpadeA-Tang <[email protected]>

SpadeA-Tang added 3 commits August 16, 2023 14:36

update

755c10f

Signed-off-by: SpadeA-Tang <[email protected]>

format

79ac364

Signed-off-by: SpadeA-Tang <[email protected]>

update

62cbf09

Signed-off-by: SpadeA-Tang <[email protected]>

SpadeA-Tang changed the title ~~wip~~ enable multiple write buffer manager Aug 16, 2023

SpadeA-Tang requested review from tabokie and Connor1996 August 16, 2023 07:11

tabokie reviewed Aug 16, 2023

View reviewed changes

update comment

8fbaf5c

Signed-off-by: SpadeA-Tang <[email protected]>

SpadeA-Tang requested review from tonyxuqqi and tabokie August 22, 2023 02:19

SpadeA-Tang added 2 commits August 23, 2023 19:00

update

85d43bf

Signed-off-by: SpadeA-Tang <[email protected]>

fix test

21b9ae5

Signed-off-by: SpadeA-Tang <[email protected]>

SpadeA-Tang force-pushed the lock-cf-manager branch from 057e182 to 21b9ae5 Compare August 24, 2023 02:07

tabokie reviewed Aug 24, 2023

View reviewed changes

SpadeA-Tang added 3 commits August 24, 2023 12:52

update

4f509cf

Signed-off-by: SpadeA-Tang <[email protected]>

format

8ecc649

Signed-off-by: SpadeA-Tang <[email protected]>

update::

947c94b

Signed-off-by: SpadeA-Tang <[email protected]>

tabokie reviewed Aug 24, 2023

View reviewed changes

SpadeA-Tang added 2 commits August 24, 2023 14:29

address comment

ebb40a1

Signed-off-by: SpadeA-Tang <[email protected]>

address comment

d79241f

Signed-off-by: SpadeA-Tang <[email protected]>

tabokie reviewed Aug 24, 2023

View reviewed changes

SpadeA-Tang added 2 commits August 24, 2023 16:27

address comment

5e404af

Signed-off-by: SpadeA-Tang <[email protected]>

Merge branch '6.29.tikv' into lock-cf-manager

35caf14

tonyxuqqi reviewed Aug 25, 2023

View reviewed changes

SpadeA-Tang changed the title ~~enable multiple write buffer manager~~ enable cf uses separete write buffer manager Aug 29, 2023

SpadeA-Tang mentioned this pull request Aug 29, 2023

raftstore-v2: support column family based write buffer manager tikv/tikv#15453

Merged

tabokie approved these changes Aug 29, 2023

View reviewed changes

SpadeA-Tang requested a review from tonyxuqqi August 29, 2023 08:43

tonyxuqqi approved these changes Aug 29, 2023

View reviewed changes

SpadeA-Tang merged commit 6121b2d into tikv:6.29.tikv Aug 29, 2023
2 checks passed

ti-chi-bot mentioned this pull request Aug 29, 2023

raftstore-v2: support column family based write buffer manager (#15453) tikv/tikv#15469

Open

v01dstar pushed a commit to v01dstar/rocksdb that referenced this pull request May 29, 2024

enable cf uses separete write buffer manager (tikv#343)

0af4d9d

Signed-off-by: Spade A <[email protected]>

v01dstar pushed a commit to v01dstar/rocksdb that referenced this pull request Oct 2, 2024

enable cf uses separete write buffer manager (tikv#343)

3042418

Signed-off-by: Spade A <[email protected]> Signed-off-by: Yang Zhang <[email protected]>

v01dstar pushed a commit to v01dstar/rocksdb that referenced this pull request Oct 2, 2024

enable cf uses separete write buffer manager (tikv#343)

de310e6

Signed-off-by: Spade A <[email protected]> Signed-off-by: Yang Zhang <[email protected]>

	std::shared_ptr<WriteBufferManager> cf_write_buffer_manager = nullptr;
	std::shared_ptr<WriteBufferManager> write_buffer_manager = nullptr;

enable cf uses separete write buffer manager #343

enable cf uses separete write buffer manager #343

Conversation

SpadeA-Tang commented Jul 25, 2023 • edited Loading

SpadeA-Tang commented Aug 16, 2023

SpadeA-Tang commented Aug 16, 2023

tabokie left a comment

Choose a reason for hiding this comment

SpadeA-Tang commented Aug 22, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tabokie left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SpadeA-Tang commented Aug 29, 2023

SpadeA-Tang commented Jul 25, 2023 •

edited

Loading

SpadeA-Tang commented Aug 22, 2023 •

edited

Loading