[server] Cdc log visible only when the data already flush to rocksDb from preWriteBuffer

[swuferhong]

Purpose

This pr is aims to fix the visibility of the CDC log, it should only become visible after the KV data has been flushed to RocksDb.

Linked issue: #134

Tests

API and Format

Documentation

[wuchong]

We need to add a test to reproduce the problem. Maybe add a test in ReplicaManagerTest to verify the high-watermark of the given record is must advanced if the given record can be read in kv.

[swuferhong]

@wuchong comments addressed.

[wuchong]

this test can reproduce the problem, because it only has one replia and ack=1, so after executing replicaManager.putRecordsToKv, the watermark has been advanced, and kv data has been flushed, thus the following test is always success without any changes to the source code.

[swuferhong]

Yes, I switched to a multi-thread way to validate this issue. In the new test, if highWatermarkUpdateTimestamp is less than lastTimestampForNullValue even once, it indicates that the CDC log visibility is earlier than the KV. Specifically, highWatermarkUpdateTimestamp is later than the actual highWatermarkUpdate time, and lastTimestampForNullValue is earlier than the collected time, which is intended to eliminate influence of collecting. Pr ready.

[swuferhong]

In my local env, if I revert the change, the test will always failed.

[swuferhong]

@wuchong comments addressed.