Replication Lag Consistently Increased During Sysbench OLTP_WRITE_ONLY Testing

[VWagen1989]

Test environment:

Hosts: Volcano Engine ecs.g3i.xlarge 4C16GB x 2

Test Tool: Sysbench with workload setting
- type: OLTP_WRITE_ONLY
- threads: 40
- table-count: 10
- table-size: 2,000,000

MySQL Configuration:
innodb_buffer_pool_size=11297935
innodb_log_file_size=536870912
innodb_flush_log_at_trx_commit=2
innodb_flush_method=O_DIRECT

This picture depicts that the replication lag continuously increased as the sysbench testing performing.

After initial troubleshooting, I identified that the issue stems from the slow write performance of DuckDB. Additionally, the current implementation in MyDuck Server flushes the data stored in the delta buffer sequentially for each table (as introduced in #93), which further exacerbates the performance bottleneck during data persistence.

To address this, I propose adopting a concurrent flushing mechanism for delta buffers across multiple tables. This approach would reduce the time required for data persistence compared to the current sequential method. However, the underlying problem remains DuckDB’s poor write performance. After implementing the concurrent flushing improvement, we should investigate and resolve the root cause of DuckDB’s slow data persistence.