ticdc v5.1.2

Compatibility changes

• Set compatible version from 5.1.0-alpha to 5.2.0-alpha #2659
• Prohibit operating TiCDC clusters across major and minor versions #2599
• Fix the CLI compatibility issue with 4.0.x clusters on the default sort-engine option #2373

Improvements

• Optimize memory management when the Unified Sorter is using memory to sort #2553
• Optimize workerpool for fewer goroutines when concurrency is high #2211
• Reduce goroutine usage when a table's region transfer away from a TiKV node #2284

Bug Fixes

• Fix a bug that json encoding may cause panic when processing a string type value that is string or []byte. #2758
• Reduce gRPC window size to avoid OOM #2202
• Fix gRPC keepalive error under high memory pressure. #2202
• Fix a bug that an unsigned tinyint causes TiCDC to panic. #2648
• Fix empty value issue in open protocol. An empty value is no longer output when there is no change in one transaction. #2612
• Fix a bug in DDL handling during manual restarts #2603
• Fix the issue that EtcdWorker's snapshot isolation might be wrongly violated when managing the metadata #2559
• Fix the issue that multiple processors could write in the same table when re-scheduling this table #2230
• Fix the issue of the ErrSchemaStorageTableMiss error and that a changefeed is reset by accident. #2422
• Fix the issue that changefeed cannot be removed when the GcTTL Exceeded Error occors #2391
• Fix the issue that replicating large tables to cdclog fails #1259#2424

ticdc v4.0.15

Improvements

• TiCDC always pulls old values from TiKV internally #2304
• Reduce the goroutine usage when a table's Regions are all transferred away from a TiKV node #2284
• Optimize workerpool for fewer goroutines when concurrency is high #2211
• Execute DDL statements asynchronously to avoid affecting other changefeeds #2471
• Add a global gRPC connection pool and share gRPC connections among KV clients #2531
• Fail fast for unrecoverable DML errors 2315
• Optimize memory management when unified sorter is using memory to sort #2553
• Add Prometheus metrics for DDL executions #2681
• Prohibit operating TiCDC clusters across major or minor versions #2601
• Remove file sorter #2325
• Clean up changefeed metrics when a changefeed is removed, and clean up processor metrics when a processor exits #2313
• Optimize the lock-resolving algorithm after a Region is initialized #2264

Bug fixes

• Fix the ErrSchemaStorageTableMiss error that occurs when the DDL Job duplication is encountered in the integrated test #2422
• Fix the bug that a changefeed cannot be removed if the ErrGCTTLExceeded error occurs #2391
• Fix the issue that outdated capture might appear in the output of the capture list command #2388
• Fix the deadlock issue in the TiCDC processor #2017
• Fix a data inconsistency issue that occurs because multiple processors might write data to the same table when this table is being re-scheduled #2495#2727
• Fix a bug that the EtcdWorker snapshot isolation is violated in metadata management #2557
• Fix the issue that the changefeed cannot be stopped due to the DDL sink error #2556
• Fix the issue of TiCDC Open Protocol: TiCDC outputs an empty value when there is no change in a transaction #2619
• Fix a bug that causes TiCDC to panic on the unsigned TINYINT type #2648
• Decrease the gRPC window size to avoid the OOM that occurs when TiCDC captures too many Regions #2202
• Fix the OOM issue that occurs when TiCDC captures too many Regions #2723
• Fix the issue of process panic that occurs when encoding the data types such as mysql.TypeString, mysql.TypeVarString, mysql.TypeVarchar into JSON #2758
• Fix the a memory leak issue that might occur when creating a new changefeed #2623
• Fix a bug that DDL handling fails when a changefeed starts at the finish TS of a schema change #2603
• Fix the issue of potential DDL loss when the owner crashes when executing DDL statements #1260
• Fix the issue of insecure concurrent access to the map in SinkManager #2298

ticdc v5.0.5

There is no release note for this version.

ticdc v5.0.4

Feature Enhancements

• Add an HTTP API to get the changefeed information and the health information of the node #1955
• Add the SASL/SCRAM support for the kafka sink #1942
• Make TiCDC support --data-dir at the server level #2070

Improvements

• Refine gRPC's reconnection logic and increase the KV client's throughput #1586 #1501 #1682 #1393 #1847 #1905 #1904
• Make the sorter I/O errors more user-friendly

Bug Fixes

• Fix the issue that TiCDC owner exits when refreshing the checkpoint #1902
• Fix a bug that some MySQL connection might leak after MySQL sink meets the error and pauses #1946
• Fix the panic issue that occurs when TiCDC fails to read /proc/meminfo #2024
• Reduce TiCDC's runtime memory consumption #2012 #1958
• Fix a bug that might cause TiCDC server panic due to the late calculation of resolved ts #1576
• Fix the potential deadlock issue for the processor #2142

ticdc v5.2.1

owner: fix gc safepoint larger by one (#2647)

ticdc v5.2.0

Feature Enhancements

• Add the binary MQ format designed for TiDB. It is more compact than the open protocols based on JSON #1621
• Remove support for file sorter #2114
• Support log rotation configurations #2182

Improvements

• Add the concurrency limit to the incremental scan of kv client #1899
• TiCDC can always pull the old value internally #2271
• TiCDC can fail and exit fast when unrecoverable DML errors occur #1928
• resolve lock cannot be run immediately after a Region is initialized #2235
• Optimize workerpool to reduce the number of goroutines under high concurrency #2201

Bug Fixes

• Fix a bug that TiCDC owner exits abnormally when refreshing the checkpoint #1902
• Fix a bug that changefeed fails immediately after its successful creation #2113
• Fix a bug that changefeed fails due to the invalid format of rules filter #1625
• Fix the potential DDL loss issue when the TiCDC owner panics #1260
• Fix the CLI compatibility issue with 4.0.x clusters on the default sort-engine option #2373
• Fix a bug that changefeed might be reset unexpectedly when TiCDC gets the ErrSchemaStorageTableMiss error #2422
• Fix a bug that changefeed cannot be removed when TiCDC gets the ErrGCTTLExceeded error #2391
• Fix a bug that TiCDC fails to synchronize large tables to cdclog #1259 #2424
• Fix a bug that multiple processors might write data to the same table when TiCDC is rescheduling the table #2230

ticdc v5.1.1

Improvements

• Remove file sorter #2327
• Improve the error message returned when a PD endpoint misses the certificate #1973

Bug fixes

• Fix the support for the new collation feature #2301
• Fix the issue that an unsynchronized access to a shared map at runtime might cause panic #2300
• Fix the potential DDL loss issue that occurs when the owner crashes while executing the DDL statement #2290
• Fix the issue of trying to resolve locks in TiDB prematurely #2188
• Fix a bug that might cause data loss if a TiCDC node is killed immediately after a table migration #2033
• Fix the handling logic of changefeed update on --sort-dir and --start-ts #1921

ticdc v4.0.14

Improvements

• Improve the error message returned when a PD endpoint misses the certificate #2184
• Make the sorter I/O errors more user-friendly #1976
• Add a concurrency limit on the Region incremental scan in the KV client to reduce the pressure of TiKV #1926
• Add metrics for the table memory consumption #1884
• Add capture-session-ttl to the TiCDC server configuration #2169

Bug fixes

• Fix the issue of extra partition dispatching after adding new table partitions #2205
• Fix the panic issue that occurs when TiCDC fails to read /proc/meminfo #2023
• Reduce TiCDC's runtime memory consumption #2011 #1957
• Fix a bug that some MySQL connection might leak after MySQL sink meets the error and pauses #1945
• Fix the issue that TiCDC changefeed cannot be created when start TS is less than current TS minus GC TTL #1839
• Reduce memory malloc in sort heap to avoid too much CPU overhead #1853
• Fix a bug that the replication task might stop when moving a table #1827

ticdc v5.0.3

Feature Enhancements

• Add an HTTP API to get the changefeed information and the health information of the node #1955
• Add the SASL/SCRAM support for the kafka sink #1942
• Make TiCDC support --data-dir at the server level #2070

Improvements

• Refine gRPC's reconnection logic and increase the KV client's throughput #1586 #1501 #1682 #1393 #1847 #1905 #1904
• Make the sorter I/O errors more user-friendly

Bug Fixes

• Fix the issue that TiCDC owner exits when refreshing the checkpoint #1902
• Fix a bug that some MySQL connection might leak after MySQL sink meets the error and pauses #1946
• Fix the panic issue that occurs when TiCDC fails to read /proc/meminfo #2024
• Reduce TiCDC's runtime memory consumption #2012 #1958
• Fix a bug that might cause TiCDC server panic due to the late calculation of resolved ts #1576
• Fix the potential deadlock issue for the processor #2142

ticdc v5.1.0

Improvements

• Improve the descriptions of some log messages to be clearer and more useful for diagnosing problems #1759
• Support the back pressure feature to allow the TiCDC scanning speed to sense the downstream processing capacity #10151
• Reduce memory usage when TiCDC performs the initial scan #10133
• Improve the cache hit rate for the TiCDC Old Value in pessimistic transactions #10089

Bug Fixes

• Fix the concurrency issue in Unified Sorter and filter the unhelpful error messages #1678
• Fix a bug that the creation of redundant directories might interrupt the replication with MinIO #1463
• Set the default value of the explicit_defaults_for_timestamp session variable to ON to make the MySQL 5.7 downstream keep the same behavior with the upstream TiDB #1585
• Fix the issue that the incorrect handling of io.EOF might cause replication interruption #1633
• Correct the TiKV CDC endpoint CPU metric in the TiCDC dashboard #1645
• Increase defaultBufferChanSize to avoid replication blocking in some cases #1259
• Fix the issue that the time zone information is lost in the Avro output #1712
• Support cleaning up stale temporary files in Unified Sorter and forbid sharing the sort-dir directory #1742
• Fix a deadlock bug in the KV client that occurs when many stale Regions exist #1599
• Fix the wrong help information in the --cert-allowed-cn flag #1697
• Revert the update for explicit_defaults_for_timestamp which requires the SUPER privilege when replicating data to MySQL #1750
• Support the sink flow control to reduce the risk of memory overflow #1840
• Fix a bug that the replication task might stop when moving a table #1828
• Fix the issue that the TiKV GC safe point is blocked due to the stagnation of TiCDC changefeed checkpoint #1759