Questions about the high availability mechanism of sidecar-syncer

[fengbao123]

Hey Team,
I read an article about Sealos that introduced how KubeBlocks handles classic primary-replica scenarios, such as MySQL and PostgreSQL. KubeBlocks is responsible for tasks like detection, decision-making, failover, rebuilding, and role correction. However, I have the following questions regarding some of its mechanisms:

1. If the server hosting POD-0 fails and the entire POD-0 becomes unavailable, who initiates the unlock task in ETCD?
2. When other pods lock in ETCD, what are the rules? Is there communication between different pods? Is it first-come, first-served? Is it based on the highest GTID? How is data consistency ensured?
3. Setting POD-0 to "Demote Readonly": I understand this as a fencing mechanism. What happens if fencing fails and POD-0 recovers before RoleLabelling?
4. What is the entire process of rebuilding the failed master node POD-0? Will there be a rewind?

Currently, there is a lack of detailed documentation on high availability mechanisms for MySQL in such scenarios. As a result, it is unclear whether its data consistency and high availability mechanisms meet production requirements.

[xuriwuyun]

Good questions and thank you for your attention to this matter!

Yes, the high availability in primary-replica scenarios need to take care of numerous details and handle carefully. While some information about Syncer in that article is outdated, it now runs as a side-process alongside the db, This enables Syncer to effectively manage database state, processes, and data.

In DB clusters with Syncer enabled, each pod has a heartbeat detection. If the primary pod becomes unavailable, other replicas detect this change and initiate an election process to get a new primary.

The primary election is complex and involves more than just simple lock acquisition. It must evaluate:

• Overall cluster health
• Individual pod status
• Ability to perform switchover/failover
• Identification of the most suitable member

During a switchover, if the primary fails to demote properly, the process pauses. It then waits for the primary to become ready for switchover until either success is achieved or manual cancellation occurs. After a successful switchover/failover, the former primary automatically recovers as a secondary node.

Syncer has been widely adopted across various use cases. Please contact us if you have any interest!