Hot write may break the constraint of placement policy

[rleungx]

Enhancement Task

Consider we have two zones: z1 and z2. And store 1 and 2 in z1, store 3, 4, 5, 6 is in z2. We use the following placement policy:

create placement policy apread1 LEADER_CONSTRAINTS="[+zone=z1]" FOLLOWER_CONSTRAINTS='{"+zone=z2": 2}';

On the PD side, it will synchronize two rules one is the leader rule, and the other is the voter rule. The hot write peer scheduling might break the constraint, and create the following operator:

move-hot-write-peer {mv peer: store [1] to [2]} (kind:hot-region,region,leader, region:1(0, 0), createAt:2024-02-23 11:53:35.168782469 +0800 CST m=+0.026292131, startAt:0001-01-01 00:00:00 +0000 UTC, currentStep:0, size:0, steps:[0:{add learner peer 11 on store 2}, 1:{transfer leader from store 1 to store 3}, 2:{use joint consensus, promote learner peer 11 on store 2 to voter, demote voter peer 1 on store 1 to learner}, 3:{leave joint state, promote learner peer 11 on store 2 to voter, demote voter peer 1 on store 1 to learner}, 4:{remove peer on store 1}], timeout:[18m0s])

But if we use another placement policy with evcit-leader attribute

create placement policy apread1 LEADER_CONSTRAINTS="[+zone=z1]" FOLLOWER_CONSTRAINTS='{"+zone=z2, #evict-leader": 2}';

On the PD side, it will synchronize two rules one is the leader rule, and the other is the follower rule, then the operator change to:

move-hot-write-peer {mv peer: store [1] to [2]} (kind:hot-region,region,leader, region:1(0, 0), createAt:2024-02-23 11:57:12.608195316 +0800 CST m=+0.029869347, startAt:0001-01-01 00:00:00 +0000 UTC, currentStep:0, size:0, steps:[0:{add learner peer 11 on store 2}, 1:{use joint consensus, promote learner peer 11 on store 2 to voter, demote voter peer 1 on store 1 to learner}, 2:{transfer leader from store 1 to store 2}, 3:{leave joint state, promote learner peer 11 on store 2 to voter, demote voter peer 1 on store 1 to learner}, 4:{remove peer on store 1}], timeout:[18m0s])

No matter the way we use to config placement policy, the operator should always be the latter one.

[rleungx]

/cc @lhy1024

[seiya-annie]

/found customer