kubeadm's etcd client member add / remove can return errors but server side there could be success

[neolit123]

had a discussion offline with @ahrtr

basically errors like this cannot be trusted because etcd is a distributed system:
https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/util/etcd/etcd.go#L436

what can happen:

• client sends member add request to server
• server returns "context deadline" error to the client. the error could be due to e.g. network blips or slow infra.
• server adds the member, regardless
• client retries to add member due to the error in a poll, member is already there

the solution is to check the member list for the given peer URL before any add (learner or normal) / remove operation.

here:
https://github.com/kubernetes/kubernetes/blob/release-1.31/cmd/kubeadm/app/util/etcd/etcd.go#L430-L431
https://github.com/kubernetes/kubernetes/blob/release-1.31/cmd/kubeadm/app/util/etcd/etcd.go#L361

[neolit123]

cc @pacoxu
this was a surprise, but in rare cases it can happen apparently.

[pacoxu]

We locked the FG to true in kubernetes/kubernetes#126374 this release cycle. The promotion can be reverted if needed.

Should we keep it beta for another 2 or 3 releases?

[SataQiu]

I think it's ok to promote etcd learner mode to GA, but we need to ensure it works well.
And perhaps a cherry-pick fix is needed. WDYT? @neolit123

[neolit123]

+1 to promote to ga and backport fix

[neolit123]

fixed for 1.32 and backported to >= 1.28 thanks to @SataQiu