Hosts cannot communicate with any kube services

[kotyara85]

Expected Behavior

curl to services within the cluster should succeed

Current Behavior

E0927 19:50:01.852008       1 available_controller.go:460] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
E0927 19:50:06.856307       1 available_controller.go:460] v1beta1.metrics.k8s.io failed with: failing or missing response from https://192.168.0.25:443/apis/metrics.k8s.io/v1beta1: Get "https://192.168.0.25:443/apis/metrics.k8s.io/v1beta1": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0927 19:50:11.856536       1 available_controller.go:460] v1beta1.metrics.k8s.io failed with: failing or missing response from https://192.168.0.25:443/apis/metrics.k8s.io/v1beta1: Get "https://192.168.0.25:443/apis/metrics.k8s.io/v1beta1": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0927 19:50:16.859525       1 available_controller.go:460] v1beta1.metrics.k8s.io failed with: failing or missing response from https://192.168.0.25:443/apis/metrics.k8s.io/v1beta1: Get "https://192.168.0.25:443/apis/metrics.k8s.io/v1beta1": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0927 19:50:21.523229       1 controller.go:102] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to download v1beta1.metrics.k8s.io: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: error trying to reach service: dial tcp 192.168.0.25:443: i/o timeout

Possible Solution

N/A

Steps to Reproduce (for bugs)

N/A

Context

We use capi on-prem to provision hosts, it seems like communication works fine at first, but it breaks at some point
We are using VXLan overlay
That seems to be happening not on all hosts, but during control plane rotation.
Thanks

Your Environment

• Calico version 3.28.0
• Orchestrator version (e.g. kubernetes, mesos, rkt): k8s 1.30.3
• Operating System and version: centos 9
• Link to your project (optional):