Cluster internal communication over private network only? Issues when using HCloud firewall

[konnextv]

First of all, thanks for all the work that went into this project so far!

I experimented with HCloud clusters using a private network for a few days now. Today I tried setting up a HCloud firewall to block all inbound traffic received on the public IPs (inbound kubernetes API traffic still went through the load balancer just fine).
At first everything was looking good but I started to notice that some kube-system pods stopped working. Additionally, kubectl logs commands timed out and I noticed that these tried to reach the backend by contacting the backend via the public IP.

I would not call myself a k8s expert but isn't internal cluster traffic like that supposed to route via the private IPs only?

Thank you in advance for enlightening me.

[batistein]

Hi @konnextv this is more a question to the config of the ccm and the cni. For example if you configure the private network for native routing you also need to configure the cni different e.g for cilium: https://docs.cilium.io/en/latest/network/concepts/routing/#native-routing

[konnextv]

Thank you for the direction.
Just to confirm I understand correctly: Cilium routes the traffic between nodes via UDP over the public IP addresses per default? I wonder why it would do that when for each node a public and private IP is provided. (e.g. when checking kubectl get nodes)
I will try out native routing though.