Internal DNS doesn't work on a newly added node

[user-name-is-taken]

After adding a third worker node with kubicctl node add <node name> I can't reach services running inside the cluster from pods running on the new node. To fix this I deleted the new node, so unfortunately I can't provide logs. If you can't reproduce this issue feel free to close it.

[0pendev]

Deployed a working kubic cluster today and I did not have the issue. Can you provide more info ?
Release of kubic, CNI in use, types of services etc

[thkukuk]

Sure that you used the defaults and not flannel? This is something we see regular with flannel, but never with weave.

[user-name-is-taken]

$ kubectl get daemonset -n kube-system
NAME         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
kube-proxy   3         3         3       3            3           kubernetes.io/os=linux   151d
kured        3         3         0       3            0           <none>                   151d
weave-net    3         3         3       3            3           <none>                   151d

# kubicctl version
kubicctl version 0.10.2

$ kubectl get nodes -o wide
NAME             STATUS   ROLES    AGE    VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE           KERNEL-VERSION     CONTAINER-RUNTIME
install          Ready    master   151d   v1.18.9   x.x.x.x   <none>        openSUSE MicroOS   5.8.10-1-default   cri-o://1.19.0
localhost        Ready    <none>   151d   v1.18.9   x.x.x.x    <none>        openSUSE MicroOS   5.8.10-1-default   cri-o://1.19.0
updated-node-2   Ready    <none>   151d   v1.18.9   x.x.x.x    <none>        openSUSE MicroOS   5.8.10-1-default   cri-o://1.19.0

$ kubectl get ns
NAME                      STATUS   AGE
default                   Active   151d
grafana                   Active   137d
harbor                    Active   63d
imagepullsecret-patcher   Active   96d
keycloak                  Active   149d
kube-node-lease           Active   151d
kube-public               Active   151d
kube-system               Active   151d
linkerd                   Active   63d
loki                      Active   83d
metallb-system            Active   151d
rook-ceph                 Active   87d
traefik-system            Active   150d
vault                     Active   52d

[0pendev]

And for example, what is the service that you can't access here ?

[user-name-is-taken]

If I added a node to that cluster I wouldn't be able to ping any pods running on the new node or ping anything from the pods running on the new nodes.