Enhancing CoreDNS configuration customisation

[Blackmamba23]

Would be nice if we could add coreDNS corefile settings via the coreDNS CRD object...Right now we have to disable this addon and install coredns separately with custom settings :/

[prometherion]

We started the discussion of this enhancement on the #kamaji channel hosted in the Kubernetes Slack workspace.

I'd say we have two options here:

1. Trying to work with the upstream Kubeadm community to support further customisations
2. Trying to tackle this at the Kamaji level, even tho I'm worried we could drift from upstream, as well as the CRD being too heavy with fields that could be problematic in the long term when scaling to 1,000 clusters which is a basic scale benchmark for us.

I'll try to create some interest in this feature request to revamp the discussion.

[jwitko]

I just like to throw out a third option:
3. Recommended path could be: "If you need advanced configuration of in-cluster DNS you can leverage the node-local-dns pattern. This would allow for full customization of DNS routing and rules, bonus cache layer and iptables conntrack utilization reduction, and we don't have to petition upstream for changes.

Maybe this is a near and mid-term work-around/solution? And petitioning upstream for configurable coredns at provision time is the long-term plan? It sure would be nice to have that.

[prometherion]

Thanks for jumping into the discussion, Jason.

The upstream petitioning could be very cumbersome and a long journey, so your proposed path seems interesting.

I'd like to understand what you're expecting from Kamaji, tho: you would install CoreDNS manually, and then Kamaj would configure the NodeLocalDNS add-on as we're doing with kube-proxy, CoreDNS, and Konnectivity?

[jwitko]

I didn't give it a lot of thought in terms of implementation just was thinking about what could accomplish the goal.
Personally I don't have a huge use-case for this so I'm not speaking to my own needs.

I think your idea is pretty solid but node-local-dns works on top of CoreDNS so I'm not sure why anything with CoreDNS needs to change from what you're doing now? (Referring to your comment about installing manually)
I think just like konnectivity it would make a lot of sense to have an optional additional container for node-local-dns with customizable config through the CRD.
Some challenges I see are:

• Do you support advanced HA configurations? If not node-local-dns becomes a single-point-of-failure per-node. Most people usually accept this but just figured I'd call it out.
• node-local-dns needs to be installed as a daemonset if its used
• It changes the way the kubelet is configured with the DNS flag, so end-users would have to be responsible for doing that