Direct policy for DNS configuration at the gateway

[maleck13]

Bringing traffic to your gateway is an important part of configuring and setting up your gateway infrastructure. Having a policy that leverages direct policy attachment to target a Gateway and declaratively expresses dns configuration that can be implemented by individual gateway providers or external integrators/policy providers to ensure a choice of implementation and dns providers would be a valuable addition to Gateway API.

Using a policy API, would allow us not to rely on annotations but instead provide a common API across gateway providers that could be extended on in the future to for e.g support targeting a HTTPRoute or add in Overrides and Defaults.

Use Case

I want to be able to define a policy that will decide which DNS provider to use and which routing strategy to use for an entire gateway or a specific listener within a gateway so that I can configure and control how traffic will reach my gateway in the same place as I define my gateway. I don't want this policy to be tied to any specific dns provider or gateway provider as there are many dns providers and so I would like to be able to choose an implementation that supports this policy and the dns providers I want to use.

Proposal

Follow up this discussion, if there is agreement on the value, with a GEP that goes more into the details, but at a high level the policy API would look something like:

apiVersion: gateway.networking.k8s.io/v1alpha1
kind: DNSPolicy
metadata:
  name: prod
  namespace: gateways
spec:
  providerRef:
    name: route53-credentials
    namespace: gateways
    kind: Secret
  targetRef:
    name: prod-external
    group: gateway.networking.k8s.io
    kind: Gateway
    sectionName someListener # if a different policy was needed for a given listener
  routingStrategy: simple # default initial strategy, there maybe others, in Kuadrant we also have LoadBalanced for multi-cluster
  controllerName: some.group.io/dnsController #used to decide which controller should apply the policy

• Policy would be allowed to target a gateway
• A default routingStrategy would be set to simple (other strategies may evolve over time)
• The outcome of this policy targeting a gateway would be an A or CNAME record setup for the listener hosts
• The providerRef secret would configure which zone id and which provider it was for

Additional Reading

We have an existing alpha policy in the kuadrant project that does some of this, it provides more advanced multi-cluster options but here is some more details on the simple strategy within a single cluster:

https://github.com/Kuadrant/architecture/blob/main/rfcs/0005-single-cluster-dnspolicy.md

Multi-Cluster

DNS based load balancing for distributed gateway instances (for Geo or Resilience) is a use case we are also interested in, but wanted to try solve what is likely the most common use case with this initial GEP. In the document linked above, you will see there is also a load balanced strategy, so the hope is that we can extend the strategy options over time.

[kflynn]

This is a pretty interesting idea, thanks! 🙂

I'd love to see a few concrete use-case examples -- actual problems you'd like to solve and what the YAML to solve them might look like. Off the top of my head, e.g. a Gateway that wants a DNS name from Route53 seems an obvious one. Is it even a thing to imagine "just get me a DNS name and trust that my GatewayClass knows how to do that"? And I'd particularly like to see more about what you're thinking about when you mention routing...

[youngnick]

I agree that we definitely need ways to configure external DNS records for a Gateway, but am wary of going too far into designing a Policy before we agree on the "what" and the "why" of the GEP. I'd strongly encourage us to think about the problem in a more user-story-focused way before leaning too heavily into a policy object.

So if this does go to a GEP, I'd like to see:

• discussion of the use cases, as @kflynn mentions
• A roundup of the options (Kuadrant, of course, but also external-dns's support, plus any other DNS config options that we can find)

After we've done that, and agreed on the details there, then we move on to talking about Policy, and start with:

• Discussion of why a policy is the best way to do this (is it better to have a way to configure DNS issuers, and then use some simpler method to select which Gateways should be actuated by which issuers? I'm thinking of something more like cert-manager here, I think, but I think that we do need to justify the complexity of introducing a separate object to some extent).

For some other feedback, the controllerName field is particularly interesting on this Policy object - this would be the first time we've added a Policy object that's not reconciled by the same implementation that the rest of the objects are reconciled by. So this is a pretty big change in that respect, that needs some time thinking about if that's in Gateway API's scope or not - but we can't really think too usefully about that unless we're confident that we understand exactly what it is that people actually need.

To be clear, I think that this problem probably is useful to solve at the Gateway API spec level, and that a Policy is an interesting way to do it. But past, very painful experience has taught us that going a bit slower is more likely to produce a result that we can actually move forward with confidently.

[candita]

I agree that this is a good idea, no doubt lots of implementors would wish to provide and manage DNS resolution for their users. In fact, some may already tie in to external-dns or other solutions, and some may be interested in what Kuadrant has to offer. It would be worth it to query the slack channel to find out how implementors are doing this without a DNSPolicy at the moment.

I noticed that external-dns had a recent update for Gateway API's GA v1.0: kubernetes-sigs/external-dns#4019, and I'm pretty sure Kuadrant is already up to date with v1.0?

I would venture a guess that most clusters would be fine with a single DNS provider and defining how to expose the provider to Gateways and xRoutes is a good and necessary use case. Use cases for how to indicate A or CNAME, how to handle wildcard hostnames, and comparison on how it would work for Kuadrant vs external-dns would be useful.

Even though DNS-based load-balancing could be a great feature, I don't think anything other than the simple routingStrategy should be approached in the first iteration, so a routingStrategy field could be omitted rather than serving as a distraction. In other words, do not include the geo-weighted strategy mentioned in https://github.com/Kuadrant/architecture/blob/main/rfcs/0005-single-cluster-dnspolicy.md.

[maleck13]

Thanks @candita @kflynn @youngnick for your responses. So my take away here is there is interest in this, but I need to put more focus and thought into the What and Why rather than jumping straight to policy attachment as a solution . Also I need to identify how you could solve this problem now without any specific API resource in gateway API and then elaborate why this problem makes sense to be solved as part of Gateway API.
I can take a stab at updating the original discussion or I can open a draft GEP. I am wondering if a draft GEP focused on these things might be better than here in a discussion?

[shaneutt]

You've already followed our official policy on the GEP process by starting with a discussion to build some consensus (this is step 1). The next step is create an issue (which we'll use to track the progress of the GEP in our GEP Tracking Project Board) and then "Agree on the goals" where you create the initial Provisional GEP.

Given the interest already seen and the discussion so far, now seems like a fine time to start the GEP process to see if this is something we can make tenable within the API. Thank you, and let us know how we can support you in this!

[maleck13]

added #2627