feat(stdlib): add dns_lookup function

[esensar]

Adds dns_lookup function, relying on the domain crate (dns_lookup crate doesn't support any options).

Fixes: #720

[esensar]

I have tried implementing this with dns_lookup crate first (which is already included), but it was very limited in its options, so I had used domain crate here. domain crate pulls in many new dependencies, so I am not sure if that is acceptable to the project. If not, I can look for other options, or go back to dns_lookup, for a more limited functionality.

hickory_client would be a good alternative, since some parts of hickory are already used in vector.

[jszwedko]

@johnhtodd do you mind taking a look at this one to make sure the interface matches what you expected?

[jszwedko]

domain does pull in quite a few new dependencies including Tokio 😅 @esensar if you are open to it, I wouldn't mind evaluating the other options more thoroughly to see if there is one that is a bit lighter.

[esensar]

domain does pull in quite a few new dependencies including Tokio 😅 @esensar if you are open to it, I wouldn't mind evaluating the other options more thoroughly to see if there is one that is a bit lighter.

Yeah, definitely. Seeing tokio made me reconsider it :D

[johnhtodd]

I admit I'm not able to follow the threads of functionality features all the way out to the individual crate options. However, the comment that "dns_lookup" supports very few options means that really can't be useful in the context that most users of this would want. The ability to specify target servers, dnssec yes/no, quantity of attempts, timeouts, or alternative DNS formats (tcp, doh, dot) make this not viable. I would be fine with hickory_client, and I think it could even be "dumbed down" to not do internal DNSSEC validation and just be a forwarding proxy. I do not think that vector should take on the tasks of DNS resolution, even in a crate add-on, so my interests are in how the functionality can be minimized in Vector and outsourced to an upstream persistent DNS resolver.

Does hickory_crate or dns_lookup provide these minimal specs:

Required in query:

• qname (mandatory)
• QTYPE (mandatory)
• namesever destination (optional - if not specified, system resolver will be used)
• class (optional, default IN)
• format of request (tcp, doh, dot) - I am not 100% firm on this one, but it just barely makes the "required" list because of my "no unencrypted data off-host" religion

Required supported additional flags/options in query:

• dnssec enabled/disabled
• number of tries and timeout per try (or some other way to limit wait time per query)
• NSID request
• recurse/no recurse
• EDNS client subnet data

Mandatory response items:

• 'standard' DNS response data and flags (see existing DNSTAP parsing code)
• EDNS EDE
• NSID

Everything else is optional, as far as my own (selfish) interests when looking at the original list of options I noted in #720

[esensar]

I think almost all of these are supported by domain crate (and probably the hickory_client too), but the issue here is that I haven't been able to find a crate for DNS lookup that doesn't depend on tokio...

If using tokio is fine in the end, then I would actually prefer hickory_client instead, since hickory_proto is used in vector too.

[jszwedko]

I think almost all of these are supported by domain crate (and probably the hickory_client too), but the issue here is that I haven't been able to find a crate for DNS lookup that doesn't depend on tokio...

If using tokio is fine in the end, then I would actually prefer hickory_client instead, since hickory_proto is used in vector too.

Thanks for researching @esensar . I guess it's not too surprising given they would need to issue asynchronous requests. If hickory_client could work to satisfy the needs, I agree that seems preferable since we already have a dependency on hickory_proto and so it'd be less of a net-new dependency.

[esensar]

I think almost all of these are supported by domain crate (and probably the hickory_client too), but the issue here is that I haven't been able to find a crate for DNS lookup that doesn't depend on tokio...
If using tokio is fine in the end, then I would actually prefer hickory_client instead, since hickory_proto is used in vector too.

Thanks for researching @esensar . I guess it's not too surprising given they would need to issue asynchronous requests. If hickory_client could work to satisfy the needs, I agree that seems preferable since we already have a dependency on hickory_proto and so it'd be less of a net-new dependency.

I have started moving this to hickory_client. It is mostly done, other than adding support for some of the options (it will not be as straightforward as with domain).

hickory_client does not parse /etc/resolv.conf by default, while domain did it. Should something like that be supported or should we make nameserver required, because domain was defaulting to whatever was defined in /etc/resolv.conf?

[johnhtodd]

hickory_client does not parse /etc/resolv.conf by default, while domain did it. Should something like that be supported or should we make nameserver required, because domain was defaulting to whatever was defined in /etc/resolv.conf?

I don't think it is a problem to always specify a resolver in the command, so if that is the solution then I'd say that's reasonable. Using /etc/resolv.conf would have been convenient as a default, but that is not a requirement.

[esensar]

hickory_client does not parse /etc/resolv.conf by default, while domain did it. Should something like that be supported or should we make nameserver required, because domain was defaulting to whatever was defined in /etc/resolv.conf?

I don't think it is a problem to always specify a resolver in the command, so if that is the solution then I'd say that's reasonable. Using /etc/resolv.conf would have been convenient as a default, but that is not a requirement.

Okay, maybe we can start with required parameter for simplicity.

[jszwedko]

I converted this back to draft since it seems like you are still iterating on it, but feel free to mark it ready when it is ready for review. Thanks for your work on this one!

[esensar]

Sorry for once again bringing up the crate concern here, but after spending some time trying to get hickory_client to work here, I am not sure if it will be beneficial in the end. To properly support all required options, this requires adding some more dependencies explicitly and generally more complex and harder to maintain function.

On the other hand domain crate can't support DoH and DoT, whereas it can be done with hickory_client (although it complicates things a bit, it is doable).

Not sure how to progress on this, I am worried about making this function unnecessarily complex and hard to maintain. If DoH and DoT are not critical to be included, I believe going the domain route would be safer, even if it resulted in more new dependencies when compared to hickory_client.

[pront]

I believe going the domain route would be safer, even if it resulted in more new dependencies when compared to hickory_client.

I agree, domain looks more mature for this use case.

[johnhtodd]

Sorry for once again bringing up the crate concern here, but after spending some time trying to get hickory_client to work here, I am not sure if it will be beneficial in the end. To properly support all required options, this requires adding some more dependencies explicitly and generally more complex and harder to maintain function.

On the other hand domain crate can't support DoH and DoT, whereas it can be done with hickory_client (although it complicates things a bit, it is doable).

Not sure how to progress on this, I am worried about making this function unnecessarily complex and hard to maintain. If DoH and DoT are not critical to be included, I believe going the domain route would be safer, even if it resulted in more new dependencies when compared to hickory_client.

DoT and DoH are not absolutely required, so if domain doesn't support those it is not a critical requirement from my perspective. Simpler = better as long as the core functionality and the mandatory flags/functions are available.

[esensar]

Unfortunately, I don't see a way to control dnssec in domain, or adding EDNS client subnet. EDNS EDE and NSID should be in OPT record which is returned by this function. Other than that, I think other required options are covered.

[johnhtodd]

Unfortunately, I don't see a way to control dnssec in domain, or adding EDNS client subnet. EDNS EDE and NSID should be in OPT record which is returned by this function. Other than that, I think other required options are covered.

DNSSEC is requested by explicitly setting/unsetting the "DO" bit on the request.

NSID has to be requested explicitly on the outbound connection in order to appear in the reply, though EDNS EDE is returned if EDNS is flagged as being supported in the outbound query (which most modern DNS stacks should do, I assume.)

[esensar]

Unfortunately, I don't see a way to control dnssec in domain, or adding EDNS client subnet. EDNS EDE and NSID should be in OPT record which is returned by this function. Other than that, I think other required options are covered.

DNSSEC is requested by explicitly setting/unsetting the "DO" bit on the request.

NSID has to be requested explicitly on the outbound connection in order to appear in the reply, though EDNS EDE is returned if EDNS is flagged as being supported in the outbound query (which most modern DNS stacks should do, I assume.)

I don't see an option to contol the do bit in domain unfortunately. Same for NSID too.

[pront]

Perhaps we should put it behind a feature flag, at least until we introduce some function result caching mechanism.

[esensar]

That makes sense, although doesn't reverse_dns also include network calls? Should that one be put behind a network flag too, or does that one utilize some kind of caching?

[pront]

That one predates me 😅 But point taken, let's just document this appropriately.

[esensar]

Sorry, just checking if I understood correctly. Based on discussion from: #720 I thought this function was supposed to be undocumented (at least until caching is sorted out), similar to reverse_dns.

I can add a warning to make it more visible that it involves network calls and should be avoided in most cases if we want to have it documented in the end though.

[pront]

for the above question cc @jszwedko

[jszwedko]

👍 I'm still fine with adding this but not documenting it, for the time being, until we can sort out a caching strategy.

[pront]

Code looks good. One thing I noticed is that the tests are not checking all the possible fields. It would be good to strengthen them.

[esensar]

Code looks good. One thing I noticed is that the tests are not checking all the possible fields. It would be good to strengthen them.

That makes sense, I will expand them.

[pront]

@esensar is this now in final state? If so, maybe @jszwedko wants to take a final look and then we can merge 🚢

[esensar]

@esensar is this now in final state? If so, maybe @jszwedko wants to take a final look and then we can merge 🚢

Yeah, I think it is. I didn't plan on adding more tests, unless you think more things should be covered.

[pront]

@esensar is this now in final state? If so, maybe @jszwedko wants to take a final look and then we can merge 🚢

Yeah, I think it is. I didn't plan on adding more tests, unless you think more things should be covered.

Test coverage is good now, but I see a few test failures. You can run the CI locally with ./scripts/checks.sh.

[esensar]

@esensar is this now in final state? If so, maybe @jszwedko wants to take a final look and then we can merge 🚢

Yeah, I think it is. I didn't plan on adding more tests, unless you think more things should be covered.

Test coverage is good now, but I see a few test failures. You can run the CI locally with ./scripts/checks.sh.

Yeah, I was just about to write about that. I will have to fix that, I don't think it will take long. It shouldn't interfere with review though, if you want to take a final look.

[jszwedko]

Nice work @esensar