[dnsdist] Tune DNSDIST for proper traffic diversion and caching for lower latency

[Remi Gacogne]

Hi Chandra,

On 10/12/2021 14:27, Chandra via dnsdist wrote:
> For #1: I didn't find a proper server policy to fit my needs but, it 
> doesn't seem to be a completely new thing to have. Currently the 
> weighted random policy does work to some extent. But there are some 
> queries which goto the fallback servers, for eg out of 30k queries at 
> least 50 of them goto the fallback servers, I do not want this. Is there 
> a way to achieve what I described in #1?

I would suggest using two different pools of servers and the 
PoolAvailableRule rule [1]. That way queries will be always routed to 
the first pool if it has at least one server available, and to the 
second one otherwise.
If you don't set any cache on the second pool the answers coming from 
these servers won't be cached. The only part of your requirements that I 
don't see how to address is the "no stale cache" part, since we will 
only check for stale entries in the cache associated to the pool that 
was selected, and therefore will not check the cache for the first pool 
when its servers are down.

[1]: https://dnsdist.org/rules-actions.html#PoolAvailableRule

> For #2:  This is the most concerning issue for me at the moment, the 
> average latency is about 80 ms (10k packet average), where as my primary 
> server latency is much lower (~50ms) and most confusing part is the 
> packet cache stats:
> 
> Entries: 86/10000
> Hits: 4894
> Misses: 21543
> Deferred inserts: 0
> Deferred lookups: 0
> Lookup Collisions: 0
> Insert Collisions: 0
> TTL Too Shorts: 0
> 
> I was under the impression that if there's a cache miss then the 
> downstream response will be cached. Testing my setup for a couple of 
> days, I have never seen my cache crossing 100.  Why is the response not 
> being cached, where there's a miss. Here are the current  extended stats:

I'm afraid it's hard to know without knowing exactly which answers are 
not cached. There are some answers that dnsdist will refuse to cache:
- when the minimum TTL is lower than minTTL, but since minTTL is set to 
0 in your configuration that should not be it ;
- when there is no records at all in the answer and the rcode is 
different from ServFail or Refused, since we don't know for how long to 
keep that entry.

I would suggest having a look at 'grepq("")' to try to understand which 
answers are not cached at the next step.

> from what I see, there are a lot of udp errors. How to fix this? Also to 
> add: all my traffic is udp based, I am not accepting TCP traffic yet.

Please note that some values are system-wide counters and might not be 
related to dnsdist at all, like udp-* and udp6-*.

"noport" errors cannot be fixed on the server side, they usually mean 
that the remote end closed its socket before receiving the response. 
"in-errors" might be caused by dnsdist not accepting UDP queries fast 
enough, which can be fixed by tuning your configuration [2] [3].

[2]: 
https://dnsdist.org/reference/tuning.html?highlight=recv#setUDPSocketBufferSize
[3]: 
https://dnsdist.org/advanced/tuning.html#udp-and-incoming-dns-over-https

Best,
-- 
Remi Gacogne
PowerDNS.COM BV - https://www.powerdns.com/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://mailman.powerdns.com/pipermail/dnsdist/attachments/20211213/177019a8/attachment.sig>