[dnsdist] dnsdist Drops, revisited

[Remi Gacogne]

Hi,

On 3/6/20 8:09 AM, Fredrik Pettai via dnsdist wrote:
>> On 6 Mar 2020, at 05:42, Michael Van Der Beek <michael.van at antlabs.com> wrote:
>> Have you noticed this setting on dnsdist.
>> setUDPTimeout(num)
> 
> Yes, I did, but I didn’t play around with that before I sent the email to the mailing list
> 
>> Set the maximum time dnsdist will wait for a response from a backend over UDP, in seconds. Defaults to 2
>> I'm not sure if timeouts are classified as drops. My guess probably, because it didn't get a response in time.
> 
> Yes they are.

"Drops", as reported by dnsdist, are almost always cause by the backend
not responding fast enough. On some setups, dealing with 100k+ qps, it
might also be caused by dnsdist not processing the responses fast
enough, but that's very easy to spot because at least one of the dnsdist
threads will use ~100% of one core.

>> Since your backend is a recursor. There are times that the recursor cannot reach or encounters a non-responsive authoritative server.  Unbound has an exponential backoff when querying such servers. I think it starts with 10s.
>> https://nlnetlabs.nl/documentation/unbound/info-timeout/
>>
>> I would suggest you set the dnsdist setUDPTImeout(10), frankly, if Unbound cannot respond to you in < 10 seconds, most likely the target authoritative server is not responding.
> 
> Good point, while I didn’t turn to the unbound documentation (thanks for the pointer) I played around with the UDPTimeout setting yesterday, 
> first increasing to setUDPTImeout(5), which yielded better results in terms of Drops (and increased the latency) and then later to 15, just to be sure that unbound really should be done with queries, and noticed that the Drops became a lot less (and latency increase again). But as you suggest, setUDPTImeout(10) is probably the ultimate setting.  

OK so that settles it, your backends are not responding fast enough to
some queries. I would really advise you to try to understand why the
backend is taking so long to respond, instead of tuning dnsdist via
setUDPTImeout(), because a latency greater than 2s is going to cause a
lot of issues anyway.

Best regards,
-- 
Remi Gacogne
PowerDNS.COM BV - https://www.powerdns.com/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://mailman.powerdns.com/pipermail/dnsdist/attachments/20200306/18534e77/attachment.sig>