[dnsdist] Unexpected not-caching behavior in 1.5.0-rc3

[Phillip R. Jaenke]

On 7/2/2020 3:42 AM, Remi Gacogne via dnsdist wrote:
> Hi Phillip,
> 
> On 7/1/20 6:57 PM, Phillip R. Jaenke via dnsdist wrote:
>> To test/evaluate performance, one of the tests I run is basically "look
>> up a consistent group of records I know exist, then do it a couple
>> hundred more times or so." Some from the local authoritative, some from
>> the Internet at large. The corresponding dnsdist configuration looks
>> like this:
>> [...]
>> Given this config my expectation would be that lookups of these would
>> get cached on the first hit of the performance run, and then dnsdist
>> would go to cache. But that's not happening - instead it's forwarding
>> every query to the pool and insisting it's a 100% miss rate over tens of
>> thousands of queries where there's actually only about 200 -unique-
>> questions and the responses are consistent.
> 
> Would you be able to describe the queries you are sending and how you
> are sending them? The first thing that comes to mind is that you are
> using a tool which adds EDNS0 Cookies to queries. dnsdist's packet cache
> currently hashes the whole packet, cookies included, and that results in
> a different entry for every cookie value, as described in [1]. We
> planned on addressing this in 1.6.0 [2].
> 
> [1]: https://github.com/PowerDNS/pdns/issues/5131
> [2]: https://github.com/PowerDNS/pdns/pull/8993

EDNS0 was the first thing I eliminated, in fact.. the whole thing looks
literally like this (the lookups changing but you get the concept):

rs=1
while [ $rs -le 100 ]; do
    for xs in a b c d e f g h i j k l m; do
        dig -t A $xs.root-servers.net @${1} +short +nocookie +noedns
        dig -t AAAA $xs.root-servers.net @${1} +short +nocookie +noedns
    done
done

So it does exactly what it says on the tin there. Looks up the same 26
records, 2600 times in total, from the same host (so EDNS shouldn't be
varying regardless, but, eliminate the knobs.) It's purely a cache test.

Your mentioning that it's caching the whole packet had me take a quick
look with tcpdump to see if perhaps there's some shenanigans going on
there and indeed what I'm seeing on the cache test failures is a 100%
rate of bad udp checksums. The dnsdist side is sending literally every
response with a fixed checksum of 0x175f while the query side shows zero
checksum failures.

It almost looks like it is trying to provide at least a partial cache
response given the UDP checksum is absolutely consistent, but, that
checksum is consistent across every single response (so a-m, not just a
or d.) It's very strange.

-Phillip "RootWyrm" Jaenke | prj at rootwyrm.com

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://mailman.powerdns.com/pipermail/dnsdist/attachments/20200702/7cb1e3a2/attachment.sig>