[Pdns-users] dnsdist marking downstream servers down: tuning issue?

[Winfried Angele]

Maybe there is some kind of ratelimt in the backend. Keep in mind that from the backend point of view all queries come from the same source

Winfried

Am 24. Juli 2021 10:28:49 MESZ schrieb Winfried Angele via Pdns-users <pdns-users at mailman.powerdns.com>:
>Right. Therefore I would recommend rather to troubleshoot the backend
>
>Winfried
>
>
>Am 23. Juli 2021 20:56:55 MESZ schrieb Dez C via Pdns-users
><pdns-users at mailman.powerdns.com>:
>>Hi,
>>
>>On 7/23/21 4:14 PM, Winfried Angele via Pdns-users wrote:
>>> Does this only happen with DoH frontends? Did you try with UDP 
>>> frontends as well? Sounds like a bottleneck on your backends imo.
>>
>>I'm only using dnsdist for DoH so I haven't tried with UDP/53. I doubt
>
>>it'll make a difference because the problem manifests itself when 
>>dnsdist is trying to query the backends (eventhough most replies
>>already 
>>exist in dnsdist's cache)
>>
>>cheers
>>
>>>
>>> Winfried
>>>
>>>
>>> Am 23. Juli 2021 13:32:39 MESZ schrieb Yannis via Pdns-users 
>>> <pdns-users at mailman.powerdns.com>:
>>>
>>>     hello,
>>>
>>>     We're using dnsdist (1.5.1 on Ubuntu 20.04, 16 cores, 32GB RAM)
>>as a DoH
>>>     proxy/LB with normal DNS/53 resolvers as backend. This is a test
>>>     installation and we're trying to figure out the performance. It
>>can
>>>     barely handle 1.5k QpS, which I consider pretty low (each
>backend
>>>     resolver can easily handle >60k QpS). It seems that each time
>the
>>>     queries rate is higher than ~1.5k, all backend servers are
>marked
>>"DOWN"
>>>     until the rate goes below 1k. I understand that dnsdist marks
>the
>>>     servers down because it's not receiving a response on its
>>healthcheck
>>>     query and I wonder why.
>>>
>>>     Should I increase "checkTimeout" and "checkInterval"? Should I
>>use a
>>>     large number for "sockets"? Am I missing other tuning options or
>>maybe
>>>     something more important?
>>>
>>>     Here's the relevant config (addresses, etc changed)
>>>
>>>     setLocal('0.0.0.0:5300')
>>>     addLocal('[::1]:5300')
>>>     controlSocket('local_public_address:xxxx')
>>>     setKey("XXX")
>>>     setConsoleACL('x.x.x.x/24')
>>>     NotRule(MaxQPSRule(50000))
>>>     setMaxUDPOutstanding(65535)
>>>     setMaxTCPClientThreads(128)
>>>     setMaxTCPQueuedConnections(10000)
>>>     setMaxTCPConnectionDuration(600)
>>>     PrimaryCache = newPacketCache(30000000, { keepStaleData=true,
>>>     maxTTL=86400, minTTL=0, numberOfShards=8, maxNegativeTTL=600,
>>staleTTL=60 })
>>>     getPool(""):setCache(PrimaryCache)
>>>     addDOHLocal('10.2.3.4', 'cert.pem', 'key.key', "/dns-query", {
>>>     reusePort=true, minTLSVersion='tls1.2' })
>>>     addDOHLocal('10.2.3.4', 'cert.pem', 'key.key', "/dns-query", {
>>>     reusePort=true, minTLSVersion='tls1.2' })
>>>     addDOHLocal('10.2.3.4', 'cert.pem', 'key.key', "/dns-query", {
>>>     reusePort=true, minTLSVersion='tls1.2' })
>>>     addDOHLocal('10.2.3.4', 'cert.pem', 'key.key', "/dns-query", {
>>>     reusePort=true, minTLSVersion='tls1.2' })
>>>     addDOHLocal('2001:DB8::443', 'cert.pem', 'key.key',
>"/dns-query",
>>{
>>>     reusePort=true, minTLSVersion='tls1.2' })
>>>     addDOHLocal('2001:DB8::443', 'cert.pem', 'key.key',
>"/dns-query",
>>{
>>>     reusePort=true, minTLSVersion='tls1.2' })
>>>     addDOHLocal('2001:DB8::443', 'cert.pem', 'key.key',
>"/dns-query",
>>{
>>>     reusePort=true, minTLSVersion='tls1.2' })
>>>     addDOHLocal('2001:DB8::443', 'cert.pem', 'key.key',
>"/dns-query",
>>{
>>>     reusePort=true, minTLSVersion='tls1.2' })
>>>     newServer({address="2001:DB8::62", qps=10000})
>>>     newServer({address="2001:DB8::61", qps=10000})
>>>     newServer({address="2001:DB8::60", qps=10000})
>>>     newServer({address="2001:DB8::59", qps=10000})
>>>     newServer({address="2001:DB8::58", qps=10000})
>>>     newServer({address="2001:DB8::57", qps=10000})
>>>     newServer({address="2001:DB8::56", qps=10000})
>>>     newServer({address="2001:DB8::55", qps=10000})
>>>     newServer({address="2001:DB8::48", qps=10000})
>>>     newServer({address="2001:DB8::47", qps=10000})
>>>     newServer({address="10.10.10.62", qps=10000})
>>>     newServer({address="10.10.10.61", qps=10000})
>>>     newServer({address="10.10.10.60", qps=10000})
>>>     newServer({address="10.10.10.59", qps=10000})
>>>     newServer({address="10.10.10.58", qps=10000})
>>>     newServer({address="10.10.10.57", qps=10000})
>>>     newServer({address="10.10.10.56", qps=10000})
>>>     newServer({address="10.10.10.55", qps=10000})
>>>     newServer({address="10.10.10.48", qps=10000})
>>>     newServer({address="10.10.10.47", qps=10000})
>>>     setServerPolicy(roundrobin)
>>>
>>>     thanks in advance, I'd appreciate any input :)
>>>    
>>------------------------------------------------------------------------
>>>     Pdns-users mailing list
>>>     Pdns-users at mailman.powerdns.com
>>>     https://mailman.powerdns.com/mailman/listinfo/pdns-users 
>><https://mailman.powerdns.com/mailman/listinfo/pdns-users>
>>>
>>>
>>> _______________________________________________
>>> Pdns-users mailing list
>>> Pdns-users at mailman.powerdns.com
>>> https://mailman.powerdns.com/mailman/listinfo/pdns-users
>
>-- 
>Diese Nachricht wurde von meinem Android-Gerät mit K-9 Mail gesendet.

-- 
Diese Nachricht wurde von meinem Android-Gerät mit K-9 Mail gesendet.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.powerdns.com/pipermail/pdns-users/attachments/20210724/674107d2/attachment.htm>