[Pdns-users] dnsdist marking downstream servers down: tuning issue?

[Winfried Angele]

Right. Therefore I would recommend rather to troubleshoot the backend

Winfried


Am 23. Juli 2021 20:56:55 MESZ schrieb Dez C via Pdns-users <pdns-users at mailman.powerdns.com>:
>Hi,
>
>On 7/23/21 4:14 PM, Winfried Angele via Pdns-users wrote:
>> Does this only happen with DoH frontends? Did you try with UDP 
>> frontends as well? Sounds like a bottleneck on your backends imo.
>
>I'm only using dnsdist for DoH so I haven't tried with UDP/53. I doubt 
>it'll make a difference because the problem manifests itself when 
>dnsdist is trying to query the backends (eventhough most replies
>already 
>exist in dnsdist's cache)
>
>cheers
>
>>
>> Winfried
>>
>>
>> Am 23. Juli 2021 13:32:39 MESZ schrieb Yannis via Pdns-users 
>> <pdns-users at mailman.powerdns.com>:
>>
>>     hello,
>>
>>     We're using dnsdist (1.5.1 on Ubuntu 20.04, 16 cores, 32GB RAM)
>as a DoH
>>     proxy/LB with normal DNS/53 resolvers as backend. This is a test
>>     installation and we're trying to figure out the performance. It
>can
>>     barely handle 1.5k QpS, which I consider pretty low (each backend
>>     resolver can easily handle >60k QpS). It seems that each time the
>>     queries rate is higher than ~1.5k, all backend servers are marked
>"DOWN"
>>     until the rate goes below 1k. I understand that dnsdist marks the
>>     servers down because it's not receiving a response on its
>healthcheck
>>     query and I wonder why.
>>
>>     Should I increase "checkTimeout" and "checkInterval"? Should I
>use a
>>     large number for "sockets"? Am I missing other tuning options or
>maybe
>>     something more important?
>>
>>     Here's the relevant config (addresses, etc changed)
>>
>>     setLocal('0.0.0.0:5300')
>>     addLocal('[::1]:5300')
>>     controlSocket('local_public_address:xxxx')
>>     setKey("XXX")
>>     setConsoleACL('x.x.x.x/24')
>>     NotRule(MaxQPSRule(50000))
>>     setMaxUDPOutstanding(65535)
>>     setMaxTCPClientThreads(128)
>>     setMaxTCPQueuedConnections(10000)
>>     setMaxTCPConnectionDuration(600)
>>     PrimaryCache = newPacketCache(30000000, { keepStaleData=true,
>>     maxTTL=86400, minTTL=0, numberOfShards=8, maxNegativeTTL=600,
>staleTTL=60 })
>>     getPool(""):setCache(PrimaryCache)
>>     addDOHLocal('10.2.3.4', 'cert.pem', 'key.key', "/dns-query", {
>>     reusePort=true, minTLSVersion='tls1.2' })
>>     addDOHLocal('10.2.3.4', 'cert.pem', 'key.key', "/dns-query", {
>>     reusePort=true, minTLSVersion='tls1.2' })
>>     addDOHLocal('10.2.3.4', 'cert.pem', 'key.key', "/dns-query", {
>>     reusePort=true, minTLSVersion='tls1.2' })
>>     addDOHLocal('10.2.3.4', 'cert.pem', 'key.key', "/dns-query", {
>>     reusePort=true, minTLSVersion='tls1.2' })
>>     addDOHLocal('2001:DB8::443', 'cert.pem', 'key.key', "/dns-query",
>{
>>     reusePort=true, minTLSVersion='tls1.2' })
>>     addDOHLocal('2001:DB8::443', 'cert.pem', 'key.key', "/dns-query",
>{
>>     reusePort=true, minTLSVersion='tls1.2' })
>>     addDOHLocal('2001:DB8::443', 'cert.pem', 'key.key', "/dns-query",
>{
>>     reusePort=true, minTLSVersion='tls1.2' })
>>     addDOHLocal('2001:DB8::443', 'cert.pem', 'key.key', "/dns-query",
>{
>>     reusePort=true, minTLSVersion='tls1.2' })
>>     newServer({address="2001:DB8::62", qps=10000})
>>     newServer({address="2001:DB8::61", qps=10000})
>>     newServer({address="2001:DB8::60", qps=10000})
>>     newServer({address="2001:DB8::59", qps=10000})
>>     newServer({address="2001:DB8::58", qps=10000})
>>     newServer({address="2001:DB8::57", qps=10000})
>>     newServer({address="2001:DB8::56", qps=10000})
>>     newServer({address="2001:DB8::55", qps=10000})
>>     newServer({address="2001:DB8::48", qps=10000})
>>     newServer({address="2001:DB8::47", qps=10000})
>>     newServer({address="10.10.10.62", qps=10000})
>>     newServer({address="10.10.10.61", qps=10000})
>>     newServer({address="10.10.10.60", qps=10000})
>>     newServer({address="10.10.10.59", qps=10000})
>>     newServer({address="10.10.10.58", qps=10000})
>>     newServer({address="10.10.10.57", qps=10000})
>>     newServer({address="10.10.10.56", qps=10000})
>>     newServer({address="10.10.10.55", qps=10000})
>>     newServer({address="10.10.10.48", qps=10000})
>>     newServer({address="10.10.10.47", qps=10000})
>>     setServerPolicy(roundrobin)
>>
>>     thanks in advance, I'd appreciate any input :)
>>    
>------------------------------------------------------------------------
>>     Pdns-users mailing list
>>     Pdns-users at mailman.powerdns.com
>>     https://mailman.powerdns.com/mailman/listinfo/pdns-users 
><https://mailman.powerdns.com/mailman/listinfo/pdns-users>
>>
>>
>> _______________________________________________
>> Pdns-users mailing list
>> Pdns-users at mailman.powerdns.com
>> https://mailman.powerdns.com/mailman/listinfo/pdns-users

-- 
Diese Nachricht wurde von meinem Android-Gerät mit K-9 Mail gesendet.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.powerdns.com/pipermail/pdns-users/attachments/20210724/71028f20/attachment-0001.htm>