[Pdns-users] dnsdist marking downstream servers down: tuning issue?

[Dez C]

Hi,

On 7/23/21 4:14 PM, Winfried Angele via Pdns-users wrote:
> Does this only happen with DoH frontends? Did you try with UDP 
> frontends as well? Sounds like a bottleneck on your backends imo.

I'm only using dnsdist for DoH so I haven't tried with UDP/53. I doubt 
it'll make a difference because the problem manifests itself when 
dnsdist is trying to query the backends (eventhough most replies already 
exist in dnsdist's cache)

cheers

>
> Winfried
>
>
> Am 23. Juli 2021 13:32:39 MESZ schrieb Yannis via Pdns-users 
> <pdns-users at mailman.powerdns.com>:
>
>     hello,
>
>     We're using dnsdist (1.5.1 on Ubuntu 20.04, 16 cores, 32GB RAM) as a DoH
>     proxy/LB with normal DNS/53 resolvers as backend. This is a test
>     installation and we're trying to figure out the performance. It can
>     barely handle 1.5k QpS, which I consider pretty low (each backend
>     resolver can easily handle >60k QpS). It seems that each time the
>     queries rate is higher than ~1.5k, all backend servers are marked "DOWN"
>     until the rate goes below 1k. I understand that dnsdist marks the
>     servers down because it's not receiving a response on its healthcheck
>     query and I wonder why.
>
>     Should I increase "checkTimeout" and "checkInterval"? Should I use a
>     large number for "sockets"? Am I missing other tuning options or maybe
>     something more important?
>
>     Here's the relevant config (addresses, etc changed)
>
>     setLocal('0.0.0.0:5300')
>     addLocal('[::1]:5300')
>     controlSocket('local_public_address:xxxx')
>     setKey("XXX")
>     setConsoleACL('x.x.x.x/24')
>     NotRule(MaxQPSRule(50000))
>     setMaxUDPOutstanding(65535)
>     setMaxTCPClientThreads(128)
>     setMaxTCPQueuedConnections(10000)
>     setMaxTCPConnectionDuration(600)
>     PrimaryCache = newPacketCache(30000000, { keepStaleData=true,
>     maxTTL=86400, minTTL=0, numberOfShards=8, maxNegativeTTL=600, staleTTL=60 })
>     getPool(""):setCache(PrimaryCache)
>     addDOHLocal('10.2.3.4', 'cert.pem', 'key.key', "/dns-query", {
>     reusePort=true, minTLSVersion='tls1.2' })
>     addDOHLocal('10.2.3.4', 'cert.pem', 'key.key', "/dns-query", {
>     reusePort=true, minTLSVersion='tls1.2' })
>     addDOHLocal('10.2.3.4', 'cert.pem', 'key.key', "/dns-query", {
>     reusePort=true, minTLSVersion='tls1.2' })
>     addDOHLocal('10.2.3.4', 'cert.pem', 'key.key', "/dns-query", {
>     reusePort=true, minTLSVersion='tls1.2' })
>     addDOHLocal('2001:DB8::443', 'cert.pem', 'key.key', "/dns-query", {
>     reusePort=true, minTLSVersion='tls1.2' })
>     addDOHLocal('2001:DB8::443', 'cert.pem', 'key.key', "/dns-query", {
>     reusePort=true, minTLSVersion='tls1.2' })
>     addDOHLocal('2001:DB8::443', 'cert.pem', 'key.key', "/dns-query", {
>     reusePort=true, minTLSVersion='tls1.2' })
>     addDOHLocal('2001:DB8::443', 'cert.pem', 'key.key', "/dns-query", {
>     reusePort=true, minTLSVersion='tls1.2' })
>     newServer({address="2001:DB8::62", qps=10000})
>     newServer({address="2001:DB8::61", qps=10000})
>     newServer({address="2001:DB8::60", qps=10000})
>     newServer({address="2001:DB8::59", qps=10000})
>     newServer({address="2001:DB8::58", qps=10000})
>     newServer({address="2001:DB8::57", qps=10000})
>     newServer({address="2001:DB8::56", qps=10000})
>     newServer({address="2001:DB8::55", qps=10000})
>     newServer({address="2001:DB8::48", qps=10000})
>     newServer({address="2001:DB8::47", qps=10000})
>     newServer({address="10.10.10.62", qps=10000})
>     newServer({address="10.10.10.61", qps=10000})
>     newServer({address="10.10.10.60", qps=10000})
>     newServer({address="10.10.10.59", qps=10000})
>     newServer({address="10.10.10.58", qps=10000})
>     newServer({address="10.10.10.57", qps=10000})
>     newServer({address="10.10.10.56", qps=10000})
>     newServer({address="10.10.10.55", qps=10000})
>     newServer({address="10.10.10.48", qps=10000})
>     newServer({address="10.10.10.47", qps=10000})
>     setServerPolicy(roundrobin)
>
>     thanks in advance, I'd appreciate any input :)
>     ------------------------------------------------------------------------
>     Pdns-users mailing list
>     Pdns-users at mailman.powerdns.com
>     https://mailman.powerdns.com/mailman/listinfo/pdns-users  <https://mailman.powerdns.com/mailman/listinfo/pdns-users>
>
>
> _______________________________________________
> Pdns-users mailing list
> Pdns-users at mailman.powerdns.com
> https://mailman.powerdns.com/mailman/listinfo/pdns-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.powerdns.com/pipermail/pdns-users/attachments/20210723/a3e7287c/attachment.htm>