[Pdns-users] dnsdist marking downstream servers down: tuning issue?

[Winfried Angele]

Does this only happen with DoH frontends? Did you try with UDP frontends as well? Sounds like a bottleneck on your backends imo.

Winfried


Am 23. Juli 2021 13:32:39 MESZ schrieb Yannis via Pdns-users <pdns-users at mailman.powerdns.com>:
>hello,
>
>We're using dnsdist (1.5.1 on Ubuntu 20.04, 16 cores, 32GB RAM) as a
>DoH 
>proxy/LB with normal DNS/53 resolvers as backend. This is a test 
>installation and we're trying to figure out the performance. It can 
>barely handle 1.5k QpS, which I consider pretty low (each backend 
>resolver can easily handle >60k QpS). It seems that each time the 
>queries rate is higher than ~1.5k, all backend servers are marked
>"DOWN" 
>until the rate goes below 1k. I understand that dnsdist marks the 
>servers down because it's not receiving a response on its healthcheck 
>query and I wonder why.
>
>Should I increase "checkTimeout" and "checkInterval"? Should I use a 
>large number for "sockets"? Am I missing other tuning options or maybe 
>something more important?
>
>Here's the relevant config (addresses, etc changed)
>
>setLocal('0.0.0.0:5300')
>addLocal('[::1]:5300')
>controlSocket('local_public_address:xxxx')
>setKey("XXX")
>setConsoleACL('x.x.x.x/24')
>NotRule(MaxQPSRule(50000))
>setMaxUDPOutstanding(65535)
>setMaxTCPClientThreads(128)
>setMaxTCPQueuedConnections(10000)
>setMaxTCPConnectionDuration(600)
>PrimaryCache = newPacketCache(30000000, { keepStaleData=true, 
>maxTTL=86400, minTTL=0, numberOfShards=8, maxNegativeTTL=600,
>staleTTL=60 })
>getPool(""):setCache(PrimaryCache)
>addDOHLocal('10.2.3.4', 'cert.pem', 'key.key', "/dns-query", { 
>reusePort=true, minTLSVersion='tls1.2' })
>addDOHLocal('10.2.3.4', 'cert.pem', 'key.key', "/dns-query", { 
>reusePort=true, minTLSVersion='tls1.2' })
>addDOHLocal('10.2.3.4', 'cert.pem', 'key.key', "/dns-query", { 
>reusePort=true, minTLSVersion='tls1.2' })
>addDOHLocal('10.2.3.4', 'cert.pem', 'key.key', "/dns-query", { 
>reusePort=true, minTLSVersion='tls1.2' })
>addDOHLocal('2001:DB8::443', 'cert.pem', 'key.key', "/dns-query", { 
>reusePort=true, minTLSVersion='tls1.2' })
>addDOHLocal('2001:DB8::443', 'cert.pem', 'key.key', "/dns-query", { 
>reusePort=true, minTLSVersion='tls1.2' })
>addDOHLocal('2001:DB8::443', 'cert.pem', 'key.key', "/dns-query", { 
>reusePort=true, minTLSVersion='tls1.2' })
>addDOHLocal('2001:DB8::443', 'cert.pem', 'key.key', "/dns-query", { 
>reusePort=true, minTLSVersion='tls1.2' })
>newServer({address="2001:DB8::62", qps=10000})
>newServer({address="2001:DB8::61", qps=10000})
>newServer({address="2001:DB8::60", qps=10000})
>newServer({address="2001:DB8::59", qps=10000})
>newServer({address="2001:DB8::58", qps=10000})
>newServer({address="2001:DB8::57", qps=10000})
>newServer({address="2001:DB8::56", qps=10000})
>newServer({address="2001:DB8::55", qps=10000})
>newServer({address="2001:DB8::48", qps=10000})
>newServer({address="2001:DB8::47", qps=10000})
>newServer({address="10.10.10.62", qps=10000})
>newServer({address="10.10.10.61", qps=10000})
>newServer({address="10.10.10.60", qps=10000})
>newServer({address="10.10.10.59", qps=10000})
>newServer({address="10.10.10.58", qps=10000})
>newServer({address="10.10.10.57", qps=10000})
>newServer({address="10.10.10.56", qps=10000})
>newServer({address="10.10.10.55", qps=10000})
>newServer({address="10.10.10.48", qps=10000})
>newServer({address="10.10.10.47", qps=10000})
>setServerPolicy(roundrobin)
>
>thanks in advance, I'd appreciate any input :)
>_______________________________________________
>Pdns-users mailing list
>Pdns-users at mailman.powerdns.com
>https://mailman.powerdns.com/mailman/listinfo/pdns-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.powerdns.com/pipermail/pdns-users/attachments/20210723/1ec3728f/attachment.htm>