[Pdns-users] dnsdist marking downstream servers down: tuning issue?

[dez at otenet.gr]

hello,

We're using dnsdist (1.5.1 on Ubuntu 20.04, 16 cores, 32GB RAM) as a DoH 
proxy/LB with normal DNS/53 resolvers as backend. This is a test 
installation and we're trying to figure out the performance. It can 
barely handle 1.5k QpS, which I consider pretty low (each backend 
resolver can easily handle >60k QpS). It seems that each time the 
queries rate is higher than ~1.5k, all backend servers are marked "DOWN" 
until the rate goes below 1k. I understand that dnsdist marks the 
servers down because it's not receiving a response on its healthcheck 
query and I wonder why.

Should I increase "checkTimeout" and "checkInterval"? Should I use a 
large number for "sockets"? Am I missing other tuning options or maybe 
something more important?

Here's the relevant config (addresses, etc changed)

setLocal('0.0.0.0:5300')
addLocal('[::1]:5300')
controlSocket('local_public_address:xxxx')
setKey("XXX")
setConsoleACL('x.x.x.x/24')
NotRule(MaxQPSRule(50000))
setMaxUDPOutstanding(65535)
setMaxTCPClientThreads(128)
setMaxTCPQueuedConnections(10000)
setMaxTCPConnectionDuration(600)
PrimaryCache = newPacketCache(30000000, { keepStaleData=true, 
maxTTL=86400, minTTL=0, numberOfShards=8, maxNegativeTTL=600, staleTTL=60 })
getPool(""):setCache(PrimaryCache)
addDOHLocal('10.2.3.4', 'cert.pem', 'key.key', "/dns-query", { 
reusePort=true, minTLSVersion='tls1.2' })
addDOHLocal('10.2.3.4', 'cert.pem', 'key.key', "/dns-query", { 
reusePort=true, minTLSVersion='tls1.2' })
addDOHLocal('10.2.3.4', 'cert.pem', 'key.key', "/dns-query", { 
reusePort=true, minTLSVersion='tls1.2' })
addDOHLocal('10.2.3.4', 'cert.pem', 'key.key', "/dns-query", { 
reusePort=true, minTLSVersion='tls1.2' })
addDOHLocal('2001:DB8::443', 'cert.pem', 'key.key', "/dns-query", { 
reusePort=true, minTLSVersion='tls1.2' })
addDOHLocal('2001:DB8::443', 'cert.pem', 'key.key', "/dns-query", { 
reusePort=true, minTLSVersion='tls1.2' })
addDOHLocal('2001:DB8::443', 'cert.pem', 'key.key', "/dns-query", { 
reusePort=true, minTLSVersion='tls1.2' })
addDOHLocal('2001:DB8::443', 'cert.pem', 'key.key', "/dns-query", { 
reusePort=true, minTLSVersion='tls1.2' })
newServer({address="2001:DB8::62", qps=10000})
newServer({address="2001:DB8::61", qps=10000})
newServer({address="2001:DB8::60", qps=10000})
newServer({address="2001:DB8::59", qps=10000})
newServer({address="2001:DB8::58", qps=10000})
newServer({address="2001:DB8::57", qps=10000})
newServer({address="2001:DB8::56", qps=10000})
newServer({address="2001:DB8::55", qps=10000})
newServer({address="2001:DB8::48", qps=10000})
newServer({address="2001:DB8::47", qps=10000})
newServer({address="10.10.10.62", qps=10000})
newServer({address="10.10.10.61", qps=10000})
newServer({address="10.10.10.60", qps=10000})
newServer({address="10.10.10.59", qps=10000})
newServer({address="10.10.10.58", qps=10000})
newServer({address="10.10.10.57", qps=10000})
newServer({address="10.10.10.56", qps=10000})
newServer({address="10.10.10.55", qps=10000})
newServer({address="10.10.10.48", qps=10000})
newServer({address="10.10.10.47", qps=10000})
setServerPolicy(roundrobin)

thanks in advance, I'd appreciate any input :)