[Pdns-users] dnsdist marking downstream servers down: tuning issue?
Dez C
dez at otenet.gr
Sun Jul 25 16:41:48 UTC 2021
Hi,
I'm also controlling the backend servers and no such rate limit exists.
As I mentioned, each backend server can easily handle 60k QpS and the
numbers of queries that should be reaching the backend servers from the
performance testing, should not be more than 100-200 QpS per backend server.
The reason I sent my dnsdist configuration is so that people -more
experienced than me- could spot obvious omissions or errors.
cheers
On 7/24/21 11:41 AM, Winfried Angele via Pdns-users wrote:
> Maybe there is some kind of ratelimt in the backend. Keep in mind that
> from the backend point of view all queries come from the same source
>
> Winfried
>
> Am 24. Juli 2021 10:28:49 MESZ schrieb Winfried Angele via Pdns-users
> <pdns-users at mailman.powerdns.com>:
>
> Right. Therefore I would recommend rather to troubleshoot the backend
>
> Winfried
>
>
> Am 23. Juli 2021 20:56:55 MESZ schrieb Dez C via Pdns-users
> <pdns-users at mailman.powerdns.com>:
>
> Hi,
>
> On 7/23/21 4:14 PM, Winfried Angele via Pdns-users wrote:
>> Does this only happen with DoH frontends? Did you try with
>> UDP frontends as well? Sounds like a bottleneck on your
>> backends imo.
>
> I'm only using dnsdist for DoH so I haven't tried with UDP/53.
> I doubt it'll make a difference because the problem manifests
> itself when dnsdist is trying to query the backends
> (eventhough most replies already exist in dnsdist's cache)
>
> cheers
>
>>
>> Winfried
>>
>>
>> Am 23. Juli 2021 13:32:39 MESZ schrieb Yannis via Pdns-users
>> <pdns-users at mailman.powerdns.com>:
>>
>> hello,
>>
>> We're using dnsdist (1.5.1 on Ubuntu 20.04, 16 cores, 32GB RAM) as a DoH
>> proxy/LB with normal DNS/53 resolvers as backend. This is a test
>> installation and we're trying to figure out the performance. It can
>> barely handle 1.5k QpS, which I consider pretty low (each backend
>> resolver can easily handle >60k QpS). It seems that each time the
>> queries rate is higher than ~1.5k, all backend servers are marked "DOWN"
>> until the rate goes below 1k. I understand that dnsdist marks the
>> servers down because it's not receiving a response on its healthcheck
>> query and I wonder why.
>>
>> Should I increase "checkTimeout" and "checkInterval"? Should I use a
>> large number for "sockets"? Am I missing other tuning options or maybe
>> something more important?
>>
>> Here's the relevant config (addresses, etc changed)
>>
>> setLocal('0.0.0.0:5300')
>> addLocal('[::1]:5300')
>> controlSocket('local_public_address:xxxx')
>> setKey("XXX")
>> setConsoleACL('x.x.x.x/24')
>> NotRule(MaxQPSRule(50000))
>> setMaxUDPOutstanding(65535)
>> setMaxTCPClientThreads(128)
>> setMaxTCPQueuedConnections(10000)
>> setMaxTCPConnectionDuration(600)
>> PrimaryCache = newPacketCache(30000000, { keepStaleData=true,
>> maxTTL=86400, minTTL=0, numberOfShards=8, maxNegativeTTL=600, staleTTL=60 })
>> getPool(""):setCache(PrimaryCache)
>> addDOHLocal('10.2.3.4', 'cert.pem', 'key.key', "/dns-query", {
>> reusePort=true, minTLSVersion='tls1.2' })
>> addDOHLocal('10.2.3.4', 'cert.pem', 'key.key', "/dns-query", {
>> reusePort=true, minTLSVersion='tls1.2' })
>> addDOHLocal('10.2.3.4', 'cert.pem', 'key.key', "/dns-query", {
>> reusePort=true, minTLSVersion='tls1.2' })
>> addDOHLocal('10.2.3.4', 'cert.pem', 'key.key', "/dns-query", {
>> reusePort=true, minTLSVersion='tls1.2' })
>> addDOHLocal('2001:DB8::443', 'cert.pem', 'key.key', "/dns-query", {
>> reusePort=true, minTLSVersion='tls1.2' })
>> addDOHLocal('2001:DB8::443', 'cert.pem', 'key.key', "/dns-query", {
>> reusePort=true, minTLSVersion='tls1.2' })
>> addDOHLocal('2001:DB8::443', 'cert.pem', 'key.key', "/dns-query", {
>> reusePort=true, minTLSVersion='tls1.2' })
>> addDOHLocal('2001:DB8::443', 'cert.pem', 'key.key', "/dns-query", {
>> reusePort=true, minTLSVersion='tls1.2' })
>> newServer({address="2001:DB8::62", qps=10000})
>> newServer({address="2001:DB8::61", qps=10000})
>> newServer({address="2001:DB8::60", qps=10000})
>> newServer({address="2001:DB8::59", qps=10000})
>> newServer({address="2001:DB8::58", qps=10000})
>> newServer({address="2001:DB8::57", qps=10000})
>> newServer({address="2001:DB8::56", qps=10000})
>> newServer({address="2001:DB8::55", qps=10000})
>> newServer({address="2001:DB8::48", qps=10000})
>> newServer({address="2001:DB8::47", qps=10000})
>> newServer({address="10.10.10.62", qps=10000})
>> newServer({address="10.10.10.61", qps=10000})
>> newServer({address="10.10.10.60", qps=10000})
>> newServer({address="10.10.10.59", qps=10000})
>> newServer({address="10.10.10.58", qps=10000})
>> newServer({address="10.10.10.57", qps=10000})
>> newServer({address="10.10.10.56", qps=10000})
>> newServer({address="10.10.10.55", qps=10000})
>> newServer({address="10.10.10.48", qps=10000})
>> newServer({address="10.10.10.47", qps=10000})
>> setServerPolicy(roundrobin)
>>
>> thanks in advance, I'd appreciate any input :)
>> ------------------------------------------------------------------------
>> Pdns-users mailing list
>> Pdns-users at mailman.powerdns.com
>> https://mailman.powerdns.com/mailman/listinfo/pdns-users <https://mailman.powerdns.com/mailman/listinfo/pdns-users>
>>
>>
>> _______________________________________________
>> Pdns-users mailing list
>> Pdns-users at mailman.powerdns.com
>> https://mailman.powerdns.com/mailman/listinfo/pdns-users
>
>
> --
> Diese Nachricht wurde von meinem Android-Gerät mit K-9 Mail gesendet.
>
> _______________________________________________
> Pdns-users mailing list
> Pdns-users at mailman.powerdns.com
> https://mailman.powerdns.com/mailman/listinfo/pdns-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.powerdns.com/pipermail/pdns-users/attachments/20210725/77e9a73d/attachment.htm>
More information about the Pdns-users
mailing list