[Pdns-users] dnsdist marking downstream servers down: tuning issue?
Dez C
dez at otenet.gr
Mon Jul 26 16:32:35 UTC 2021
Hello again,
just talking to myself here :) . After making sure that there was no
issue with the backend resolvers, I noticed a lot of TCP resets from the
testing subnet towards the DoH endpoint, so I went ahead and increased
setMaxTCPClientThreads from 128 to 4096, I also increased file
descriptors to 2^18 and finally upgraded to dnsdist 1.6.0 . Everything
is OK now, I can easily reach 70k QpS (probably more)
cheers,
Yannis
On 7/25/21 7:41 PM, Dez C via Pdns-users wrote:
> Hi,
>
> I'm also controlling the backend servers and no such rate limit
> exists. As I mentioned, each backend server can easily handle 60k QpS
> and the numbers of queries that should be reaching the backend servers
> from the performance testing, should not be more than 100-200 QpS per
> backend server.
>
> The reason I sent my dnsdist configuration is so that people -more
> experienced than me- could spot obvious omissions or errors.
>
> cheers
>
> On 7/24/21 11:41 AM, Winfried Angele via Pdns-users wrote:
>> Maybe there is some kind of ratelimt in the backend. Keep in mind
>> that from the backend point of view all queries come from the same source
>>
>> Winfried
>>
>> Am 24. Juli 2021 10:28:49 MESZ schrieb Winfried Angele via Pdns-users
>> <pdns-users at mailman.powerdns.com>:
>>
>> Right. Therefore I would recommend rather to troubleshoot the backend
>>
>> Winfried
>>
>>
>> Am 23. Juli 2021 20:56:55 MESZ schrieb Dez C via Pdns-users
>> <pdns-users at mailman.powerdns.com>:
>>
>> Hi,
>>
>> On 7/23/21 4:14 PM, Winfried Angele via Pdns-users wrote:
>>> Does this only happen with DoH frontends? Did you try with
>>> UDP frontends as well? Sounds like a bottleneck on your
>>> backends imo.
>>
>> I'm only using dnsdist for DoH so I haven't tried with
>> UDP/53. I doubt it'll make a difference because the problem
>> manifests itself when dnsdist is trying to query the backends
>> (eventhough most replies already exist in dnsdist's cache)
>>
>> cheers
>>
>>>
>>> Winfried
>>>
>>>
>>> Am 23. Juli 2021 13:32:39 MESZ schrieb Yannis via Pdns-users
>>> <pdns-users at mailman.powerdns.com>:
>>>
>>> hello,
>>>
>>> We're using dnsdist (1.5.1 on Ubuntu 20.04, 16 cores, 32GB RAM) as a DoH
>>> proxy/LB with normal DNS/53 resolvers as backend. This is a test
>>> installation and we're trying to figure out the performance. It can
>>> barely handle 1.5k QpS, which I consider pretty low (each backend
>>> resolver can easily handle >60k QpS). It seems that each time the
>>> queries rate is higher than ~1.5k, all backend servers are marked "DOWN"
>>> until the rate goes below 1k. I understand that dnsdist marks the
>>> servers down because it's not receiving a response on its healthcheck
>>> query and I wonder why.
>>>
>>> Should I increase "checkTimeout" and "checkInterval"? Should I use a
>>> large number for "sockets"? Am I missing other tuning options or maybe
>>> something more important?
>>>
>>> Here's the relevant config (addresses, etc changed)
>>>
>>> setLocal('0.0.0.0:5300')
>>> addLocal('[::1]:5300')
>>> controlSocket('local_public_address:xxxx')
>>> setKey("XXX")
>>> setConsoleACL('x.x.x.x/24')
>>> NotRule(MaxQPSRule(50000))
>>> setMaxUDPOutstanding(65535)
>>> setMaxTCPClientThreads(128)
>>> setMaxTCPQueuedConnections(10000)
>>> setMaxTCPConnectionDuration(600)
>>> PrimaryCache = newPacketCache(30000000, { keepStaleData=true,
>>> maxTTL=86400, minTTL=0, numberOfShards=8, maxNegativeTTL=600, staleTTL=60 })
>>> getPool(""):setCache(PrimaryCache)
>>> addDOHLocal('10.2.3.4', 'cert.pem', 'key.key', "/dns-query", {
>>> reusePort=true, minTLSVersion='tls1.2' })
>>> addDOHLocal('10.2.3.4', 'cert.pem', 'key.key', "/dns-query", {
>>> reusePort=true, minTLSVersion='tls1.2' })
>>> addDOHLocal('10.2.3.4', 'cert.pem', 'key.key', "/dns-query", {
>>> reusePort=true, minTLSVersion='tls1.2' })
>>> addDOHLocal('10.2.3.4', 'cert.pem', 'key.key', "/dns-query", {
>>> reusePort=true, minTLSVersion='tls1.2' })
>>> addDOHLocal('2001:DB8::443', 'cert.pem', 'key.key', "/dns-query", {
>>> reusePort=true, minTLSVersion='tls1.2' })
>>> addDOHLocal('2001:DB8::443', 'cert.pem', 'key.key', "/dns-query", {
>>> reusePort=true, minTLSVersion='tls1.2' })
>>> addDOHLocal('2001:DB8::443', 'cert.pem', 'key.key', "/dns-query", {
>>> reusePort=true, minTLSVersion='tls1.2' })
>>> addDOHLocal('2001:DB8::443', 'cert.pem', 'key.key', "/dns-query", {
>>> reusePort=true, minTLSVersion='tls1.2' })
>>> newServer({address="2001:DB8::62", qps=10000})
>>> newServer({address="2001:DB8::61", qps=10000})
>>> newServer({address="2001:DB8::60", qps=10000})
>>> newServer({address="2001:DB8::59", qps=10000})
>>> newServer({address="2001:DB8::58", qps=10000})
>>> newServer({address="2001:DB8::57", qps=10000})
>>> newServer({address="2001:DB8::56", qps=10000})
>>> newServer({address="2001:DB8::55", qps=10000})
>>> newServer({address="2001:DB8::48", qps=10000})
>>> newServer({address="2001:DB8::47", qps=10000})
>>> newServer({address="10.10.10.62", qps=10000})
>>> newServer({address="10.10.10.61", qps=10000})
>>> newServer({address="10.10.10.60", qps=10000})
>>> newServer({address="10.10.10.59", qps=10000})
>>> newServer({address="10.10.10.58", qps=10000})
>>> newServer({address="10.10.10.57", qps=10000})
>>> newServer({address="10.10.10.56", qps=10000})
>>> newServer({address="10.10.10.55", qps=10000})
>>> newServer({address="10.10.10.48", qps=10000})
>>> newServer({address="10.10.10.47", qps=10000})
>>> setServerPolicy(roundrobin)
>>>
>>> thanks in advance, I'd appreciate any input :)
>>> ------------------------------------------------------------------------
>>> Pdns-users mailing list
>>> Pdns-users at mailman.powerdns.com
>>> https://mailman.powerdns.com/mailman/listinfo/pdns-users <https://mailman.powerdns.com/mailman/listinfo/pdns-users>
>>>
>>>
>>> _______________________________________________
>>> Pdns-users mailing list
>>> Pdns-users at mailman.powerdns.com
>>> https://mailman.powerdns.com/mailman/listinfo/pdns-users
>>
>>
>> --
>> Diese Nachricht wurde von meinem Android-Gerät mit K-9 Mail gesendet.
>>
>> _______________________________________________
>> Pdns-users mailing list
>> Pdns-users at mailman.powerdns.com
>> https://mailman.powerdns.com/mailman/listinfo/pdns-users
>
>
> _______________________________________________
> Pdns-users mailing list
> Pdns-users at mailman.powerdns.com
> https://mailman.powerdns.com/mailman/listinfo/pdns-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.powerdns.com/pipermail/pdns-users/attachments/20210726/3b8c4350/attachment.htm>
More information about the Pdns-users
mailing list