<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
Hello again,<br>
<br>
just talking to myself here :) . After making sure that there was no
issue with the backend resolvers, I noticed a lot of TCP resets from
the testing subnet towards the DoH endpoint, so I went ahead and
increased setMaxTCPClientThreads from 128 to 4096, I also increased
file descriptors to 2^18 and finally upgraded to dnsdist 1.6.0 .
Everything is OK now, I can easily reach 70k QpS (probably more)<br>
<br>
cheers,<br>
Yannis<br>
<br>
<div class="moz-cite-prefix">On 7/25/21 7:41 PM, Dez C via
Pdns-users wrote:<br>
</div>
<blockquote type="cite"
cite="mid:1c90d88f-94e6-bd61-a7f4-f4074a1ba0b5@otenet.gr">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
Hi,<br>
<br>
I'm also controlling the backend servers and no such rate limit
exists. As I mentioned, each backend server can easily handle 60k
QpS and the numbers of queries that should be reaching the backend
servers from the performance testing, should not be more than
100-200 QpS per backend server.<br>
<br>
The reason I sent my dnsdist configuration is so that people -more
experienced than me- could spot obvious omissions or errors. <br>
<br>
cheers<br>
<br>
<div class="moz-cite-prefix">On 7/24/21 11:41 AM, Winfried Angele
via Pdns-users wrote:<br>
</div>
<blockquote type="cite"
cite="mid:248BE592-13FE-4520-AEBB-2250A26FF768@t-ipnet.net">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8">
Maybe there is some kind of ratelimt in the backend. Keep in
mind that from the backend point of view all queries come from
the same source<br>
<br>
Winfried<br>
<br>
<div class="gmail_quote">Am 24. Juli 2021 10:28:49 MESZ schrieb
Winfried Angele via Pdns-users <a
class="moz-txt-link-rfc2396E"
href="mailto:pdns-users@mailman.powerdns.com"
moz-do-not-send="true"><pdns-users@mailman.powerdns.com></a>:
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt
0.8ex; border-left: 1px solid rgb(204, 204, 204);
padding-left: 1ex;"> Right. Therefore I would recommend
rather to troubleshoot the backend<br>
<br>
Winfried<br>
<br>
<br>
<div class="gmail_quote">Am 23. Juli 2021 20:56:55 MESZ
schrieb Dez C via Pdns-users <a
class="moz-txt-link-rfc2396E"
href="mailto:pdns-users@mailman.powerdns.com"
moz-do-not-send="true"><pdns-users@mailman.powerdns.com></a>:
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt
0.8ex; border-left: 1px solid rgb(204, 204, 204);
padding-left: 1ex;"> Hi,<br>
<br>
<div class="moz-cite-prefix">On 7/23/21 4:14 PM,
Winfried Angele via Pdns-users wrote:<br>
</div>
<blockquote type="cite"
cite="mid:53F0796E-3088-4334-8FB9-BA825CA191A5@t-ipnet.net">
<meta http-equiv="content-type" content="text/html;
charset=UTF-8">
Does this only happen with DoH frontends? Did you try
with UDP frontends as well? Sounds like a bottleneck
on your backends imo.<br>
</blockquote>
<br>
I'm only using dnsdist for DoH so I haven't tried with
UDP/53. I doubt it'll make a difference because the
problem manifests itself when dnsdist is trying to query
the backends (eventhough most replies already exist in
dnsdist's cache)<br>
<br>
cheers<br>
<br>
<blockquote type="cite"
cite="mid:53F0796E-3088-4334-8FB9-BA825CA191A5@t-ipnet.net"><br>
Winfried<br>
<br>
<br>
<div class="gmail_quote">Am 23. Juli 2021 13:32:39
MESZ schrieb Yannis via Pdns-users <a
class="moz-txt-link-rfc2396E"
href="mailto:pdns-users@mailman.powerdns.com"
moz-do-not-send="true"><pdns-users@mailman.powerdns.com></a>:
<blockquote class="gmail_quote" style="margin: 0pt
0pt 0pt 0.8ex; border-left: 1px solid rgb(204,
204, 204); padding-left: 1ex;">
<pre class="k9mail">hello,
We're using dnsdist (1.5.1 on Ubuntu 20.04, 16 cores, 32GB RAM) as a DoH
proxy/LB with normal DNS/53 resolvers as backend. This is a test
installation and we're trying to figure out the performance. It can
barely handle 1.5k QpS, which I consider pretty low (each backend
resolver can easily handle >60k QpS). It seems that each time the
queries rate is higher than ~1.5k, all backend servers are marked "DOWN"
until the rate goes below 1k. I understand that dnsdist marks the
servers down because it's not receiving a response on its healthcheck
query and I wonder why.
Should I increase "checkTimeout" and "checkInterval"? Should I use a
large number for "sockets"? Am I missing other tuning options or maybe
something more important?
Here's the relevant config (addresses, etc changed)
setLocal('0.0.0.0:5300')
addLocal('[::1]:5300')
controlSocket('local_public_address:xxxx')
setKey("XXX")
setConsoleACL('x.x.x.x/24')
NotRule(MaxQPSRule(50000))
setMaxUDPOutstanding(65535)
setMaxTCPClientThreads(128)
setMaxTCPQueuedConnections(10000)
setMaxTCPConnectionDuration(600)
PrimaryCache = newPacketCache(30000000, { keepStaleData=true,
maxTTL=86400, minTTL=0, numberOfShards=8, maxNegativeTTL=600, staleTTL=60 })
getPool(""):setCache(PrimaryCache)
addDOHLocal('10.2.3.4', 'cert.pem', 'key.key', "/dns-query", {
reusePort=true, minTLSVersion='tls1.2' })
addDOHLocal('10.2.3.4', 'cert.pem', 'key.key', "/dns-query", {
reusePort=true, minTLSVersion='tls1.2' })
addDOHLocal('10.2.3.4', 'cert.pem', 'key.key', "/dns-query", {
reusePort=true, minTLSVersion='tls1.2' })
addDOHLocal('10.2.3.4', 'cert.pem', 'key.key', "/dns-query", {
reusePort=true, minTLSVersion='tls1.2' })
addDOHLocal('2001:DB8::443', 'cert.pem', 'key.key', "/dns-query", {
reusePort=true, minTLSVersion='tls1.2' })
addDOHLocal('2001:DB8::443', 'cert.pem', 'key.key', "/dns-query", {
reusePort=true, minTLSVersion='tls1.2' })
addDOHLocal('2001:DB8::443', 'cert.pem', 'key.key', "/dns-query", {
reusePort=true, minTLSVersion='tls1.2' })
addDOHLocal('2001:DB8::443', 'cert.pem', 'key.key', "/dns-query", {
reusePort=true, minTLSVersion='tls1.2' })
newServer({address="2001:DB8::62", qps=10000})
newServer({address="2001:DB8::61", qps=10000})
newServer({address="2001:DB8::60", qps=10000})
newServer({address="2001:DB8::59", qps=10000})
newServer({address="2001:DB8::58", qps=10000})
newServer({address="2001:DB8::57", qps=10000})
newServer({address="2001:DB8::56", qps=10000})
newServer({address="2001:DB8::55", qps=10000})
newServer({address="2001:DB8::48", qps=10000})
newServer({address="2001:DB8::47", qps=10000})
newServer({address="10.10.10.62", qps=10000})
newServer({address="10.10.10.61", qps=10000})
newServer({address="10.10.10.60", qps=10000})
newServer({address="10.10.10.59", qps=10000})
newServer({address="10.10.10.58", qps=10000})
newServer({address="10.10.10.57", qps=10000})
newServer({address="10.10.10.56", qps=10000})
newServer({address="10.10.10.55", qps=10000})
newServer({address="10.10.10.48", qps=10000})
newServer({address="10.10.10.47", qps=10000})
setServerPolicy(roundrobin)
thanks in advance, I'd appreciate any input :)<hr>Pdns-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Pdns-users@mailman.powerdns.com" moz-do-not-send="true">Pdns-users@mailman.powerdns.com</a>
<a href="https://mailman.powerdns.com/mailman/listinfo/pdns-users" moz-do-not-send="true">https://mailman.powerdns.com/mailman/listinfo/pdns-users</a>
</pre>
</blockquote>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
Pdns-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Pdns-users@mailman.powerdns.com" moz-do-not-send="true">Pdns-users@mailman.powerdns.com</a>
<a class="moz-txt-link-freetext" href="https://mailman.powerdns.com/mailman/listinfo/pdns-users" moz-do-not-send="true">https://mailman.powerdns.com/mailman/listinfo/pdns-users</a>
</pre>
</blockquote>
<br>
</blockquote>
</div>
</blockquote>
</div>
<br>
-- <br>
Diese Nachricht wurde von meinem Android-Gerät mit K-9 Mail
gesendet. <br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
Pdns-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Pdns-users@mailman.powerdns.com" moz-do-not-send="true">Pdns-users@mailman.powerdns.com</a>
<a class="moz-txt-link-freetext" href="https://mailman.powerdns.com/mailman/listinfo/pdns-users" moz-do-not-send="true">https://mailman.powerdns.com/mailman/listinfo/pdns-users</a>
</pre>
</blockquote>
<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
Pdns-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Pdns-users@mailman.powerdns.com">Pdns-users@mailman.powerdns.com</a>
<a class="moz-txt-link-freetext" href="https://mailman.powerdns.com/mailman/listinfo/pdns-users">https://mailman.powerdns.com/mailman/listinfo/pdns-users</a>
</pre>
</blockquote>
<br>
</body>
</html>