<div dir="ltr">Hi Rais,<div><br></div><div>A couple of comments on your config.</div><div>First of all I'lll check how your Cache is working. </div><div>In our setup (and it depends a lot of your type of users, mobile, home, enterprise)</div><div>we have more than 97% cache hit. Use:</div><div>> getPool("poolname"):getCache():printStats()<br>Entries: 1002292/2000000<br>Hits: 23988295086<br>Misses: 352368758<br>Deferred inserts: 254866<br>Deferred lookups: 519271<br>Lookup Collisions: 32863<br>Insert Collisions: 32791<br>TTL Too Shorts: 0<br></div><div><br></div><div>with this cache hit rate, our dnsdist server answers ~200 KQPS but only sends 4 Kqps to the backends </div><div><br></div><div>We also have this flags in the cache settings: numberOfShards=50, deferrableInsertLock=true, maxNegativeTTL=3600</div><div><br></div><div>And also:</div><div>setCacheCleaningDelay(30)<br>setCacheCleaningPercentage(50)<br>setMaxUDPOutstanding(65535)<br></div><div><br></div><div>But I think the main problem you have is that your 16 backend resolvers are only 2 instances.</div><div>And they just can't handle your load.</div><div><br></div><div>Try to find your real backend performance (kqps), your dnsdist cache miss rate to get your</div><div>backend real needs and create as many independent backend resolvers to handle that.</div><div><br></div><div>HTH.</div><div>Regards!</div><div>Nico</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Mar 23, 2022 at 6:06 PM Rais Ahmed via dnsdist <<a href="mailto:dnsdist@mailman.powerdns.com">dnsdist@mailman.powerdns.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi,<br>
Thanks for reply...!<br>
<br>
We have configured setMaxUDPOutstanding(65535) and still we are seeing backend down, logs are showing frequently as below.<br>
<br>
Timeout while waiting for the health check response from backend <a href="http://192.168.1.1:53" rel="noreferrer" target="_blank">192.168.1.1:53</a> Timeout while waiting for the health check response from backend <a href="http://192.168.1.2:53" rel="noreferrer" target="_blank">192.168.1.2:53</a><br>
<br>
Please have a look at below dnsdist configuration and help us to find misconfiguration (16 Listeners & 8+8 backends added as per vCPUs available (2 Socket x 8 Cores):<br>
<br>
controlSocket('<a href="http://127.0.0.1:5199" rel="noreferrer" target="_blank">127.0.0.1:5199</a>')<br>
setKey("")<br>
<br>
---- Listen addresses<br>
addLocal('<a href="http://192.168.0.1:53" rel="noreferrer" target="_blank">192.168.0.1:53</a>', { reusePort=true }) <br>
addLocal('<a href="http://192.168.0.1:53" rel="noreferrer" target="_blank">192.168.0.1:53</a>', { reusePort=true }) <br>
addLocal('<a href="http://192.168.0.1:53" rel="noreferrer" target="_blank">192.168.0.1:53</a>', { reusePort=true }) <br>
addLocal('<a href="http://192.168.0.1:53" rel="noreferrer" target="_blank">192.168.0.1:53</a>', { reusePort=true }) <br>
addLocal('<a href="http://192.168.0.1:53" rel="noreferrer" target="_blank">192.168.0.1:53</a>', { reusePort=true }) <br>
addLocal('<a href="http://192.168.0.1:53" rel="noreferrer" target="_blank">192.168.0.1:53</a>', { reusePort=true }) <br>
addLocal('<a href="http://192.168.0.1:53" rel="noreferrer" target="_blank">192.168.0.1:53</a>', { reusePort=true }) <br>
addLocal('<a href="http://192.168.0.1:53" rel="noreferrer" target="_blank">192.168.0.1:53</a>', { reusePort=true }) <br>
addLocal('<a href="http://192.168.0.1:53" rel="noreferrer" target="_blank">192.168.0.1:53</a>', { reusePort=true })<br>
addLocal('<a href="http://192.168.0.1:53" rel="noreferrer" target="_blank">192.168.0.1:53</a>', { reusePort=true }) <br>
addLocal('<a href="http://192.168.0.1:53" rel="noreferrer" target="_blank">192.168.0.1:53</a>', { reusePort=true }) <br>
addLocal('<a href="http://192.168.0.1:53" rel="noreferrer" target="_blank">192.168.0.1:53</a>', { reusePort=true }) <br>
addLocal('<a href="http://192.168.0.1:53" rel="noreferrer" target="_blank">192.168.0.1:53</a>', { reusePort=true }) <br>
addLocal('<a href="http://192.168.0.1:53" rel="noreferrer" target="_blank">192.168.0.1:53</a>', { reusePort=true }) <br>
addLocal('<a href="http://192.168.0.1:53" rel="noreferrer" target="_blank">192.168.0.1:53</a>', { reusePort=true }) <br>
addLocal('<a href="http://192.168.0.1:53" rel="noreferrer" target="_blank">192.168.0.1:53</a>', { reusePort=true })<br>
<br>
---- Back-end server<br>
newServer({address='192.168.1.1', maxCheckFailures=3, checkInterval=5, weight=4, qps=40000, order=1}) <br>
newServer({address='192.168.1.1', maxCheckFailures=3, checkInterval=5, weight=4, qps=40000, order=2})<br>
newServer({address='192.168.1.1', maxCheckFailures=3, checkInterval=5, weight=4, qps=40000, order=3}) <br>
newServer({address='192.168.1.1', maxCheckFailures=3, checkInterval=5, weight=4, qps=40000, order=4}) <br>
newServer({address='192.168.1.1', maxCheckFailures=3, checkInterval=5, weight=4, qps=40000, order=5}) <br>
newServer({address='192.168.1.1', maxCheckFailures=3, checkInterval=5, weight=4, qps=40000, order=6}) <br>
newServer({address='192.168.1.1', maxCheckFailures=3, checkInterval=5, weight=4, qps=40000, order=7}) <br>
newServer({address='192.168.1.1', maxCheckFailures=3, checkInterval=5, weight=4, qps=40000, order=8}) <br>
newServer({address='192.168.1.2', maxCheckFailures=3, checkInterval=5, weight=4, qps=40000, order=9}) <br>
newServer({address='192.168.1.2', maxCheckFailures=3, checkInterval=5, weight=4, qps=40000, order=10}) <br>
newServer({address='192.168.1.2', maxCheckFailures=3, checkInterval=5, weight=4, qps=40000, order=11}) <br>
newServer({address='192.168.1.2', maxCheckFailures=3, checkInterval=5, weight=4, qps=40000, order=12}) <br>
newServer({address='192.168.1.2', maxCheckFailures=3, checkInterval=5, weight=4, qps=40000, order=13}) <br>
newServer({address='192.168.1.2', maxCheckFailures=3, checkInterval=5, weight=4, qps=40000, order=14}) <br>
newServer({address='192.168.1.2', maxCheckFailures=3, checkInterval=5, weight=4, qps=40000, order=15}) <br>
newServer({address='192.168.1.2', maxCheckFailures=3, checkInterval=5, weight=4, qps=40000, order=16})<br>
<br>
setMaxUDPOutstanding(65535)<br>
<br>
---- Server Load Balancing Policy<br>
setServerPolicy(leastOutstanding)<br>
<br>
---- Web-server<br>
webserver('<a href="http://192.168.0.1:8083" rel="noreferrer" target="_blank">192.168.0.1:8083</a>')<br>
setWebserverConfig({acl='<a href="http://192.168.0.0/24" rel="noreferrer" target="_blank">192.168.0.0/24</a>', password='Secret'})<br>
<br>
---- Customers Policy<br>
customerACLs={'<a href="http://192.168.1.0/24" rel="noreferrer" target="_blank">192.168.1.0/24</a>'}<br>
setACL(customerACLs)<br>
<br>
pc = newPacketCache(300000, {maxTTL=86400, minTTL=0, temporaryFailureTTL=60, staleTTL=60, dontAge=false})<br>
getPool(""):setCache(pc)<br>
<br>
setVerboseHealthChecks(true)<br>
<br>
Servers Specs are as below:<br>
Dnsdist LB Server Specs: 16 vCPUs, 16 GB RAM, Virtio NIC (10G) with 16 Multiqueues.<br>
Backend bind9 servers Specs: 16 vCPUs, 16GM RAM, Virtio NIC (10G) with 16 Multiqueues. <br>
<br>
We are trying to handle 500K qps (will increase hardware specs, If required) or with above specs atleast 100K qps.<br>
<br>
<br>
Regards,<br>
Rais <br>
<br>
-----Original Message-----<br>
From: dnsdist <<a href="mailto:dnsdist-bounces@mailman.powerdns.com" target="_blank">dnsdist-bounces@mailman.powerdns.com</a>> On Behalf Of <a href="mailto:dnsdist-request@mailman.powerdns.com" target="_blank">dnsdist-request@mailman.powerdns.com</a><br>
Sent: Wednesday, March 23, 2022 5:00 PM<br>
To: <a href="mailto:dnsdist@mailman.powerdns.com" target="_blank">dnsdist@mailman.powerdns.com</a><br>
Subject: dnsdist Digest, Vol 79, Issue 3<br>
<br>
Send dnsdist mailing list submissions to<br>
<a href="mailto:dnsdist@mailman.powerdns.com" target="_blank">dnsdist@mailman.powerdns.com</a><br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="https://mailman.powerdns.com/mailman/listinfo/dnsdist" rel="noreferrer" target="_blank">https://mailman.powerdns.com/mailman/listinfo/dnsdist</a><br>
or, via email, send a message with subject or body 'help' to<br>
<a href="mailto:dnsdist-request@mailman.powerdns.com" target="_blank">dnsdist-request@mailman.powerdns.com</a><br>
<br>
You can reach the person managing the list at<br>
<a href="mailto:dnsdist-owner@mailman.powerdns.com" target="_blank">dnsdist-owner@mailman.powerdns.com</a><br>
<br>
When replying, please edit your Subject line so it is more specific than "Re: Contents of dnsdist digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. dnsdist[29321]: Marking downstream IP:53 as 'down' (Rais Ahmed)<br>
2. Re: dnsdist[29321]: Marking downstream IP:53 as 'down'<br>
(Remi Gacogne)<br>
<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Tue, 22 Mar 2022 23:00:25 +0000<br>
From: Rais Ahmed <<a href="mailto:rais.ahmed@tes.com.pk" target="_blank">rais.ahmed@tes.com.pk</a>><br>
To: "<a href="mailto:dnsdist@mailman.powerdns.com" target="_blank">dnsdist@mailman.powerdns.com</a>" <<a href="mailto:dnsdist@mailman.powerdns.com" target="_blank">dnsdist@mailman.powerdns.com</a>><br>
Subject: [dnsdist] dnsdist[29321]: Marking downstream IP:53 as 'down'<br>
Message-ID:<br>
<<a href="mailto:PAXPR08MB70737E4E1CCEFC4A7F61E1E6A0179@PAXPR08MB7073.eurprd08.prod.outlook.com" target="_blank">PAXPR08MB70737E4E1CCEFC4A7F61E1E6A0179@PAXPR08MB7073.eurprd08.prod.outlook.com</a>><br>
<br>
Content-Type: text/plain; charset="us-ascii"<br>
<br>
Hi,<br>
<br>
We have configured dnsdist instance to handle around 500k QPS, but we are seeing downstream down frequently once QPS reached above 25k. below are the logs which we found to relative issue.<br>
<br>
dnsdist[29321]: Marking downstream server1 IP:53 as 'down'<br>
dnsdist[29321]: Marking downstream server2 IP:53 as 'down'<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://mailman.powerdns.com/pipermail/dnsdist/attachments/20220322/2befd6e2/attachment-0001.htm" rel="noreferrer" target="_blank">http://mailman.powerdns.com/pipermail/dnsdist/attachments/20220322/2befd6e2/attachment-0001.htm</a>><br>
<br>
------------------------------<br>
<br>
Message: 2<br>
Date: Wed, 23 Mar 2022 10:32:22 +0100<br>
From: Remi Gacogne <<a href="mailto:remi.gacogne@powerdns.com" target="_blank">remi.gacogne@powerdns.com</a>><br>
To: Rais Ahmed <<a href="mailto:rais.ahmed@tes.com.pk" target="_blank">rais.ahmed@tes.com.pk</a>>, "<a href="mailto:dnsdist@mailman.powerdns.com" target="_blank">dnsdist@mailman.powerdns.com</a>"<br>
<<a href="mailto:dnsdist@mailman.powerdns.com" target="_blank">dnsdist@mailman.powerdns.com</a>><br>
Subject: Re: [dnsdist] dnsdist[29321]: Marking downstream IP:53 as<br>
'down'<br>
Message-ID: <<a href="mailto:5a95cbeb-7c82-9bc1-0b4c-8726f814432e@powerdns.com" target="_blank">5a95cbeb-7c82-9bc1-0b4c-8726f814432e@powerdns.com</a>><br>
Content-Type: text/plain; charset=UTF-8; format=flowed<br>
<br>
Hi,<br>
<br>
> We have configured dnsdist instance to handle around 500k QPS, but we > are seeing downstream down frequently once QPS reached above 25k. below > are the logs which we found to relative issue.<br>
><br>
> dnsdist[29321]: Marking downstream server1 IP:53 as 'down'<br>
><br>
> dnsdist[29321]: Marking downstream server2 IP:53 as 'down'<br>
<br>
You might be able to get more information about why the health-checks are failing by adding setVerboseHealthChecks(true) to your configuration.<br>
<br>
It usually happens because the backend is overwhelmed and needs to be tuned to handle the load, but it might also be caused by a network issue, like a link reaching its maximum capacity, or by dnsdist itself being overwhelmed and needing tuning (like increasing the number of<br>
newServer() directives, see [1]).<br>
<br>
[1]: <br>
<a href="https://dnsdist.org/advanced/tuning.html#udp-and-incoming-dns-over-https" rel="noreferrer" target="_blank">https://dnsdist.org/advanced/tuning.html#udp-and-incoming-dns-over-https</a><br>
<br>
Best regards,<br>
--<br>
Remi Gacogne<br>
PowerDNS.COM BV - <a href="https://www.powerdns.com/" rel="noreferrer" target="_blank">https://www.powerdns.com/</a><br>
<br>
<br>
------------------------------<br>
<br>
Subject: Digest Footer<br>
<br>
_______________________________________________<br>
dnsdist mailing list<br>
<a href="mailto:dnsdist@mailman.powerdns.com" target="_blank">dnsdist@mailman.powerdns.com</a><br>
<a href="https://mailman.powerdns.com/mailman/listinfo/dnsdist" rel="noreferrer" target="_blank">https://mailman.powerdns.com/mailman/listinfo/dnsdist</a><br>
<br>
<br>
------------------------------<br>
<br>
End of dnsdist Digest, Vol 79, Issue 3<br>
**************************************<br>
_______________________________________________<br>
dnsdist mailing list<br>
<a href="mailto:dnsdist@mailman.powerdns.com" target="_blank">dnsdist@mailman.powerdns.com</a><br>
<a href="https://mailman.powerdns.com/mailman/listinfo/dnsdist" rel="noreferrer" target="_blank">https://mailman.powerdns.com/mailman/listinfo/dnsdist</a><br>
</blockquote></div>