<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Working configuration with small cache and threaded for 4 cores.. for reference (v6 is an afterthought)</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
-- DNSdist ns1
<div>-- Mike Willis 2-17-2022</div>
<div><br>
</div>
<div>-- Basic rules</div>
<div>-- If client passes base ACL and dynamic blocks/rate limits</div>
<div>-- Deliver traffic to NS1 if it is up</div>
<div>-- If not deliver traffic to NS2</div>
<div><br>
</div>
<div>----Set encrypton key for console-----------------------</div>
<div>setKey("redacted")</div>
<div>controlSocket('127.0.0.1:5199')</div>
<div><br>
</div>
<div>----Webserver/API config-------------------------------</div>
<div>---ACL is NMS OAM Natted Egress</div>
<div>webserver("x.x.x.237:8083")</div>
<div>setWebserverConfig({password="redacted", apiKey="redacted", acl="x.x.x.x.x/32"})</div>
<div><br>
</div>
<div>----- Local binds for DNS and ACLs --------------------</div>
<div>--ns1 public vip</div>
<div>addLocal("x.x.x.x:53", {reusePort=true})</div>
<div>addLocal("x.x.x.x:53", {reusePort=true})</div>
<div>addLocal("x.x.x.x:53", {reusePort=true})</div>
<div>addLocal("x.x.x.x:53", {reusePort=true})</div>
<div>--ns1 public</div>
<div>addLocal("x.x.x.x:53")</div>
<div>--ns1 V6 vip</div>
<div>addLocal("x:x::x:10")</div>
<div>--ns1 V6 Local</div>
<div>addLocal("x:x:x:x::1101")</div>
<div>--dnsdist01 V4 DMZ</div>
<div>addLocal("10.50.50.41:53")</div>
<div>-------------------------------------------------------</div>
<div>--ACLs HERE</div>
<div>setACL({'0.0.0.0/0', '::/0'})</div>
<div><br>
</div>
<div>-----------------------------------------------------</div>
<div>---Performance Tuning -------------------------------</div>
<div>--setRingBuffersSize(num[, numberOfShards])</div>
<div>setRingBuffersSize(500000,10)</div>
<div><br>
</div>
<div>--packet cache</div>
<div>pc = newPacketCache(10000, {maxTTL=14400, minTTL=900, temporaryFailureTTL=60, staleTTL=60, dontAge=false})</div>
<div>getPool("ns1"):setCache(pc)</div>
<div><br>
</div>
<div>pc = newPacketCache(10000, {maxTTL=14400, minTTL=900, temporaryFailureTTL=60, staleTTL=60, dontAge=false})</div>
<div>getPool("ns2"):setCache(pc)</div>
<div><br>
</div>
<div>----Dynamic blocking rules to mitigate abuse -----</div>
<div>--Note: For testing purposes, argument 2/time interval usually needs to elapse before the rule is evaluated and applied</div>
<div>--From the console ("dnsdist -c 127.0.0.1:5199")</div>
<div>--Run "showDynBlocks()" to show the active blocks</div>
<div>--Run "clearDynBlocks()" to clear all blocks</div>
<div><br>
</div>
<div>local dbr = dynBlockRulesGroup()</div>
<div>dbr:setQueryRate(1000, 10, "Exceeded query rate", 60)</div>
<div>dbr:setRCodeRate(DNSRCode.NXDOMAIN, 400, 10, "Exceeded NXD rate", 60)</div>
<div>dbr:setRCodeRate(DNSRCode.SERVFAIL, 500, 10, "Exceeded ServFail rate", 120)</div>
<div>dbr:setQTypeRate(DNSQType.ANY, 100, 10, "Exceeded ANY rate", 120)</div>
<div>dbr:setResponseByteRate(500000, 10, "Exceeded resp BW rate", 60)</div>
<div><br>
</div>
<div> function maintenance()</div>
<div> dbr:apply()</div>
<div> end</div>
<div><br>
</div>
<div>--Enable this if you want to give a response</div>
<div>--setDynBlocksAction(DNSAction.Refused)</div>
<div><br>
</div>
<div>--NOTE: Rules are processed in order, and some rules stop processing of additional rules</div>
<div>--IE: Some rules should be the last to run for a given flow.</div>
<div><br>
</div>
<div><br>
</div>
<div>----------------- Logging ----------------------</div>
<div>---This should be turned off in prod ---</div>
<div>--LogAction([filename[, binary[, append[, buffered[, verboseOnly[, includeTimestamp]]]]]])</div>
<div>--Note will not work if buffering is true</div>
<div>--addAction(AllRule(), LogAction("/var/log/dnsdist.log", false, true, false, false, true))</div>
<div>-------------------------------------------------</div>
<div><br>
</div>
<div>----- Pool Availability rules and failover ------</div>
<div>--Send traffic to ns1 if it is up</div>
<div>addAction(PoolAvailableRule("ns1"), PoolAction("ns1"))</div>
<div>--Send traffic to ns2 if ns1 is down</div>
<div>addAction(AllRule(), PoolAction("ns2"))</div>
<div>------------------------------------------------</div>
<div><br>
</div>
<div><br>
</div>
<div>----------- Load balanced servers and pool definitions ---------------</div>
<div>--ns1 infoblox</div>
<div><br>
</div>
<div>newServer({address="10.50.50.36", source="ens192", pool="ns1", checkType="A", checkType=DNSClass.IN, checkName="www.x.net.", mustResolve=true})</div>
<div><br>
</div>
<div>--ns2 infoblox</div>
<div>newServer({address="10.50.51.52", source="ens192", pool="ns2", checkType="A", checkType=DNSClass.IN, checkName="www.x.net.", mustResolve=true})</div>
<div><br>
</div>
<div>---------------------------------------------------------------------</div>
<span></span><br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Willis, Michael <Michael.Willis@cinbell.com><br>
<b>Sent:</b> Wednesday, March 23, 2022 5:56 PM<br>
<b>To:</b> dnsdist@mailman.powerdns.com <dnsdist@mailman.powerdns.com>; Rais Ahmed <rais.ahmed@tes.com.pk>; Willis, Michael <Michael.Willis@cinbell.com><br>
<b>Subject:</b> Re: [dnsdist] dnsdist[29321]: Marking downstream IP:53 as 'down'</font>
<div> </div>
</div>
<style type="text/css" style="display:none">
<!--
p
{margin-top:0;
margin-bottom:0}
-->
</style>
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
OP,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I'm not an expert here by any sense...</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
In my humble opinion, you should be using pools and dynamic rules for the pools..</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Do you have the ability to do Anycast via BGP and "ExaBGP"?... It works really well.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
-yngmike</div>
<div id="x_appendonsend"></div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> dnsdist <dnsdist-bounces@mailman.powerdns.com> on behalf of Willis, Michael via dnsdist <dnsdist@mailman.powerdns.com><br>
<b>Sent:</b> Wednesday, March 23, 2022 5:27 PM<br>
<b>To:</b> dnsdist@mailman.powerdns.com <dnsdist@mailman.powerdns.com>; Rais Ahmed <rais.ahmed@tes.com.pk><br>
<b>Subject:</b> Re: [dnsdist] dnsdist[29321]: Marking downstream IP:53 as 'down'</font>
<div> </div>
</div>
<style type="text/css" style="display:none">
<!--
p
{margin-top:0;
margin-bottom:0}
-->
</style>
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Bump...</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I am personally interested in this as well. I'm trying to something similar albeit on a smaller scale.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
-Mike</div>
<div id="x_x_appendonsend"></div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> dnsdist <dnsdist-bounces@mailman.powerdns.com> on behalf of Rais Ahmed via dnsdist <dnsdist@mailman.powerdns.com><br>
<b>Sent:</b> Wednesday, March 23, 2022 5:01 PM<br>
<b>To:</b> dnsdist@mailman.powerdns.com <dnsdist@mailman.powerdns.com><br>
<b>Subject:</b> [dnsdist] dnsdist[29321]: Marking downstream IP:53 as 'down'</font>
<div> </div>
</div>
<div class="x_x_BodyFragment"><font size="2"><span style="font-size:11pt">
<div class="x_x_PlainText">Hi,<br>
Thanks for reply...!<br>
<br>
We have configured setMaxUDPOutstanding(65535) and still we are seeing backend down, logs are showing frequently as below.<br>
<br>
Timeout while waiting for the health check response from backend 192.168.1.1:53<br>
Timeout while waiting for the health check response from backend 192.168.1.2:53<br>
<br>
Please have a look at below dnsdist configuration and help us to find misconfiguration (16 Listeners & 8+8 backends added as per vCPUs available (2 Socket x 8 Cores):<br>
<br>
controlSocket('127.0.0.1:5199')<br>
setKey("")<br>
<br>
---- Listen addresses<br>
addLocal('192.168.0.1:53', { reusePort=true })<br>
addLocal('192.168.0.1:53', { reusePort=true })<br>
addLocal('192.168.0.1:53', { reusePort=true })<br>
addLocal('192.168.0.1:53', { reusePort=true })<br>
addLocal('192.168.0.1:53', { reusePort=true })<br>
addLocal('192.168.0.1:53', { reusePort=true })<br>
addLocal('192.168.0.1:53', { reusePort=true })<br>
addLocal('192.168.0.1:53', { reusePort=true })<br>
addLocal('192.168.0.1:53', { reusePort=true })<br>
addLocal('192.168.0.1:53', { reusePort=true })<br>
addLocal('192.168.0.1:53', { reusePort=true })<br>
addLocal('192.168.0.1:53', { reusePort=true })<br>
addLocal('192.168.0.1:53', { reusePort=true })<br>
addLocal('192.168.0.1:53', { reusePort=true })<br>
addLocal('192.168.0.1:53', { reusePort=true })<br>
addLocal('192.168.0.1:53', { reusePort=true })<br>
<br>
---- Back-end server<br>
newServer({address='192.168.1.1', maxCheckFailures=3, checkInterval=5, weight=4, qps=40000, order=1})<br>
newServer({address='192.168.1.1', maxCheckFailures=3, checkInterval=5, weight=4, qps=40000, order=2})<br>
newServer({address='192.168.1.1', maxCheckFailures=3, checkInterval=5, weight=4, qps=40000, order=3})<br>
newServer({address='192.168.1.1', maxCheckFailures=3, checkInterval=5, weight=4, qps=40000, order=4})<br>
newServer({address='192.168.1.1', maxCheckFailures=3, checkInterval=5, weight=4, qps=40000, order=5})<br>
newServer({address='192.168.1.1', maxCheckFailures=3, checkInterval=5, weight=4, qps=40000, order=6})<br>
newServer({address='192.168.1.1', maxCheckFailures=3, checkInterval=5, weight=4, qps=40000, order=7})<br>
newServer({address='192.168.1.1', maxCheckFailures=3, checkInterval=5, weight=4, qps=40000, order=8})<br>
newServer({address='192.168.1.2', maxCheckFailures=3, checkInterval=5, weight=4, qps=40000, order=9})<br>
newServer({address='192.168.1.2', maxCheckFailures=3, checkInterval=5, weight=4, qps=40000, order=10})<br>
newServer({address='192.168.1.2', maxCheckFailures=3, checkInterval=5, weight=4, qps=40000, order=11})<br>
newServer({address='192.168.1.2', maxCheckFailures=3, checkInterval=5, weight=4, qps=40000, order=12})<br>
newServer({address='192.168.1.2', maxCheckFailures=3, checkInterval=5, weight=4, qps=40000, order=13})<br>
newServer({address='192.168.1.2', maxCheckFailures=3, checkInterval=5, weight=4, qps=40000, order=14})<br>
newServer({address='192.168.1.2', maxCheckFailures=3, checkInterval=5, weight=4, qps=40000, order=15})<br>
newServer({address='192.168.1.2', maxCheckFailures=3, checkInterval=5, weight=4, qps=40000, order=16})<br>
<br>
setMaxUDPOutstanding(65535)<br>
<br>
---- Server Load Balancing Policy<br>
setServerPolicy(leastOutstanding)<br>
<br>
---- Web-server<br>
webserver('192.168.0.1:8083')<br>
setWebserverConfig({acl='192.168.0.0/24', password='Secret'})<br>
<br>
---- Customers Policy<br>
customerACLs={'192.168.1.0/24'}<br>
setACL(customerACLs)<br>
<br>
pc = newPacketCache(300000, {maxTTL=86400, minTTL=0,<br>
temporaryFailureTTL=60, staleTTL=60, dontAge=false})<br>
getPool(""):setCache(pc)<br>
<br>
setVerboseHealthChecks(true)<br>
<br>
Servers Specs are as below:<br>
Dnsdist LB Server Specs: 16 vCPUs, 16 GB RAM, Virtio NIC (10G) with 16 Multiqueues.<br>
Backend bind9 servers Specs: 16 vCPUs, 16GM RAM, Virtio NIC (10G) with 16 Multiqueues.
<br>
<br>
We are trying to handle 500K qps (will increase hardware specs, If required) or with above specs atleast 100K qps.<br>
<br>
<br>
Regards,<br>
Rais <br>
<br>
-----Original Message-----<br>
From: dnsdist <dnsdist-bounces@mailman.powerdns.com> On Behalf Of dnsdist-request@mailman.powerdns.com<br>
Sent: Wednesday, March 23, 2022 5:00 PM<br>
To: dnsdist@mailman.powerdns.com<br>
Subject: dnsdist Digest, Vol 79, Issue 3<br>
<br>
Send dnsdist mailing list submissions to<br>
dnsdist@mailman.powerdns.com<br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="https://mailman.powerdns.com/mailman/listinfo/dnsdist">https://mailman.powerdns.com/mailman/listinfo/dnsdist</a><br>
or, via email, send a message with subject or body 'help' to<br>
dnsdist-request@mailman.powerdns.com<br>
<br>
You can reach the person managing the list at<br>
dnsdist-owner@mailman.powerdns.com<br>
<br>
When replying, please edit your Subject line so it is more specific than "Re: Contents of dnsdist digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. dnsdist[29321]: Marking downstream IP:53 as 'down' (Rais Ahmed)<br>
2. Re: dnsdist[29321]: Marking downstream IP:53 as 'down'<br>
(Remi Gacogne)<br>
<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Tue, 22 Mar 2022 23:00:25 +0000<br>
From: Rais Ahmed <rais.ahmed@tes.com.pk><br>
To: "dnsdist@mailman.powerdns.com" <dnsdist@mailman.powerdns.com><br>
Subject: [dnsdist] dnsdist[29321]: Marking downstream IP:53 as 'down'<br>
Message-ID:<br>
<PAXPR08MB70737E4E1CCEFC4A7F61E1E6A0179@PAXPR08MB7073.eurprd08.prod.outlook.com><br>
<br>
Content-Type: text/plain; charset="us-ascii"<br>
<br>
Hi,<br>
<br>
We have configured dnsdist instance to handle around 500k QPS, but we are seeing downstream down frequently once QPS reached above 25k. below are the logs which we found to relative issue.<br>
<br>
dnsdist[29321]: Marking downstream server1 IP:53 as 'down'<br>
dnsdist[29321]: Marking downstream server2 IP:53 as 'down'<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://mailman.powerdns.com/pipermail/dnsdist/attachments/20220322/2befd6e2/attachment-0001.htm">http://mailman.powerdns.com/pipermail/dnsdist/attachments/20220322/2befd6e2/attachment-0001.htm</a>><br>
<br>
------------------------------<br>
<br>
Message: 2<br>
Date: Wed, 23 Mar 2022 10:32:22 +0100<br>
From: Remi Gacogne <remi.gacogne@powerdns.com><br>
To: Rais Ahmed <rais.ahmed@tes.com.pk>, "dnsdist@mailman.powerdns.com"<br>
<dnsdist@mailman.powerdns.com><br>
Subject: Re: [dnsdist] dnsdist[29321]: Marking downstream IP:53 as<br>
'down'<br>
Message-ID: <5a95cbeb-7c82-9bc1-0b4c-8726f814432e@powerdns.com><br>
Content-Type: text/plain; charset=UTF-8; format=flowed<br>
<br>
Hi,<br>
<br>
> We have configured dnsdist instance to handle around 500k QPS, but we > are seeing downstream down frequently once QPS reached above 25k. below > are the logs which we found to relative issue.<br>
><br>
> dnsdist[29321]: Marking downstream server1 IP:53 as 'down'<br>
><br>
> dnsdist[29321]: Marking downstream server2 IP:53 as 'down'<br>
<br>
You might be able to get more information about why the health-checks are failing by adding setVerboseHealthChecks(true) to your configuration.<br>
<br>
It usually happens because the backend is overwhelmed and needs to be tuned to handle the load, but it might also be caused by a network issue, like a link reaching its maximum capacity, or by dnsdist itself being overwhelmed and needing tuning (like increasing
the number of<br>
newServer() directives, see [1]).<br>
<br>
[1]: <br>
<a href="https://dnsdist.org/advanced/tuning.html#udp-and-incoming-dns-over-https">https://dnsdist.org/advanced/tuning.html#udp-and-incoming-dns-over-https</a><br>
<br>
Best regards,<br>
--<br>
Remi Gacogne<br>
PowerDNS.COM BV - <a href="https://www.powerdns.com/">https://www.powerdns.com/</a><br>
<br>
<br>
------------------------------<br>
<br>
Subject: Digest Footer<br>
<br>
_______________________________________________<br>
dnsdist mailing list<br>
dnsdist@mailman.powerdns.com<br>
<a href="https://mailman.powerdns.com/mailman/listinfo/dnsdist">https://mailman.powerdns.com/mailman/listinfo/dnsdist</a><br>
<br>
<br>
------------------------------<br>
<br>
End of dnsdist Digest, Vol 79, Issue 3<br>
**************************************<br>
_______________________________________________<br>
dnsdist mailing list<br>
dnsdist@mailman.powerdns.com<br>
<a href="https://mailman.powerdns.com/mailman/listinfo/dnsdist">https://mailman.powerdns.com/mailman/listinfo/dnsdist</a><br>
</div>
</span></font></div>
</div>
</div>
</body>
</html>