<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#faf8f6" bgcolor="#3e3e3e">
<p>Hello Rais,</p>
<p>i noticed that you are increasing nf_conntrack_max. I am not sure
how the backend servers are connected,</p>
<p>but i suggest not to use connection tracking/NAT at all. You can
use for example dedicated interface for backend</p>
<p>management and other one to connect to dnsdist.</p>
<p>r.<br>
</p>
<div class="moz-cite-prefix">On 24/03/2022 11:11, Rais Ahmed via
dnsdist wrote:<br>
</div>
<blockquote type="cite"
cite="mid:PAXPR08MB70739A6AF3AD06239D4B6632A0199@PAXPR08MB7073.eurprd08.prod.outlook.com">
<pre class="moz-quote-pre" wrap="">Hi,
Thanks for the guidance...!
We are testing with multiple scenarios, with/without kernel tuning. We observed UDP packets errors on both backend servers (not a single UDP error on dnsdist LB server).
Tested with resperf 15K QPS
resperf -s 192.168.0.1 -R -d queryfile-example-10million-201202 -C 100 -c 300 -r 0 -m 15000 -q 200000
Backend 1: 192.168.1.1 (without Kernel tuning):
netstat -su
IcmpMsg:
InType3: 2229
InType8: 6
InType11: 194
OutType0: 6
OutType3: 762
Udp:
1634847 packets received
843 packets to unknown port received.
193891 packet receive errors
1859642 packets sent
193891 receive buffer errors
0 send buffer errors
UdpLite:
IpExt:
InOctets: 580762744
OutOctets: 237368675
InNoECTPkts: 1995692
InECT0Pkts: 27
Backend 2: 192.168.1.2 (with Kernel Tuning):
netstat -su
IcmpMsg:
InType3: 19177
InType8: 5802
InType11: 2645
OutType0: 5802
OutType3: 5122
Udp:
10798358 packets received
6846 packets to unknown port received.
4815377 packet receive errors
11949871 packets sent
4815377 receive buffer errors
0 send buffer errors
UdpLite:
IpExt:
InNoRoutes: 11
InOctets: 3312682950
OutOctets: 1741771756
InNoECTPkts: 16355120
InECT1Pkts: 72
InECT0Pkts: 92
InCEPkts: 4
Kernel Tuning configured in /etc/rc.local
ethtool -L eth0 combined 16
echo 52428800 > /proc/sys/net/netfilter/nf_conntrack_max
sysctl -w net.core.rmem_max=33554432
sysctl -w net.core.wmem_max=33554432
sysctl -w net.core.rmem_default=16777216
sysctl -w net.core.wmem_default=16777216
sysctl -w net.core.netdev_max_backlog=65536
sysctl -w net.core.somaxconn=1024
ulimit -n 16000
Network config/ specs are same on all three servers, are we doing something wrong?
Regards,
Rais
-----Original Message-----
From: Klaus Darilion <a class="moz-txt-link-rfc2396E" href="mailto:klaus.darilion@nic.at"><klaus.darilion@nic.at></a>
Sent: Thursday, March 24, 2022 12:38 PM
To: Rais Ahmed <a class="moz-txt-link-rfc2396E" href="mailto:rais.ahmed@tes.com.pk"><rais.ahmed@tes.com.pk></a>; <a class="moz-txt-link-abbreviated" href="mailto:dnsdist@mailman.powerdns.com">dnsdist@mailman.powerdns.com</a>
Subject: AW: [dnsdist] dnsdist[29321]: Marking downstream IP:53 as 'down'
Have you tested how many Qps your Backend is capably to handle? First test your Backend performance to know how much qps a single backend can handle. I guess 500k qps might be difficult to achieve with bind. If you need more performance switch the Backend to NSD or Knot.
regards
Klaus
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">-----Ursprüngliche Nachricht-----
Von: dnsdist <a class="moz-txt-link-rfc2396E" href="mailto:dnsdist-bounces@mailman.powerdns.com"><dnsdist-bounces@mailman.powerdns.com></a> Im Auftrag von
Rais Ahmed via dnsdist
Gesendet: Mittwoch, 23. März 2022 22:02
An: <a class="moz-txt-link-abbreviated" href="mailto:dnsdist@mailman.powerdns.com">dnsdist@mailman.powerdns.com</a>
Betreff: [dnsdist] dnsdist[29321]: Marking downstream IP:53 as 'down'
Hi,
Thanks for reply...!
We have configured setMaxUDPOutstanding(65535) and still we are seeing
backend down, logs are showing frequently as below.
Timeout while waiting for the health check response from backend
192.168.1.1:53
Timeout while waiting for the health check response from backend
192.168.1.2:53
Please have a look at below dnsdist configuration and help us to find
misconfiguration (16 Listeners & 8+8 backends added as per vCPUs
available
(2 Socket x 8 Cores):
controlSocket('127.0.0.1:5199')
setKey("")
---- Listen addresses
addLocal('192.168.0.1:53', { reusePort=true })
addLocal('192.168.0.1:53', { reusePort=true })
addLocal('192.168.0.1:53', { reusePort=true })
addLocal('192.168.0.1:53', { reusePort=true })
addLocal('192.168.0.1:53', { reusePort=true })
addLocal('192.168.0.1:53', { reusePort=true })
addLocal('192.168.0.1:53', { reusePort=true })
addLocal('192.168.0.1:53', { reusePort=true })
addLocal('192.168.0.1:53', { reusePort=true })
addLocal('192.168.0.1:53', { reusePort=true })
addLocal('192.168.0.1:53', { reusePort=true })
addLocal('192.168.0.1:53', { reusePort=true })
addLocal('192.168.0.1:53', { reusePort=true })
addLocal('192.168.0.1:53', { reusePort=true })
addLocal('192.168.0.1:53', { reusePort=true })
addLocal('192.168.0.1:53', { reusePort=true })
---- Back-end server
newServer({address='192.168.1.1', maxCheckFailures=3, checkInterval=5,
weight=4, qps=40000, order=1}) newServer({address='192.168.1.1',
maxCheckFailures=3, checkInterval=5, weight=4, qps=40000, order=2})
newServer({address='192.168.1.1', maxCheckFailures=3, checkInterval=5,
weight=4, qps=40000, order=3}) newServer({address='192.168.1.1',
maxCheckFailures=3, checkInterval=5, weight=4, qps=40000, order=4})
newServer({address='192.168.1.1', maxCheckFailures=3, checkInterval=5,
weight=4, qps=40000, order=5}) newServer({address='192.168.1.1',
maxCheckFailures=3, checkInterval=5, weight=4, qps=40000, order=6})
newServer({address='192.168.1.1', maxCheckFailures=3, checkInterval=5,
weight=4, qps=40000, order=7}) newServer({address='192.168.1.1',
maxCheckFailures=3, checkInterval=5, weight=4, qps=40000, order=8})
newServer({address='192.168.1.2', maxCheckFailures=3, checkInterval=5,
weight=4, qps=40000, order=9}) newServer({address='192.168.1.2',
maxCheckFailures=3, checkInterval=5, weight=4, qps=40000, order=10})
newServer({address='192.168.1.2', maxCheckFailures=3, checkInterval=5,
weight=4, qps=40000, order=11}) newServer({address='192.168.1.2',
maxCheckFailures=3, checkInterval=5, weight=4, qps=40000, order=12})
newServer({address='192.168.1.2', maxCheckFailures=3, checkInterval=5,
weight=4, qps=40000, order=13}) newServer({address='192.168.1.2',
maxCheckFailures=3, checkInterval=5, weight=4, qps=40000, order=14})
newServer({address='192.168.1.2', maxCheckFailures=3, checkInterval=5,
weight=4, qps=40000, order=15}) newServer({address='192.168.1.2',
maxCheckFailures=3, checkInterval=5, weight=4, qps=40000, order=16})
setMaxUDPOutstanding(65535)
---- Server Load Balancing Policy
setServerPolicy(leastOutstanding)
---- Web-server
webserver('192.168.0.1:8083')
setWebserverConfig({acl='192.168.0.0/24', password='Secret'})
---- Customers Policy
customerACLs={'192.168.1.0/24'}
setACL(customerACLs)
pc = newPacketCache(300000, {maxTTL=86400, minTTL=0,
temporaryFailureTTL=60, staleTTL=60, dontAge=false})
getPool(""):setCache(pc)
setVerboseHealthChecks(true)
Servers Specs are as below:
Dnsdist LB Server Specs: 16 vCPUs, 16 GB RAM, Virtio NIC (10G) with 16
Multiqueues.
Backend bind9 servers Specs: 16 vCPUs, 16GM RAM, Virtio NIC (10G) with
16 Multiqueues.
We are trying to handle 500K qps (will increase hardware specs, If
required) or with above specs atleast 100K qps.
Regards,
Rais
-----Original Message-----
From: dnsdist <a class="moz-txt-link-rfc2396E" href="mailto:dnsdist-bounces@mailman.powerdns.com"><dnsdist-bounces@mailman.powerdns.com></a> On Behalf Of
<a class="moz-txt-link-abbreviated" href="mailto:dnsdist-request@mailman.powerdns.com">dnsdist-request@mailman.powerdns.com</a>
Sent: Wednesday, March 23, 2022 5:00 PM
To: <a class="moz-txt-link-abbreviated" href="mailto:dnsdist@mailman.powerdns.com">dnsdist@mailman.powerdns.com</a>
Subject: dnsdist Digest, Vol 79, Issue 3
Send dnsdist mailing list submissions to
<a class="moz-txt-link-abbreviated" href="mailto:dnsdist@mailman.powerdns.com">dnsdist@mailman.powerdns.com</a>
To subscribe or unsubscribe via the World Wide Web, visit
<a class="moz-txt-link-freetext" href="https://mailman.powerdns.com/mailman/listinfo/dnsdist">https://mailman.powerdns.com/mailman/listinfo/dnsdist</a>
or, via email, send a message with subject or body 'help' to
<a class="moz-txt-link-abbreviated" href="mailto:dnsdist-request@mailman.powerdns.com">dnsdist-request@mailman.powerdns.com</a>
You can reach the person managing the list at
<a class="moz-txt-link-abbreviated" href="mailto:dnsdist-owner@mailman.powerdns.com">dnsdist-owner@mailman.powerdns.com</a>
When replying, please edit your Subject line so it is more specific than "Re:
Contents of dnsdist digest..."
Today's Topics:
1. dnsdist[29321]: Marking downstream IP:53 as 'down' (Rais Ahmed)
2. Re: dnsdist[29321]: Marking downstream IP:53 as 'down'
(Remi Gacogne)
----------------------------------------------------------------------
Message: 1
Date: Tue, 22 Mar 2022 23:00:25 +0000
From: Rais Ahmed <a class="moz-txt-link-rfc2396E" href="mailto:rais.ahmed@tes.com.pk"><rais.ahmed@tes.com.pk></a>
To: <a class="moz-txt-link-rfc2396E" href="mailto:dnsdist@mailman.powerdns.com">"dnsdist@mailman.powerdns.com"</a> <a class="moz-txt-link-rfc2396E" href="mailto:dnsdist@mailman.powerdns.com"><dnsdist@mailman.powerdns.com></a>
Subject: [dnsdist] dnsdist[29321]: Marking downstream IP:53 as 'down'
Message-ID:
<a class="moz-txt-link-rfc2396E" href="mailto:PAXPR08MB70737E4E1CCEFC4A7F61E1E6A0179@PAXPR08MB7073.eurprd08.prod.outlook.com"><PAXPR08MB70737E4E1CCEFC4A7F61E1E6A0179@PAXPR08MB7073.e
urprd08.prod.outlook.com></a>
Content-Type: text/plain; charset="us-ascii"
Hi,
We have configured dnsdist instance to handle around 500k QPS, but we
are seeing downstream down frequently once QPS reached above 25k.
below are the logs which we found to relative issue.
dnsdist[29321]: Marking downstream server1 IP:53 as 'down'
dnsdist[29321]: Marking downstream server2 IP:53 as 'down'
-------------- next part -------------- An HTML attachment was
scrubbed...
URL:
<a class="moz-txt-link-rfc2396E" href="http://mailman.powerdns.com/pipermail/dnsdist/attachments/20220322/2befd6e2/attachment-0001.htm"><http://mailman.powerdns.com/pipermail/dnsdist/attachments/20220322/2
befd6e2/attachment-0001.htm></a>
------------------------------
Message: 2
Date: Wed, 23 Mar 2022 10:32:22 +0100
From: Remi Gacogne <a class="moz-txt-link-rfc2396E" href="mailto:remi.gacogne@powerdns.com"><remi.gacogne@powerdns.com></a>
To: Rais Ahmed <a class="moz-txt-link-rfc2396E" href="mailto:rais.ahmed@tes.com.pk"><rais.ahmed@tes.com.pk></a>, <a class="moz-txt-link-rfc2396E" href="mailto:dnsdist@mailman.powerdns.com">"dnsdist@mailman.powerdns.com"</a>
<a class="moz-txt-link-rfc2396E" href="mailto:dnsdist@mailman.powerdns.com"><dnsdist@mailman.powerdns.com></a>
Subject: Re: [dnsdist] dnsdist[29321]: Marking downstream IP:53 as
'down'
Message-ID: <a class="moz-txt-link-rfc2396E" href="mailto:5a95cbeb-7c82-9bc1-0b4c-8726f814432e@powerdns.com"><5a95cbeb-7c82-9bc1-0b4c-8726f814432e@powerdns.com></a>
Content-Type: text/plain; charset=UTF-8; format=flowed
Hi,
> We have configured dnsdist instance to handle around 500k QPS, but
we > are seeing downstream down frequently once QPS reached above 25k.
below > are the logs which we found to relative issue.
>
> dnsdist[29321]: Marking downstream server1 IP:53 as 'down'
>
> dnsdist[29321]: Marking downstream server2 IP:53 as 'down'
You might be able to get more information about why the health-checks
are failing by adding setVerboseHealthChecks(true) to your configuration.
It usually happens because the backend is overwhelmed and needs to be
tuned to handle the load, but it might also be caused by a network
issue, like a link reaching its maximum capacity, or by dnsdist itself
being overwhelmed and needing tuning (like increasing the number of
newServer() directives, see [1]).
[1]:
<a class="moz-txt-link-freetext" href="https://dnsdist.org/advanced/tuning.html#udp-and-incoming-dns-over">https://dnsdist.org/advanced/tuning.html#udp-and-incoming-dns-over</a>-
https
Best regards,
--
Remi Gacogne
PowerDNS.COM BV - <a class="moz-txt-link-freetext" href="https://www.powerdns.com/">https://www.powerdns.com/</a>
------------------------------
Subject: Digest Footer
_______________________________________________
dnsdist mailing list
<a class="moz-txt-link-abbreviated" href="mailto:dnsdist@mailman.powerdns.com">dnsdist@mailman.powerdns.com</a>
<a class="moz-txt-link-freetext" href="https://mailman.powerdns.com/mailman/listinfo/dnsdist">https://mailman.powerdns.com/mailman/listinfo/dnsdist</a>
------------------------------
End of dnsdist Digest, Vol 79, Issue 3
**************************************
_______________________________________________
dnsdist mailing list
<a class="moz-txt-link-abbreviated" href="mailto:dnsdist@mailman.powerdns.com">dnsdist@mailman.powerdns.com</a>
<a class="moz-txt-link-freetext" href="https://mailman.powerdns.com/mailman/listinfo/dnsdist">https://mailman.powerdns.com/mailman/listinfo/dnsdist</a>
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">_______________________________________________
dnsdist mailing list
<a class="moz-txt-link-abbreviated" href="mailto:dnsdist@mailman.powerdns.com">dnsdist@mailman.powerdns.com</a>
<a class="moz-txt-link-freetext" href="https://mailman.powerdns.com/mailman/listinfo/dnsdist">https://mailman.powerdns.com/mailman/listinfo/dnsdist</a>
</pre>
</blockquote>
</body>
</html>