<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.E-MailFormatvorlage18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 2.0cm 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="DE-AT" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">In my recent tests (not Dnsdist but Knot) my bottleneck was IRQ handling of NICs. A simple "apt install irqbalance" doubled the QPS.
So check if your NIC has multiqueue enabled and those queues/interrupts are all handled by different cores.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">If this is a problem you would see it in htop with some CPUs are 100% while others are idle. Also when using htop disable "hide kernel
threads".<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">regards<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Klaus
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="DE" style="font-size:11.0pt;font-family:"Calibri",sans-serif">Von:</span></b><span lang="DE" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> dnsdist <dnsdist-bounces@mailman.powerdns.com>
<b>Im Auftrag von </b>Dan McCombs via dnsdist<br>
<b>Gesendet:</b> Donnerstag, 22. Dezember 2022 18:19<br>
<b>An:</b> dnsdist@mailman.powerdns.com<br>
<b>Betreff:</b> [dnsdist] Performance/tuning sanity check<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Hi Everyone,<br>
<br>
I've been doing some performance testing and tuning of dnsdist and was hoping to get a sanity check that these numbers seem reasonable for the given hardware, or if there's any performance tuning I'm overlooking.<br>
<br>
I'm testing against a single 32-core machine (4 x Intel® Xeon® Silver 4110 @2.10GHz) with 96GB of RAM running dnsdist 1.5. I'm running NS1's Flamethrower test from 700 clients querying a single FQDN via UDP with a 300 second TTL for 10 minutes to be almost
entirely answering from dnsdist cache. This is querying against 2 addresses that the same dnsdist instance is listening on.<br>
<br>
For dnsdist performance tuning, the following things are in place:<br>
* A single dnsdist instance<br>
* One listener on <a href="http://127.0.0.1:53">127.0.0.1:53</a><br>
* 7 listeners each using addLocal and reusePort=true on each of the 2 external addresses<br>
* A packet cache with numberOfShards set to 20<br>
* setMaxUDPOutstanding set to 65535<br>
* setRingBuffersSize set to 100000, 20<br>
* setRingBuffersLockRetries set to 5<br>
* setUDPMultipleMessagesVectorSize set to 1024<br>
<br>
On the OS level, I can't think of any tuning in place besides netfilter connection tracking being disabled.<br>
<br>
With this configuration past 210,000 QPS I start seeing missed responses to queries. If I add some dnsdist rules that drop some queries, I can send more queries and get expected responses for others that are not picked up by the rules. The threshold seems to
be around 420,000 QPS combined queries and responses, where past that things start to have issues, whether those are 210k answered queries or 350k queries with only 70k expected to be answered.<br>
<br>
Do those numbers seem reasonable for this hardware? I've gone through the performance tuning documentation, but is there anything else I'm missing? Is the number of listeners appropriate for this number of CPU cores with 2 listening addresses?<br>
<br>
Thanks for any experience and information,<br>
<br>
-Dan<o:p></o:p></p>
</div>
</div>
</div>
</body>
</html>