[dnsdist] Performance/tuning sanity check

[Dan McCombs]

Thanks for the response Klaus.

I don't think that's an issue in my case, as it looks like multiqueue is
enabled (I see 32 numbered queues in ethtool -S <interface>) and traffic is
pretty well balanced across all 32 cores in htop with kernel threads
visible.

-Dan

On Fri, Dec 23, 2022 at 2:50 AM Klaus Darilion <klaus.darilion at nic.at>
wrote:

> In my recent tests (not Dnsdist but Knot) my bottleneck was IRQ handling
> of NICs. A simple "apt install irqbalance" doubled the QPS. So check if
> your NIC has multiqueue enabled and those queues/interrupts are all handled
> by different cores.
>
>
>
> If this is a problem you would see it in htop with some CPUs are 100%
> while others are idle. Also when using htop disable "hide kernel threads".
>
>
>
> regards
>
> Klaus
>
>
>
> *Von:* dnsdist <dnsdist-bounces at mailman.powerdns.com> *Im Auftrag von *Dan
> McCombs via dnsdist
> *Gesendet:* Donnerstag, 22. Dezember 2022 18:19
> *An:* dnsdist at mailman.powerdns.com
> *Betreff:* [dnsdist] Performance/tuning sanity check
>
>
>
> Hi Everyone,
>
> I've been doing some performance testing and tuning of dnsdist and was
> hoping to get a sanity check that these numbers seem reasonable for the
> given hardware, or if there's any performance tuning I'm overlooking.
>
> I'm testing against a single 32-core machine (4 x Intel® Xeon® Silver
> 4110 @2.10GHz) with 96GB of RAM running dnsdist 1.5. I'm running NS1's
> Flamethrower test from 700 clients querying a single FQDN via UDP with a
> 300 second TTL for 10 minutes to be almost entirely answering from dnsdist
> cache. This is querying against 2 addresses that the same dnsdist instance
> is listening on.
>
> For dnsdist performance tuning, the following things are in place:
> * A single dnsdist instance
> * One listener on 127.0.0.1:53
> * 7 listeners each using addLocal and reusePort=true on each of the 2
> external addresses
> * A packet cache with numberOfShards set to 20
> * setMaxUDPOutstanding set to 65535
> * setRingBuffersSize set to 100000, 20
> * setRingBuffersLockRetries set to 5
> * setUDPMultipleMessagesVectorSize set to 1024
>
> On the OS level, I can't think of any tuning in place besides netfilter
> connection tracking being disabled.
>
> With this configuration past 210,000 QPS I start seeing missed responses
> to queries. If I add some dnsdist rules that drop some queries, I can send
> more queries and get expected responses for others that are not picked up
> by the rules. The threshold seems to be around 420,000 QPS combined queries
> and responses, where past that things start to have issues, whether those
> are 210k answered queries or 350k queries with only 70k expected to be
> answered.
>
> Do those numbers seem reasonable for this hardware? I've gone through the
> performance tuning documentation, but is there anything else I'm missing?
> Is the number of listeners appropriate for this number of CPU cores with 2
> listening addresses?
>
> Thanks for any experience and information,
>
> -Dan
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.powerdns.com/pipermail/dnsdist/attachments/20230103/d3ecf093/attachment.htm>