[dnsdist] Performance/tuning sanity check

[Klaus Darilion]

In my recent tests (not Dnsdist but Knot) my bottleneck was IRQ handling of NICs. A simple "apt install irqbalance" doubled the QPS. So check if your NIC has multiqueue enabled and those queues/interrupts are all handled by different cores.

If this is a problem you would see it in htop with some CPUs are 100% while others are idle. Also when using htop disable "hide kernel threads".

regards
Klaus

Von: dnsdist <dnsdist-bounces at mailman.powerdns.com> Im Auftrag von Dan McCombs via dnsdist
Gesendet: Donnerstag, 22. Dezember 2022 18:19
An: dnsdist at mailman.powerdns.com
Betreff: [dnsdist] Performance/tuning sanity check

Hi Everyone,

I've been doing some performance testing and tuning of dnsdist and was hoping to get a sanity check that these numbers seem reasonable for the given hardware, or if there's any performance tuning I'm overlooking.

I'm testing against a single 32-core machine (4 x Intel® Xeon® Silver 4110 @2.10GHz) with 96GB of RAM running dnsdist 1.5. I'm running NS1's Flamethrower test from 700 clients querying a single FQDN via UDP with a 300 second TTL for 10 minutes to be almost entirely answering from dnsdist cache. This is querying against 2 addresses that the same dnsdist instance is listening on.

For dnsdist performance tuning, the following things are in place:
* A single dnsdist instance
* One listener on 127.0.0.1:53<http://127.0.0.1:53>
* 7 listeners each using addLocal and reusePort=true on each of the 2 external addresses
* A packet cache with numberOfShards set to 20
* setMaxUDPOutstanding set to 65535
* setRingBuffersSize set to 100000, 20
* setRingBuffersLockRetries set to 5
* setUDPMultipleMessagesVectorSize set to 1024

On the OS level, I can't think of any tuning in place besides netfilter connection tracking being disabled.

With this configuration past 210,000 QPS I start seeing missed responses to queries. If I add some dnsdist rules that drop some queries, I can send more queries and get expected responses for others that are not picked up by the rules. The threshold seems to be around 420,000 QPS combined queries and responses, where past that things start to have issues, whether those are 210k answered queries or 350k queries with only 70k expected to be answered.

Do those numbers seem reasonable for this hardware? I've gone through the performance tuning documentation, but is there anything else I'm missing? Is the number of listeners appropriate for this number of CPU cores with 2 listening addresses?

Thanks for any experience and information,

-Dan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.powerdns.com/pipermail/dnsdist/attachments/20221223/cf1e8816/attachment.htm>