[dnsdist] Tweaking kernel paramaters for heavy loaded site with dnsdist

[Aleš Rygl]

Hi Remi,

I should have mentioned the hw and os spec at the benining; so:
Server HP Proliant DL360 G6, 16GiB RAM, 2x Xeon X5550 @2.67 GHz, Debian 8.7, 
kernel 3.16.0-4-amd64. 

> On 02/20/2017 02:31 PM, Aleš Rygl wrote:
> > I would like to ask you for recommendations regarding network performance
> > optimization for a server running Linux with dnsdist. I am still
> > experincing
> > RcvbufErrors even if my setting for receive/send buffers is far from 
default:
> OK, you have set net.core.rmem_max to a large value but
> net.core.rmem_default is still low, you might want to increase that a
> bit, for example to 16777216


OK, done.

> > I am receiwing about 20-25 kqps of UDP traffic.
> 
> That's quite low, can you tell us more about your dnsdist configuration,
> kernel version, hardware specs and the kind or rules you are using?


The rules are not so complex. Quarrantine pools with rate limiting, dynamic 
blocks, some per-domain rules and ~ 40 per host/network qps limiting rules.

> 
> On dnsdist's side, the first things you'll need to check are:
> - setMaxUDPOutstanding() is set to a large enough value, I'd recommend 65535
> - if you reach 100% of one core, you probably want to use reuseport and
> multiple addLocal(xxx, true, true) to use several cores
> - you can add the same backend several times with newServer(), so that
> the responses are handled by more threads
> - if you use Lua a lot, you might also want to consider using LuaJIT
> instead of Lua

It looks like setMaxUDPOutstanding() is the key for this moment! There are no 
RcvbufErrors from the moment I have increased it to max value. Thanks!

I will have look at multiple cores usage. dnsdist consumes ~ 120% cpu.

> > Is there something else I can do apart from buying better hw?
> 
> That's probably the first question I should have asked, but have you
> explicitly disabled any kind of connection state tracking? Otherwise you
> can be sure the conntrack will be the bottleneck.

There is no connection tracking, no iptables modules are loaded. Firewalling 
is done on a Cisco box in front of the dnsdist.

> 
> If you have a quite old kernel, consider upgrading. That's especially
> true if you are using IPv6 (3.x were doing a very bad job there) but
> even for IPv4 there has been a lot of improvements in the processing of
> UDP datagrams.

There is not too much IPv6 traffic, just up to 1kqps. 

I will be back with my observations soon.
Thanks Remi!

BR
Ales