[dnsdist] dnsdist performance

[Remi Gacogne]

Hi Nico,

On 04/05/2017 12:24 AM, Nico wrote:
> Test with rule REFUSED:
> dnsdist from rpm: ~450 Kqps
> dnsdist-concur: ~530kqps    
> 17% improvement

I guess that's the move from CLOCK_MONOTONIC_RAW to CLOCK_MONOTONIC,
quite impressive.

> Test with answer from cache:
> 1 dnsdist from rpm 12 listeners: ~130 kqps
> 3 dnsdist from rpm 6 listeners each: ~370kqps
> 1 dnsdist-concur 12 listeners ~170 kqps
> 1 dnsdist-concur 20 listeners  ~170 kqps
> 3 dnsdist-concur 6 listeners each ~450 kqps

So quite a noticeable gain but it looks like lock contention is still an
issue. I would like to understand why, if you don't mind answering a few
questions.

- You mentioned having 32 cores, are they real cores or is it with
hyper-threading? Intel reports [1] only 8 real cores for the E5-2660, so
you should probably stick with at most 8 total threads per CPU
(listeners mostly in your case).

- I'd be interested in the results of the dumpStats() and
cache:printStats() commands during your test, as well as a perf top,
ideally with a vanilla dnsdist and a dnsdist-concur.

- The cache cleaning algo might be a bit aggressive by default, you can
tweak it with:
setCacheCleaningDelay(30)
setCacheCleaningPercentage(20)

- Exporting carbon data to our public metronome instance would be great
too, as it would immediately make a lot of metrics available to us. You
can do that with: carbonServer('37.252.122.50  ', '<yourname>', 30)

- Tuning the network buffer might also help:
net.core.rmem_max=33554432
net.core.wmem_max=33554432
net.core.rmem_default=16777216
net.core.wmem_default=16777216

- Would you consider upgrading your kernel? There has been a lot of
improvements since 3.10.0, and we noticed huge performance increases in
the past just by upgrading to a 4.x one.

Oh and if you didn't already, would you mind setting
setMaxUDPOutstanding() to 65535? Even at a 99% cache hit ratio, that
leaves quite a few requests going to the backend so we better be sure we
don't mess up these. The cache in dnsdist tries very hard not to degrade
performance, so we prefer skipping the cache and passing the query to a
backend rather than waiting for a cache lock, for example.

Thanks,

Remi

[1]:
https://ark.intel.com/products/64584/Intel-Xeon-Processor-E5-2660-20M-Cache-2_20-GHz-8_00-GTs-Intel-QPI


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://mailman.powerdns.com/pipermail/dnsdist/attachments/20170405/03ce8f21/attachment.sig>