[Pdns-users] Recursor QPS ceiling

[Ciro Iriarte]

2014-12-22 17:30 GMT-03:00 bert hubert <bert.hubert at netherlabs.nl>:
> On Mon, Dec 22, 2014 at 02:39:49PM +0100, Morten Stevens wrote:
>> Here is my result compared with bind and unbound:
>> 1) pdns-recursor 3.6.2: 169k QPS
>> 2) unbound 1.5.1: 327k QPS
>> 3) bind 9.9.4-P2: 251k QPS
>
> Well - let me describe how we like to do our benchmarking.
>
> 1) Use tcpdump to record actual traffic that is representative of your
users
> 2) Use dnsreplay with the --speedup flag to see how much 'headroom' you
have
> on your platform
> 3) Study the graphs like http://xs.powerdns.com/metronome/ to see if
> response times & CPU loads are where you want them
>
> The issue is that fully cached performance is great, but this is like how
> fast a car drives on a sunny day on a straight road - perhaps 250kph.
>
> In real life, what we want to know is how the car handles in actual
traffic,
> on rough roads, through corners etc.
>
> The real work in DNS is for uncached queries, for difficult queries, for
> hammering clients, for domains with timeouts.
>
> Of course, every once in a while, we too suffer from a benchmark-driven
> optimization round. And then we might suddenly care a lot about 100%
cached
> queries too ;-)
>
> But for now, winning the battle for "rock solid service under elevated
real
> loads" is our goal.
>
>> I am surprised that bind9 is much faster than pdns-recursor in my test...
>
> Fwiw, at a recent (few weeks ago) huge nordic deployment, bind9 was
replaced
> by pdns-recursor because bind9 could not take the (real life) load.
>
> Further to what Robert noted down the thread, actual benchmarking at
> 200kqps+ is hard work on the benchmarking tools as well. In some of the
more
> realistic tests, multiple client systems have been used to get actual
> numbers.
>
> dnsreplay (described lightly on
https://doc.powerdns.com/md/tools/analysis/
> ) sometimes also does not achieve the query rates you want on a single
> system.
>
>         Bert
>
> _______________________________________________
> Pdns-users mailing list
> Pdns-users at mailman.powerdns.com
> http://mailman.powerdns.com/mailman/listinfo/pdns-users

Thanks a lot for the insight. What should we be looking for on the
dnsreplay output?. I run it on a cold-cache-recursor and got this numbers:

         Questions - Pend. - Drop = Answers = (On time + Late) = (Err + Ok)
Orig     5894888     0       217808 11527843   0         320346   0     0
Refer.   5894888     0       390969 5491046    0         409381   0     0
0 queries that could still come in on time, 6728 outstanding
we late: 409381, orig late: 320346, 5894888 questions sent, 11527843
original answers, 1749152 perfect, 3281086 mostly correct, 58024 we better,
8279 ori better (0 diff)
we never: 390969, orig never: 217808
original questions from IP addresses for which recursion was not available:
102115
Unmatched from us: 87234, unmatched from original: 1492121 ( - decoding
err: 1491725)
DNS decoding errors from us: 0, from original: 396, exact duplicates from
client: 155773


I'm currently running it with 2x speed, but is not really clear how good or
bad the new recursor is. For example I see all the answers listed in the
"Late" columns which sounds bad, and the "unmatched from original" is a lot
higher than "unmatched from us" plus, the "we better" is higher than "orig
better", which both sound good


         Questions - Pend. - Drop = Answers = (On time + Late) = (Err + Ok)
Orig     3226515     1497    134997 6281328    0         231733   0     0
Refer.   3226515     8249    1424352 1684913   0         1477002  0     0
8249 queries that could still come in on time, 57265 outstanding
we late: 1477002, orig late: 231733, 3226515 questions sent, 6281328
original answers, 510530 perfect, 980609 mostly correct, 22153 we better,
3400 orig better (0 diff)
we never: 1424352, orig never: 134997
original questions from IP addresses for which recursion was not available:
61821
Unmatched from us: 49229, unmatched from original: 451438 ( - decoding err:
451231)
DNS decoding errors from us: 0, from original: 207, exact duplicates from
client: 128090

Comments?

Regards,
-- 
Ciro Iriarte
http://iriarte.it
--
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.powerdns.com/pipermail/pdns-users/attachments/20141229/507f05d3/attachment-0001.html>