[Pdns-announce] Large performance increases in Authoritative Server & Recursor

[bert hubert]

Hi everybody,

Short version: if you've had disappointing benchmark results, please rerun
them using the latest Authoritative AND Recursor packages from
https://repo.powerdns.com/ , and let us know your results.

Pretty graphs:
https://github.com/PowerDNS/pdns/pull/4467#issuecomment-246932898

Longer version:

Some weeks ago, the kind people of the RIPE NCC were benchmarking various
nameserver products on real root server traffic, and they reported the
PowerDNS Authoritative Server 4.0.1 performance was very disappointing.

(note, even the disappointing results delivered many times more performance
than typical PowerDNS Authoritative Servers we know in production, so it is
not a disaster)

When we studied their results (with help from Anand), also using telemetry
via Metronome (see https://blog.powerdns.com/2014/12/11/powerdns-graphing-as-a-service/ ), 
we found that PowerDNS Authoritative was not well set-up to operate as a
root server.

The traffic we had optimized ourselves for ('hundreds of thousands of
domains with lots of content') is very different from what a root-server
sees: almost only delegations, very large answer packets, many unique
queries.

We then embarked on a large refactoring of Authoritative to improve
root server performance, and have achieved at least a factor of five (or, a
400%) improvement (see graphs below).

This now allows a run of the mill 4-core desktop to more than saturate a
gigabit/s connection when operating as a root server.

Most of us do not run a root server, but the benefits of this cleanup extend
to all Authoritative installations and even to the PowerDNS Recursor, since
some of the core speedups impact packet generation speed, which more than
doubled. Your improvements will vary depending on backend and cache
hitrates.

So, if you recently did any benchmarking of Authoritative or Recursor and
found the results disappointing, or if you want to see better results,
please redo your measurements against the latest packages from
https://repo.powerdns.com/ and let us know your results!

How did we achieve a 400% improvement? This came in smaller and larger
chunks, mostly involving:

 * Far faster DNS label compression code (large packet generation is now twice as fast)
 * Removing a lot of malloc/new/free/delete traffic
 * Keeping data in binary from directly from the backend, reducing string
   manipulation
 * Addition of hashed indexes to BIND backend, Packet cache and query cache

Further details are on:
https://github.com/PowerDNS/pdns/pull/4467 and
https://github.com/PowerDNS/pdns/pull/4373

Finally, Kees Monshouwer and I have a few more things in mind to increase
performance, so there is more to come in the 4.1/4.2 timeframe!

	Bert