[Pdns-users] Questions about PowerDNS - CNAME at APEX, Capacity, management, etc...

[Klaus Darilion]

Hi Jake!

This answers are probably worth some $$$$ for consulting. Anyways...

> -----Ursprüngliche Nachricht-----
> Von: Pdns-users <pdns-users-bounces at mailman.powerdns.com> Im
...
> Does PowerDNS load all of the zones into memory, and then start serving
> (like BIND), or does it load each zone and start serving said zone
> immediately upon load (like KNOT)?

It may depend on the backend. I can only talk about the postgresql backend. As the zone data is stored in the backend, PowerDNS will not load any zones into memory. Hence it will answer queries immediately. Where immediately means, that of course the backend hast o load the data from the disk. But this is not done in advance but on request.

Btw: My Knots (secondaries) will only start answering if ALL zones are loaded.


> When in "secondary" mode, how does PowerDNS select which "primary" to
> retrieve a zone from?  Is it based on the configured list, is it based on
> performance, or is it based on which server it first received a notify
> from?

For regular SOA Checks it should select randomly. For SOA checks triggered by NOTIFY it should query the name server that sent the NOTIFY but I have not tested this (implemented quite recently)

> What's the best method for updating a backend with DNS changes in an
> instance where there are 6 million+ zones?  Assuming updating the database
> isn't the best method, what kind of volume can the API or dynamic update
> paths handle?  Is my assumption correct that I only have to worry about
> this in a "primary" configuration, and that a "secondary" configuration
> can rely entirely on standard zone transfers?

Why wouldn't updateing the DB not the best method? It is a possibilty but it requires some actions to make it correctly (rectify zone). Yes, the API would take care that everyting is correct in the DB.

You could also use AXFR inbound into to your "primary" if zones are coming from some other systems.

Regarding secondary you have 2 choices: standard AXFR or database replication. Standard AXFR gives you the possibilty to also integrate other name server products, but database replication would also solve the provisioning.

When talking about secondaries, the zone transfer is the most easiest part. The challenges are:
1. Zone provisioning on the secondary (how will powerdns knot that it should be secondary for a certain zone?)
Solutions:
a) out of band management
b) PowerDNs Superslave Feature (deletion of zones must be done out of band/manually)
c) Catalog zones (not yet implemented in PDNS) 
d) database replication (the easiest solution)

2. Checking the Sync status. You probably want to know if some zone at some secondary is not in sync with the primary. NOTIFYs can get lost, AXFR can fail. There can and will be temporary network issues to your secondaries, so during 1 minute network issues you may miss plenty of NOTIFYs. To work around you could configure SOA refresh checks every hour. With 6 mio zones that would be constantly 1700q/s on your primary. With 20 secondaries you have 34kq/s only for soa checks (which fully the the backend on your primary as you usually want to disable caching on the primary, at least if you plan to manipulate the backend DB directly).
Customers always detect this very fast an complain. Hence, you do want to detect such issues automatically and automatically retransfer zones.
Solutions:
a) plenty of SOA checks
b) out of band check
c) database replication (the easiest solution) - here you just monitor the replication lag which is a single metric for all your zones

> CNAME @ APEX questions:
Actually it is: CNAME besides other RRs, other RRs below DNAME, .... 

> Would I be right in my assumption that if one wants to use the CNAME @
> APEX recursion hack, that the entire ecosystem (both distribution servers
> and cloud servers) would need to be PowerDNS to accomplish this, given
> that BIND and KNOT will not load a zone with CNAME @ APEX?

Knot master will accept CNAMEs/DNAME errors if semantic checks are set to "soft". NSD accepts such zones as secondary. MAybe Bind also accepts them if you disable all the syntax check. Of course, when disabling such check, you can never know if the response is what you expect - as it is not defined how the answer should look  like. For example Knot with CNAME checks disabled responds differently as PowerDNS for CNAMEs at APEX.

So, stop accepting CNAME bugs, and have work arounds for existing zones until the are fixed (I know this is not easy)

> What about DNS software vendor diversity?  Anyone else out there doing
> CNAME @ APEX that you find is a good mix with PowerDNS?
> 
> 
> 
> Capacity questions:
> 
> Can a PowerDNS instance handle 6 million zones and 300 million records?

It depends on the server and the query load. The more queries, the more CPUs you need. The more ressource records you have, the more RAM you need to have.

So, with a very fat server you can handle it.

> Does one backend perform better than the others at this capacity?

Backend perform different. Regardless of the number of records. DB-backends are SLOW, the Bind backend might be compareble to Bind, the lmdb backend is probably the fastest.

> If operating a global anycast network, is the backend database sitting on
> the DNS server itself, or does one dedicate a node or two in each site as
> part of a backend database cluster?  How much does an on-server database
> impact query performance?

By using 1 server for PDNS and 1 server for the DB you might achieve more performance if you have in total more CPUs.
But if you use ie 64 cores at 1 server, or 2 servers with 32 cores each, then probably a single server would be faster as accessing the DB via Unix sockets is faster than using TCP (at least with postgresql)

> At what point do you hit the requirement for multiple backends?

This depends where you have your servers. If they are baremetal owned by you, cou can order what you like. If you are in a cloud, you have to see what your cloud provider offers. Server with more than 128G RAM are rare. But you can easily get  easily 10 servers with 32G RAM at the same provider.

So, hardware/VMs limit and growing userbase will probably take you to a setup where each secondary is actually server servers (several frontends and several backend and some kind of load balancing for such a big setup)

> Can anyone provide me a fuzzy number on RAM footprint?
The number of zones does not matter - it is the number of records. Ideally the whole database must be in RAM for accaptable performance, at least the indexes must be in RAM. PowerDNS does not need much RAM, only for the Chache. I guess for 300M records you need around 200GB RAM.

> example:  "I have $x million records, and that's using $y amount of RAM"
> 
> What is the startup timing on a large PowerDNS installation?
With db backend: 1 ms

PS: With that amount of zones and a "normal" mix of customer zones you would probably have 24x7 crap traffic (dictionary queries, random subdomain attacks, ...) which really hurts PDNS with database backends. So, public facing secondaries with DB backend won'T make you happy in the long term, although it is the fastest and reliable way to start.

regards
Klaus