[Pdns-users] PowerDNS in an ISP environment

[Michael Loftis]

On Tue, Aug 16, 2011 at 1:38 AM, Chris Russell
<Chris.Russell at knowledgeit.co.uk> wrote:
> Hi All,
>
>
>
> Quick question – is anyone on the list using PDNS in an ISP environment,
> especially for auth services ?

Up until a couple years ago I worked as Sr. SA/Ops Manager at Modwest,
we used PowerDNS then, and they still do today.  Something like 10k or
15k domains at the time, no idea how many today honestly.  As with
many the draw was a database backend.  There wasn't much else out
there at the time, and certainly nothing stable like PowerDNS.  With
10k+ domains BIND would take a very LONG time to start/restart or even
check for updates.  There was also the headaches involved in
maintaining slave and master zone configs too.  Authoritative DNS
only.  There's a cluster of BIND servers for resolver functionality.
The actual NS records point at load balanced clusters of DNS servers.
To the outside it looks like there are only a handful of
geographically diverse nameservers, in reality there's multiple
PowerDNS servers behind each IP.  Makes doing upgrades REALLY easy,
you just pull one out of the load balancer, upgrade it.  Then you can
do all the testing you want (one thing I did was to play back DNS
queries and observe/systematically check the responses, without
letting any actual traffic out) -- if it doesn't work out you can then
use whatever process you have to roll that machine back and put it
back into the cluster, or, more deeply investigate the failure.  This
was a situation though where there was a very well proven and trusted
load balancer infrastructure in place already so it absolutely made
sense to deploy externally facing DNS services behind this same setup.
 It definitely requires thought to do it that way (chicken-and-egg
scenarios come to mind, you can not have your load balancers depend on
DNS if you're going to run DNS behind them!!!) but it is reliable when
done right.

There have definitely been a few pains here and there.  Some of them
were caused by the fact that wildcard records are used.  Some of the
issues I had were caused by MySQL's sometimes flaky replication,
monitoring them was an absolute must, making sure that they were all
in sync and up to date was also absolutely required.  The benefits far
outweighed the costs at that scale for certain.