[Pdns-users] Question about Powerdns remote database backend best practices

[Thomas Mieslinger]

Hi,

I wouldn't be able to sleep if I had to make sure to keep 5M zones
consistent with AXFR/IXFR.

Having a database with a tranaction-ids and where I can monitor "slave
seconds behind master" on the slaves makes my job doable.

I think I have not yet understood your concern about scaling horizontally.

Lets say your database size is 40GB. You buy 64GB RAM Servers with 16
cores, install Linux, MariaDB, PowerDNS. PowerDNS connects locally over
unix domain socket. Set MariaDB Page Cache to 40GB. Upon startup of
MariaDB a script is run to select everything, so that no query from pdns
triggers a physical IO. This setup will easily do 40k req/s.

If you have 12 locations with 4 servers each, you have potentially ~2M qps.

Cheers

Thomas

On 11/28/20 11:01 AM, Jianan Wang via Pdns-users wrote:
> Hi Chris,
>
> Thanks for your detailed explanation on the scaling plan for power dns
> deployment! It is very helpful from my current standing point, where we
> use mostly bind based solutions and replication is more through
> axfr/ixfr. Since it is pretty inspiring, I would like to follow up with
> some question if you would not mind:
>
> (1) Let's say you have master location A and B serving as writer, so it
> means location C, D's local regional zone will also be written into
> location A & B's dns instance? Sounds like the trade-off is between
> simplicity of management compared to write availability for non-master
> locations. Is the location you mentioned more like within a country or
> world-wide?
>
> (2) In your deployment here, it seems like your authoritative servers
> are the ones being replicated, and you will have local resolvers
> pointing to them behind the dnslist pairs you mentioned, right?
>
> (3) I bet managing 5 million zones will be a pretty complicated
> question, so just curious are you guys more a UI based management or git
> based management in this case?
>
> (4) Is there any good practice to manage the mysql instance ip? Are you
> using lvs, vip style or ip anycast solution to point to master so it
> would be free of changes? Normally we use domain names so that changing
> ip of mysql instance will not impact all the services. However, since
> powerdns is relying on this database anyways, meaning we could only use
> ip or dns served by other instances.
>
> Thanks for your help!
> Jianan.
>
> On Tue, Nov 17, 2020 at 10:11 PM Chris <mail at gbe0.com
> <mailto:mail at gbe0.com>> wrote:
>
>     Hi,
>
>     The way I scale it out (this is scaled out to 12 locations currently):
>
>     We picked 2 of our large locations to act as master locations. These
>     2 different locations have an active/standby mariadb server cluster
>     using corosync/drbd (so only 1 server in each location is master at
>     a time). The two locations are setup as master/master (mysql
>     replication). These are the servers for which all write changes are
>     applied to. I only put them in two locations as this is much easier
>     to manage and cuts down on replication problems.
>
>     Each of the 12 locations we are in then has at least a pair of
>     servers for PowerDNS. Each of these servers has their own individual
>     copy of the MySQL DB; their copy is read only and they replicate it
>     from the master servers. We only use MySQL to replicate the
>     databases, we don't handle that at the DNS level (eg. axfr/ixfr).
>     This way if the masters are taken offline (since they are only in
>     two locations) at least DNS continues to work from the read only
>     copy without changes. It would have to be a very bad day for all 4
>     master servers to do be dead anyway.
>
>     Each of the 12 locations also has at least a pair of dnsdist
>     frontends to accept queries from clients and direct it to the
>     various PowerDNS backends.
>
>     There is a bit more to it (delayed slaves for DR purposes, different
>     local/remote pools for dnsdist etc.) of course, but thats the basics.
>
>     We have over 5 million zones and over 40 million records (plus
>     additional things like DNSSEC keys, metadata entries and so on) and
>     so far this has scaled nicely for us.
>
>
>     ------------------------------------------------------------------------
>     *From:* Pdns-users <pdns-users-bounces at mailman.powerdns.com
>     <mailto:pdns-users-bounces at mailman.powerdns.com>> on behalf of
>     Jianan Wang via Pdns-users <pdns-users at mailman.powerdns.com
>     <mailto:pdns-users at mailman.powerdns.com>>
>     *Sent:* Wednesday, 18 November 2020 00:05
>     *To:* pdns-users at mailman.powerdns.com
>     <mailto:pdns-users at mailman.powerdns.com>
>     <pdns-users at mailman.powerdns.com
>     <mailto:pdns-users at mailman.powerdns.com>>
>     *Subject:* Re: [Pdns-users] Question about Powerdns remote database
>     backend best practices
>     Hi Thomas,
>
>     Thanks for your response on this! IIUC, what you recommend is still
>     having MySQL and PowerDNS created on one instance and scale the
>     instance setup horizontally? If that’s the case, will the MySQL on
>     the single node be bounded by its resources locally to scale when we
>     have a lot of records?
>
>     Thanks.
>     Jianan.--
>     Jianan Wang
>
>
>
>
> --
> Jianan Wang
>
> _______________________________________________
> Pdns-users mailing list
> Pdns-users at mailman.powerdns.com
> https://mailman.powerdns.com/mailman/listinfo/pdns-users
>