[Pdns-users] Odd master/slave behavior for large domains

[Kenneth Marshall]

Hi Bert,

I do not know if you saw my post on this problem earlier today,
but I do not think that there is a bug in the PDNS PostgreSQL
support, per se. I spent a bit of time with pdns's PostgreSQL
backend debugging the AXFR transfer problem that I described
in my earlier E-mail today. I performed a pretty thorough
analysis of the PostgreSQL backend versus the BIND and MySQL
backends. The performance problems stem from the MVCC nature
of the PostgreSQL database versus the file based nature of
the BIND backend and the in memory update nature of the MySQL
MyISM backend. The textbook translation of the AXFR process
while technically correct has bad performance implications
when used with the PostgreSQL backend. I will comment inline
on the source of the problems below.

On Fri, Sep 11, 2009 at 08:56:27PM +0200, bert hubert wrote:
> On Fri, Sep 11, 2009 at 7:14 AM, thomas morgan <tm at zerigo.com> wrote:
> > I created a single zone on the server and added 2 million host records. I
> > know that's a bunch, but it is a specific use case, not just an attempt to
> > break things.
> 
> Thomas,
> 
> Many thanks for your interesting and detailed bug report! I've done
> some initial investigation, and my guess is that this is a deficiency
> in our PostgreSQL support, or perhaps in the PostgreSQL client library
> (unlikely I think).
> 
> Sadly I am very busy right now, but you could help debugging this if
> you could reproduce this issue (or perhaps, fail to) using the MySQL
> backend or the BIND backend.
> 

The actual transfer problem Thomas documents is caused by the Linux
OOM process killing the database process before it can complete. This
is not the fault of the PDNS PostgreSQL module or the PostgreSQL database.
This is a system misconfiguration that is exacerbated by the basic PDNS
AXFR implementation. In particular, the info-zone-query on the master
performs "select id,name,master,last_check,notified_serial,type from domains
where name='%s'". This has the effect of materializing the entire zone in
the memory footprint of the database. For this large zone, this results in
a substantially larger amount of memory used on the server. The usual
method to avoid this huge increase in memory resources used is to use
a CURSOR to retrieve the zone entries in smaller chunks. The OOM process
also targets processes using more memory and this increase can cause it
to trigger and the AXFR will fail.

> If this clears up the issue, I know where to look.
> 
> >
> > Oddity #1:
> > The master seems to send 3-4 NOTIFYs when the zone is updated -- at least
> > the slave is reporting in the logs to have received multiple NOTIFYs.
> > They're pretty consistently spaced: in several instances, 4 NOTIFYs with
> > intervals 3sec, 5sec, 9sec.
> 
> This is just laziness - we keep sending NOTIFY packets until they are
> acknowledged, which PowerDNS only does after the AXFR succeeded.
> 
This is not really a problem as you have mentioned.

> > Oddity #2:
> > The slave, upon receiving multiple NOTIFYs, initiates multiple AXFRs for the
> > same zone. For a small zone this wouldn't be a big deal, but for a large
> > zone it's fatal.
> 
> This one is stupid on our side.
> 
If the zone's serial number is included in the NOTIFY, it should be
possible to tell if another AXFR is needed or not.

> > Oddity #3:
> > The master consumes a *huge* amount of RAM to do each AXFR: 283 MB worth per
> > AXFR. Memory usage on the slave seems tolerable though.
> >
> > Oddity #4:
> > If an AXFR doesn't finish properly, the memory is never released. I've
> > managed to reproduce this using dig to perform the AXFR as well.
> 
> 3 and 4 should go away if we debug the (probable) PostgreSQL support
> problem in PowerDNS.
> 
3 and 4 are caused by the select * from the domain and the triggering
of the Linux OOM process killer. Thomas did not mention it, but AXFR's
for large zones are very slow to a PDNS slave with a PostgreSQL backend.
Adding a simple hook to the commit of the transfer of the zone allows
for a much more efficient AXFR transfer. I would certainly like to help
with any improvements that can be made to the PostgreSQL support. I can
bring my commit-zone-axfr-query hook patch up to date if that would
help. Thank you again for a great name server.

Regards,
Ken

> > Anything I'm missing or that I can do to help figure out what's wrong? Some
> > logs are included below; they are not identical in every case, but this is
> > representative.
> 
> Reproducing with the BIND backend should help find the cause of this.
> You can just ask it to load the output of dig -t AXFR > zone.
> 
> Please let me know! From there we'll look at the other issues you found.
> 
>     Bert
> _______________________________________________
> Pdns-users mailing list
> Pdns-users at mailman.powerdns.com
> http://mailman.powerdns.com/mailman/listinfo/pdns-users
>