[Pdns-users] 24 tickets were closed, new snapshot 998
Kenneth Marshall
ktm at rice.edu
Mon Mar 26 14:09:14 UTC 2007
On Sun, Mar 25, 2007 at 09:00:35PM +0200, bert hubert wrote:
> Hi everybody,
>
> Heading up to 2.9.21, today saw the closing of 24 tickets. This is reflected
> in snapshot 998, which is getting pretty close to release material.
>
> The list of closed tickets + tickets includes:
>
> * everything related to multi-part TXT and SPF ('Domain key') records
> * zone2sql simply did not work in recent snapshots
> * sparse error logging by the built-in webserver
> * bind backend has problems immediately after notify
> * NAPTR mis-parsing
> * comment misparsing in TXT records
> * CNAME records to delegations didn't work
>
> Additionally, this snapshot also contains all fixes reported yesterday for
> the BIND backend.
>
> For more details, see http://wiki.powerdns.com/cgi-bin/trac.fcgi/timeline
>
> Please try snapshot 998 on http://svn.powerdns.com/snapshots/998 and report
> if your issue is fixed.
>
> Thanks!
>
Dear Mr. Hubert,
I am looking forward to the release of 2.9.21. I do have some questions
about the performance of the AXFR transfer when PDNS is acting as the
slave nameserver. As you know, the semantics of the AXFR process are
rudimentary and map very well to a simple file transfer process such as
that used by BIND. The mapping of this process to the SQL database
results in much more work being done to effect the same results. This
results in very poor zone transfer performance for large domains. In
our case, it takes about 3 hours to perform the zone update. The fix
is to take advantage of the database to minimize the record churn and
limit it to the records that must be changed, instead of a wholesale
delete and replace operation.
To enable this functionality, the query definitions for the AXFR process
need to be made orthogonal by adding a final "commit-zone-axfr-query"
instead of the hard coded current query "COMMIT;". By adding this
functionality, you can populate a temporary table with the AXFR transfer
records and then calculate a differential update to the current set of
records and apply only those changes. This allows the previously mentioned
3+ hours AXFR transfer to take less than 9 minutes. I have attached a
patch that I made for 2.9.20 rev. 924 and a variety of sample queries
that could be used to take advantage of this extra commit query functionality.
The simplest should run on the widest select of SQL backends.
I would appreciate any feedback that you may have and would like to see
this functionality included in the new release. I would be happy to generate
a new patch against the new trunk if it would help.
Regards,
Ken Marshall
-------------- next part --------------
Index: pdns/docs/pdns.sgml
===================================================================
--- pdns/docs/pdns.sgml (revision 924)
+++ pdns/docs/pdns.sgml (working copy)
@@ -9960,6 +9960,17 @@
</para>
</listitem>
</varlistentry>
+ <varlistentry>
+ <term>commit-zone-axfr-query</term>
+ <listitem>
+ <para>
+ Called to commit an incoming AXFR. The domain_id is available in the query using %d.
+ Default: <command>
+ commit
+ </command>
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
<sect2><title>Fancy records</title>
<para>
Index: pdns/dnsbackend.hh
===================================================================
--- pdns/dnsbackend.hh (revision 924)
+++ pdns/dnsbackend.hh (working copy)
@@ -100,7 +100,7 @@
}
//! commits the transaction started by startTransaction
- virtual bool commitTransaction()
+ virtual bool commitTransaction(uint32_t domain_id)
{
return false;
}
Index: pdns/backends/gsql/gsqlbackend.cc
===================================================================
--- pdns/backends/gsql/gsqlbackend.cc (revision 924)
+++ pdns/backends/gsql/gsqlbackend.cc (working copy)
@@ -239,6 +239,7 @@
d_UpdateLastCheckofZoneQuery=getArg("update-lastcheck-query");
d_InfoOfAllMasterDomainsQuery=getArg("info-all-master-query");
d_DeleteZoneQuery=getArg("delete-zone-query");
+ d_CommitZoneAXFRQuery=getArg("commit-zone-axfr-query");
}
@@ -414,14 +415,17 @@
return true;
}
-bool GSQLBackend::commitTransaction()
+bool GSQLBackend::commitTransaction(uint32_t domain_id)
{
+ char output[1024];
+ snprintf(output,sizeof(output)-1,d_CommitZoneAXFRQuery.c_str(),domain_id);
try {
- d_db->doCommand("commit");
+ d_db->doCommand(output);
}
catch (SSqlException &e) {
throw AhuException("Database failed to commit transaction: "+e.txtReason());
}
+
return true;
}
Index: pdns/backends/gsql/gsqlbackend.hh
===================================================================
--- pdns/backends/gsql/gsqlbackend.hh (revision 924)
+++ pdns/backends/gsql/gsqlbackend.hh (working copy)
@@ -28,7 +28,7 @@
bool isMaster(const string &domain, const string &ip);
bool startTransaction(const string &domain, int domain_id=-1);
- bool commitTransaction();
+ bool commitTransaction(uint32_t domain_id);
bool abortTransaction();
bool feedRecord(const DNSResourceRecord &r);
bool createSlaveDomain(const string &ip, const string &domain, const string &account);
@@ -66,5 +66,6 @@
string d_UpdateLastCheckofZoneQuery;
string d_InfoOfAllMasterDomainsQuery;
string d_DeleteZoneQuery;
+ string d_CommitZoneAXFRQuery;
};
Index: pdns/backends/bind/bindbackend.hh
===================================================================
--- pdns/backends/bind/bindbackend.hh (revision 924)
+++ pdns/backends/bind/bindbackend.hh (working copy)
@@ -189,7 +189,7 @@
bool startTransaction(const string &qname, int id);
// bool BindBackend::stopTransaction(const string &qname, int id);
bool feedRecord(const DNSResourceRecord &r);
- bool commitTransaction();
+ bool commitTransaction(uint32_t domain_id);
bool abortTransaction();
void insert(int id, const string &qname, const string &qtype, const string &content, int ttl, int prio);
void rediscover(string *status=0);
Index: pdns/backends/bind/bindbackend2.cc
===================================================================
--- pdns/backends/bind/bindbackend2.cc (revision 924)
+++ pdns/backends/bind/bindbackend2.cc (working copy)
@@ -155,7 +155,7 @@
return true;
}
-bool Bind2Backend::commitTransaction()
+bool Bind2Backend::commitTransaction(uint32_t domain_id)
{
delete d_of;
d_of=0;
Index: pdns/backends/bind/bindbackend2.hh
===================================================================
--- pdns/backends/bind/bindbackend2.hh (revision 924)
+++ pdns/backends/bind/bindbackend2.hh (working copy)
@@ -108,7 +108,7 @@
bool startTransaction(const string &qname, int id);
// bool Bind2Backend::stopTransaction(const string &qname, int id);
bool feedRecord(const DNSResourceRecord &r);
- bool commitTransaction();
+ bool commitTransaction(uint32_t domain_id);
bool abortTransaction();
typedef map<uint32_t, BB2DomainInfo> id_zone_map_t;
Index: pdns/backends/bind/bindbackend.cc
===================================================================
--- pdns/backends/bind/bindbackend.cc (revision 924)
+++ pdns/backends/bind/bindbackend.cc (working copy)
@@ -126,7 +126,7 @@
return true;
}
-bool BindBackend::commitTransaction()
+bool BindBackend::commitTransaction(uint32_t domain_id)
{
delete d_of;
d_of=0;
Index: modules/gsqlitebackend/gsqlitebackend.cc
===================================================================
--- modules/gsqlitebackend/gsqlitebackend.cc (revision 924)
+++ modules/gsqlitebackend/gsqlitebackend.cc (working copy)
@@ -75,6 +75,7 @@
declare( suffix, "update-lastcheck-query", "", "update domains set last_check=%d where id=%d");
declare( suffix, "info-all-master-query", "", "select id,name,master,last_check,notified_serial,type from domains where type='MASTER'");
declare( suffix, "delete-zone-query", "", "delete from records where domain_id=%d");
+ declare( suffix, "commit-zone-axfr-query", "", "commit");
}
//! Constructs a new gSQLiteBackend object.
Index: modules/goraclebackend/goraclebackend.cc
===================================================================
--- modules/goraclebackend/goraclebackend.cc (revision 924)
+++ modules/goraclebackend/goraclebackend.cc (working copy)
@@ -68,6 +68,7 @@
declare(suffix,"update-lastcheck-query","", "update domains set last_check=%d where id=%d");
declare(suffix,"info-all-master-query","", "select id,name,master,last_check,notified_serial,type from domains where type='MASTER'");
declare(suffix,"delete-zone-query","", "delete from records where domain_id=%d");
+ declare(suffix,"commit-zone-axfr-query","", "commit");
}
Index: modules/gmysqlbackend/gmysqlbackend.cc
===================================================================
--- modules/gmysqlbackend/gmysqlbackend.cc (revision 924)
+++ modules/gmysqlbackend/gmysqlbackend.cc (working copy)
@@ -74,6 +74,7 @@
declare(suffix,"update-lastcheck-query","", "update domains set last_check=%d where id=%d");
declare(suffix,"info-all-master-query","", "select id,name,master,last_check,notified_serial,type from domains where type='MASTER'");
declare(suffix,"delete-zone-query","", "delete from records where domain_id=%d");
+ declare(suffix,"commit-zone-axfr-query","", "commit");
}
Index: modules/gpgsqlbackend/gpgsqlbackend.cc
===================================================================
--- modules/gpgsqlbackend/gpgsqlbackend.cc (revision 924)
+++ modules/gpgsqlbackend/gpgsqlbackend.cc (working copy)
@@ -74,6 +74,7 @@
declare(suffix,"update-lastcheck-query","", "update domains set last_check=%d where id=%d");
declare(suffix,"info-all-master-query","", "select id,name,master,last_check,notified_serial,type from domains where type='MASTER'");
declare(suffix,"delete-zone-query","", "delete from records where domain_id=%d");
+ declare(suffix,"commit-zone-axfr-query","", "commit");
}
-------------- next part --------------
launch=gpgsql
gpgsql-host=10.0.0.1
gpgsql-dbname=pdns
gpgsql-user=postgres
gpgsql-password=""
#Update AXFR queries to use stage table for staging
gpgsql-delete-zone-query=DELETE FROM stage WHERE domain_id = %d;
gpgsql-insert-record-query=INSERT INTO stage \
(content,ttl,prio,type,domain_id,name) VALUES ('%s',%d,%d,'%s',%d,'%s');
gpgsql-update-serial-query=BEGIN; \
CREATE TEMPORARY TABLE axfrvars (name VARCHAR(6), value INT) ON COMMIT DROP; \
INSERT INTO axfrvars (name, value) VALUES ('serial', %d); \
INSERT INTO axfrvars (name, value) VALUES ('domain', %d); \
DELETE FROM records \
WHERE domain_id = \
(SELECT value FROM axfrvars WHERE name = 'domain') AND \
records.id NOT IN \
(SELECT records.id FROM records INNER JOIN stage \
USING (domain_id, name, type, content, ttl, prio)); \
INSERT INTO records (domain_id, name, type, content, ttl, prio) \
SELECT domain_id, name, type, content, ttl, prio FROM stage \
WHERE stage.domain_id = \
(SELECT value FROM axfrvars WHERE name = 'domain') EXCEPT \
SELECT domain_id, name, type, content, ttl, prio from records \
WHERE records.domain_id = \
(SELECT value FROM axfrvars WHERE name = 'domain'); \
UPDATE domains SET notified_serial = \
(SELECT value FROM axfrvars WHERE name = 'serial') \
WHERE id = \
(SELECT value FROM axfrvars WHERE name = 'domain'); \
COMMIT;
-------------- next part --------------
launch=gpgsql
gpgsql-host=10.0.0.1
gpgsql-dbname=pdns
gpgsql-user=postgres
gpgsql-password=""
#Update AXFR queries to use stage table for staging
gpgsql-delete-zone-query=CREATE TEMPORARY TABLE stage (id INT, domain_id INT, name VARCHAR(255), type VARCHAR(6), content VARCHAR(255), ttl INT, prio INT);
gpgsql-insert-record-query=INSERT INTO stage (content,ttl,prio,type,domain_id,name) VALUES ('%s',%d,%d,'%s',%d,'%s');
gpgsql-update-serial-query=BEGIN; \
CREATE TEMPORARY TABLE axfrvars (name VARCHAR(6), value INT) ON COMMIT DROP; \
INSERT INTO axfrvars (name, value) VALUES ('serial', %d); \
INSERT INTO axfrvars (name, value) VALUES ('domain', %d); \
DELETE FROM records \
WHERE domain_id = \
(SELECT value FROM axfrvars WHERE name = 'domain') AND \
records.id NOT IN \
(SELECT records.id FROM records INNER JOIN stage \
USING (domain_id, name, type, content, ttl, prio)); \
INSERT INTO records (domain_id, name, type, content, ttl, prio) \
SELECT domain_id, name, type, content, ttl, prio FROM stage \
WHERE stage.domain_id = \
(SELECT value FROM axfrvars WHERE name = 'domain') EXCEPT \
SELECT domain_id, name, type, content, ttl, prio from records \
WHERE records.domain_id = \
(SELECT value FROM axfrvars WHERE name = 'domain'); \
UPDATE domains SET notified_serial = \
(SELECT value FROM axfrvars WHERE name = 'serial') \
WHERE id = \
(SELECT value FROM axfrvars WHERE name = 'domain'); \
COMMIT; DROP TABLE stage;
-------------- next part --------------
launch=gpgsql
gpgsql-host=10.0.0.1
gpgsql-dbname=pdns
gpgsql-user=postgres
gpgsql-password=""
#Update AXFR queries to use stage table for staging
gpgsql-delete-zone-query=CREATE TEMPORARY TABLE stage (id INT, domain_id INT, name VARCHAR(255), type VARCHAR(6), content VARCHAR(255), ttl INT, prio INT);PREPARE axfrinsert (varchar, int, int, varchar, int, varchar) AS INSERT INTO stage (content,ttl,prio,type,domain_id,name) VALUES ($1,$2,$3,$4,$5,$6);
gpgsql-insert-record-query=EXECUTE axfrinsert ('%s',%d,%d,'%s',%d,'%s');
gpgsql-update-serial-query=BEGIN; \
CREATE TEMPORARY TABLE axfrvars (name VARCHAR(6), value INT) ON COMMIT DROP; \
INSERT INTO axfrvars (name, value) VALUES ('serial', %d); \
INSERT INTO axfrvars (name, value) VALUES ('domain', %d); \
DELETE FROM records \
WHERE domain_id = \
(SELECT value FROM axfrvars WHERE name = 'domain') AND \
records.id NOT IN \
(SELECT records.id FROM records INNER JOIN stage \
USING (domain_id, name, type, content, ttl, prio)); \
INSERT INTO records (domain_id, name, type, content, ttl, prio) \
SELECT domain_id, name, type, content, ttl, prio FROM stage \
WHERE stage.domain_id = \
(SELECT value FROM axfrvars WHERE name = 'domain') EXCEPT \
SELECT domain_id, name, type, content, ttl, prio from records \
WHERE records.domain_id = \
(SELECT value FROM axfrvars WHERE name = 'domain'); \
UPDATE domains SET notified_serial = \
(SELECT value FROM axfrvars WHERE name = 'serial') \
WHERE id = \
(SELECT value FROM axfrvars WHERE name = 'domain'); \
DELETE FROM stage WHERE domain_id = \
(SELECT value FROM axfrvars WHERE name = 'domain'); \
COMMIT;
More information about the Pdns-users
mailing list