[Pdns-users] SQLite3 problem during stress

Sat Jul 18 16:51:08 UTC 2009

Hello.

We are trying to migrate from bind9 to PowerDNS.
We have our back-end supermaster successfully constructed using PowerDNS +
postgresql 8.3 + alsonotify-patch (http://wiki.powerdns.com/trac/ticket/216)
The slaves have to have a very small footprint and therefore we have chosen
sqlite3 as our target database.

After combating some weird incompatibility with Debian Lenny 5.0 where PDNSs
gsqlite3 refused to write anything to the database it begun to crash when we
did the initial transfer (notify of several hundred domains).
I have tried with the OpenDBX backend which seems to work much better
regarding the sqlite3 incompatibility but also crashes during heavy transfer
load.
Setting the distributor-threads to 1 helps it somewhat, but I think there
might be a different thread that messes things up.

Log excerpt:
Jul 18 16:45:40 Reading random entropy from '/dev/urandom'
Jul 18 16:45:40  [OpendbxBackend] This is the opendbx module version 2.9.22
(Jul 18 2009, 14:19:38) reporting
Jul 18 16:45:40  [OpendbxBackend] This is the opendbx module version 2.9.22
(Jul 18 2009, 14:19:28) reporting
Jul 18 16:45:40 This is a standalone pdns
Jul 18 16:45:40 It is advised to bind to explicit addresses with the
--local-address option
Jul 18 16:45:40 UDP server bound to 0.0.0.0:53
Jul 18 16:45:40 TCP server bound to 0.0.0.0:53
Jul 18 16:45:40 PowerDNS 2.9.22 (C) 2001-2009 PowerDNS.COM BV (Jul 18 2009,
14:23:39, gcc 4.3.2) starting up
Jul 18 16:45:40 PowerDNS comes with ABSOLUTELY NO WARRANTY. This is free
software, and you are welcome to redistribute it according to the terms of
the GPL version 2.
Jul 18 16:45:40 Set effective group id to 104
Jul 18 16:45:40 Set effective user id to 102
Jul 18 16:45:40 DNS Proxy launched, local port 27636, remote 127.0.0.1:1053
Jul 18 16:45:40 Master/slave communicator launching
Jul 18 16:45:40 Launched webserver on 127.0.0.1:8081
Jul 18 16:45:40 Creating backend connection for TCP
% Jul 18 16:45:40 [OpendbxBackend] Database connection (read) to
'/srv/pdns/' succeeded
Jul 18 16:45:40 [OpendbxBackend] Database connection (write) to '/srv/pdns/'
succeeded
Jul 18 16:45:40 211 slave domains need checking
Jul 18 16:45:40 [OpendbxBackend] Database connection (read) to '/srv/pdns/'
succeeded
Jul 18 16:45:40 [OpendbxBackend] Database connection (write) to '/srv/pdns/'
succeeded
Jul 18 16:45:40 About to create 1 backend threads for UDP
Jul 18 16:45:40 [OpendbxBackend] Database connection (read) to '/srv/pdns/'
succeeded
Jul 18 16:45:40 [OpendbxBackend] Database connection (write) to '/srv/pdns/'
succeeded
Jul 18 16:45:40 Done launching threads, ready to distribute questions
Jul 18 16:45:47 Engaging bypass - now operating unthreaded
Jul 18 16:45:47 [OpendbxBackend] Database connection (read) to '/srv/pdns/'
succeeded
Jul 18 16:45:47 [OpendbxBackend] Database connection (write) to '/srv/pdns/'
succeeded
Jul 18 16:45:47 Received NOTIFY for stadsbudcentralen.nu from
213.132.111.158 for which we are not authoritative
Jul 18 16:45:47 Created new slave zone 'stadsbudcentralen.nu' from
supermaster 213.132.111.158, queued axfr
Jul 18 16:45:47 Received NOTIFY for stadsbudcentralen.se from
213.132.111.158 for which we are not authoritative
Jul 18 16:45:47 Initiating transfer of 'stadsbudcentralen.nu' from remote
'213.132.111.158'
Jul 18 16:45:47 [OpendbxBackend] Database connection (read) to '/srv/pdns/'
succeeded
Jul 18 16:45:47 [OpendbxBackend] Database connection (write) to '/srv/pdns/'
succeeded
Jul 18 16:45:47 Created new slave zone 'stadsbudcentralen.se' from
supermaster 213.132.111.158, queued axfr
Jul 18 16:45:47 Received NOTIFY for westberg.info from 213.132.111.158 for
which we are not authoritative
Jul 18 16:45:47 AXFR started for 'stadsbudcentralen.nu', transaction started
Jul 18 16:45:47 AXFR done for 'stadsbudcentralen.nu', zone committed
Jul 18 16:45:47 Initiating transfer of 'stadsbudcentralen.se' from remote
'213.132.111.158'
*Jul 18 16:45:47 [OpendbxBackend] Database connection (read) to '/srv/pdns/'
succeeded
Jul 18 16:45:47 Created new slave zone 'westberg.info' from supermaster
213.132.111.158, queued axfr
Jul 18 16:45:47 [OpendbxBackend] Database connection (write) to '/srv/pdns/'
succeeded
*Jul 18 16:45:47 Received NOTIFY for trevlig.nu from 213.132.111.158 for
which we are not authoritative
Jul 18 16:45:47 Unable to find backend willing to host trevlig.nu for
potential supermaster 213.132.111.158
Jul 18 16:45:47 Received NOTIFY for drproduction.com from 213.132.111.158
for which we are not authoritative
Jul 18 16:45:47 Unable to find backend willing to host drproduction.com for
potential supermaster 213.132.111.158
Jul 18 16:45:47 Received NOTIFY for floopy.be from 213.132.111.158 for which
we are not authoritative
Jul 18 16:45:47 Unable to find backend willing to host floopy.be for
potential supermaster 213.132.111.158
Jul 18 16:45:47 Received NOTIFY for gronastuganikalmar.se from
213.132.111.158 for which we are not authoritative
Jul 18 16:45:47 Unable to find backend willing to host
gronastuganikalmar.sefor potential supermaster 213.132.111.158
Jul 18 16:45:47 Received NOTIFY for kigsrdr.org from 213.132.111.158 for
which we are not authoritative*
Jul 18 16:45:47 [OpendbxBackend] getRecord: Unable to get next result -
database is locked
Jul 18 16:45:47 Communicator thread died because of error: Error:
odbx_result() failed*
Jul 18 16:45:47 Unable to find backend willing to host kigsrdr.org for
potential supermaster 213.132.111.158
Jul 18 16:45:47 Received NOTIFY for kalmargamecenter.com from
213.132.111.158 for which we are not authoritative
Jul 18 16:45:47 Unable to find backend willing to host
kalmargamecenter.comfor potential supermaster 213.132.111.158
Jul 18 16:45:47 Received NOTIFY for larvig.nu from 213.132.111.158 for which
we are not authoritative
Jul 18 16:45:47 Unable to find backend willing to host larvig.nu for
potential supermaster 213.132.111.158
Jul 18 16:45:47 Received NOTIFY for lyktan-vilshult.se from 213.132.111.158
for which we are not authoritative

This is an example where it happens quite early. Sometimes it gets a couple
of hundred domains before it crashes.
Tried with both gsqlite3 and opendbx.

Note: I have only been able to test the distributor-threads = 1 with
opendbx. But I would assume that option and thus the error is backend
agnostic in that it only requires the backend to handle one write/read
connection.

Please advice.

Christian "BC" Svensson
Codelead Systems - http://www.codelead.se
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.powerdns.com/pipermail/pdns-users/attachments/20090718/57ce62de/attachment.html>