[Pdns-users] powerdns_server crashes after a few minutes

[Scott Wunsch]

Hi all,

I've been testing out PowerDNS as a potential replacement for BIND on a
server that I manage.  Most of our load comes from users' recursive lookups
(around 350 queries/sec), so I figured I'd temporarily throw that traffic
at pdns_server and pdns_recursor running on a test server and see how it
held up.  It, uh, didn't :-).

It would work fine for a few minutes (usually around five minutes), and
then suddenly crash something like this:

 May 24 16:02:11 pdns_server(_ZN9__gnu_cxx9hashtableISt4pairIKSsSt6vectorI16BBResourceRecordSaIS4_EEESs11hash_stringSt10_Select1stIS7_E14compare_stringSaIS6_EE6resizeEj+0xf6) [0x8137a52]
 May 24 16:02:11 pdns_server(_ZN9__gnu_cxx9hashtableISt4pairIKSsSt6vectorI16BBResourceRecordSaIS4_EEESs11hash_stringSt10_Select1stIS7_E14compare_stringSaIS6_EE14find_or_insertERKS7_+0x25) [0x8137aed]
 May 24 16:02:11 pdns_server(_ZN9__gnu_cxx8hash_mapISsSt6vectorI16BBResourceRecordSaIS2_EE11hash_string14compare_stringSaIS4_EEixERKSs+0xbc) [0x8137dc8]
 May 24 16:02:11 pdns_server(_ZN11BindBackend6lookupERK5QTypeRKSsP9DNSPacketi+0xe3) [0x812c62b]
 May 24 16:02:11 pdns_server(_ZN12UeberBackend6lookupERK5QTypeRKSsP9DNSPacketi+0x53c) [0x80cbce4]
 May 24 16:02:11 pdns_server(_ZN13PacketHandler11makeCanonicEP9DNSPacketS1_RSs+0xbe) [0x808d672]
 May 24 16:02:11 pdns_server(_ZN13PacketHandler8questionEP9DNSPacket+0x6ed) [0x8094431]
 May 24 16:02:11 pdns_server(_ZN11DistributorI9DNSPacketS0_13PacketHandlerE10makeThreadEPv+0x4e6) [0x80fbea6]
 May 24 16:02:11 /lib/tls/libpthread.so.0 [0xf503ae]
 May 24 16:02:11 /lib/tls/libc.so.6(__clone+0x5e) [0x9ecaee]
 Aborted

It seems relatively consistent to my untrained eye, but here's another
example:

 May 24 16:12:04 Got a signal 11, attempting to print trace:
 May 24 16:12:04 pdns_server [0x80bf0e0]
 May 24 16:12:04 /lib/tls/libc.so.6 [0x4d8a48]
 May 24 16:12:04 pdns_server(_ZN9__gnu_cxx9hashtableISt4pairIKSsSt6vectorI16BBResourceRecordSaIS4_EEESs11hash_stringSt10_Select1stIS7_E14compare_stringSaIS6_EE6resizeEj+0x104) [0x8137a60]
 May 24 16:12:04 pdns_server(_ZN9__gnu_cxx9hashtableISt4pairIKSsSt6vectorI16BBResourceRecordSaIS4_EEESs11hash_stringSt10_Select1stIS7_E14compare_stringSaIS6_EE14find_or_insertERKS7_+0x25) [0x8137aed]
 May 24 16:12:04 pdns_server(_ZN9__gnu_cxx8hash_mapISsSt6vectorI16BBResourceRecordSaIS2_EE11hash_string14compare_stringSaIS4_EEixERKSs+0xbc) [0x8137dc8]
 May 24 16:12:04 pdns_server(_ZN11BindBackend6lookupERK5QTypeRKSsP9DNSPacketi+0xe3) [0x812c62b]
 May 24 16:12:04 pdns_server(_ZN12UeberBackend6lookupERK5QTypeRKSsP9DNSPacketi+0x53c) [0x80cbce4]
 May 24 16:12:04 pdns_server(_ZN13PacketHandler17doWildcardRecordsEP9DNSPacketS1_RSs+0x155) [0x809046d]
 May 24 16:12:04 pdns_server(_ZN13PacketHandler8questionEP9DNSPacket+0x3112) [0x8096e56]
 May 24 16:12:04 pdns_server(_ZN11DistributorI9DNSPacketS0_13PacketHandlerE10makeThreadEPv+0x4e6) [0x80fbea6]
 May 24 16:12:04 /lib/tls/libpthread.so.0 [0x4863ae]
 May 24 16:12:04 /lib/tls/libc.so.6(__clone+0x5e) [0x577aee]
 Aborted

In both cases, the vact majority of the queries were recursive queries that
would have been handled by pdns_recursor (which survived just fine).  In
the first example, I had a couple small zones being handled by the bind
backend; those were removed for the second example.

Any suggestions as to what I should check, or what I can try to deal with
hhese crashes?

-- 
Take care,
Scott \\'unsch

... Smoking is a leading cause of statistics.