[Pdns-users] Possible DNS DOS?

Mon Jun 22 22:27:58 UTC 2009

I have an interesting problem that has been happening for about 2 weeks.
First a little about my setup, currently I am running the following:

Debian 5.0 (Lenny)

Pdns-server 2.9.22-1

Pdns-backend-mysql 2.9.21.2-1

Pdns-recursor 3.1.7-1

Hardware Platform is a Dell 1850 (dual processor), 8 GIG ram running a
VMWARE virtualized environment.

We are hosting about 100 forwarding lookup domains and a lot of reverse
delegation zones (we are an ISP with about 40,000 IP addresses we currently
manage).

Our system is fairly busy but under normal traffic I very rarely see much
load on the processor/network cards.

This server is the primary server and we have a few (mysql slaves) that
replicate off of its database.  Under normal circumstances (4 or 5 days in a
row) database queue averages 0 and spikes to 2 (so the database is keeping
up just fine).

What I have been seeing recently show up in the logs is:

Jun 22 09:09:38 dns1 pdns[10948]: 5003 questions waiting for database
attention. Limit is 5000, respawning

Jun 22 09:09:39 dns1 pdns[2538]: Our pdns instance exited with code 1

Jun 22 09:09:39 dns1 pdns[2538]: Respawning

Jun 22 09:09:39 dns1 kernel: [724751.668503] UDP: bad checksum. From
71.113.153.36:61250 to 208.187.180.2:53 ulen 46

Jun 22 09:09:40 dns1 pdns[10957]: Guardian is launching an instance

Jun 22 09:09:40 dns1 pdns[10957]: Reading random entropy from '/dev/urandom'

Jun 22 09:09:40 dns1 pdns[10957]: This is module gmysqlbackend.so reporting

Jun 22 09:09:40 dns1 pdns[10957]: This is a guarded instance of pdns

Jun 22 09:09:40 dns1 pdns[10957]: It is advised to bind to explicit
addresses with the --local-address option

Jun 22 09:09:40 dns1 pdns[10957]: UDP server bound to 0.0.0.0:53

Jun 22 09:09:40 dns1 pdns[10957]: TCP server bound to 0.0.0.0:53

Jun 22 09:09:40 dns1 pdns[10957]: PowerDNS 2.9.22 (C) 2001-2009 PowerDNS.COM
BV (Mar 22 2009, 16:58:52, gcc 4.3.2) starting up

Jun 22 09:09:40 dns1 pdns[10957]: PowerDNS comes with ABSOLUTELY NO
WARRANTY. This is free software, and you are welcome to redistribute it
according to the terms of the GPL version 2.

Jun 22 09:09:40 dns1 pdns[10957]: DNS Proxy launched, local port 24312,
remote 127.0.0.1:5300

Jun 22 09:09:40 dns1 pdns[10957]: Master/slave communicator launching

Jun 22 09:09:40 dns1 pdns[10957]: Creating backend connection for TCP

Jun 22 09:09:40 dns1 pdns[10957]: gmysql Connection succesful

Jun 22 09:09:40 dns1 pdns[10957]: gmysql Connection succesful

Jun 22 09:09:40 dns1 pdns[10957]: About to create 3 backend threads for UDP

Jun 22 09:09:40 dns1 pdns[10957]: gmysql Connection succesful

Jun 22 09:09:40 dns1 pdns[10957]: All slave domains are fresh

Jun 22 09:09:40 dns1 pdns[10957]: gmysql Connection succesful

Jun 22 09:09:40 dns1 pdns[10957]: gmysql Connection succesful

Jun 22 09:09:40 dns1 pdns[10957]: Done launching threads, ready to
distribute questions

I will see this 11 to 12 times in less than 1 minute, network traffic and
eth0 interrupts spike quickly during this time (feeling a little like a DNS
denial of service).  After this happens about 11 times I see the following
in the logs:

Jun 22 09:09:41 dns1 pdns[10957]: 5029 questions waiting for database
attention. Limit is 5000, respawning

Jun 22 09:09:41 dns1 pdns[10957]: Got a signal 11, attempting to print
trace:

Jun 22 09:09:41 dns1 pdns[10957]: /usr/sbin/pdns_server-instance [0x80ba397]

Jun 22 09:09:41 dns1 pdns[10957]: [0xb7f83400]

Jun 22 09:09:41 dns1 pdns[10957]:
/usr/sbin/pdns_server-instance(_ZN5boost11multi_index6detail13ordered_indexI
NS0_13composite_keyIN11PacketCache10CacheEntryENS0_6memberIS5_SsXadL_ZNS5_5q
nameEEEEENS6_IS5_tXadL_

ZNS5_5qtypeEEEEENS6_IS5_tXadL_ZNS5_5ctypeEEEEENS6_IS5_iXadL_ZNS5_6zoneIDEEEE
ENS6_IS5_bXadL_ZNS5_15meritsRecursionEEEEENS_6tuples9null_typeESD_SD_SD_SD_E
ENS0_21composite_key_compareI24CIBackwardsStringCompareSt

4lessItESI_SH_IiESH_IbESD_SD_SD_SD_SD_EENS1_9nth_layerILi1ES5_NS0_10indexed_
byINS0_14ordered_uniqueISE_SL_N4mpl_2naEEENS0_9sequencedINS0_3tagISQ_SQ_SQ_S
Q_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_EEEESQ_

SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_EESaIS5_EEENS_3mpl7vector
0ISQ_EENS1_18ordered_unique_tagEE10link_pointERKNS0_20composite_key_resultIS
E_EERNS13_9link_infoES12_+0x286) [0x809f606]

Jun 22 09:09:41 dns1 pdns[10957]:
/usr/sbin/pdns_server-instance(_ZN11PacketCache6insertERKSsRK5QTypeNS_14Cach
eEntryTypeES1_jib+0x103) [0x809a3c3]

Jun 22 09:09:41 dns1 pdns[10957]:
/usr/sbin/pdns_server-instance(_ZN12UeberBackend11addNegCacheERKNS_8Question
E+0x8e) [0x80c32de]

Jun 22 09:09:41 dns1 pdns[10957]:
/usr/sbin/pdns_server-instance(_ZN12UeberBackend3getER17DNSResourceRecord+0x
12f) [0x80c351f]

After this entry PDNS is down and stays down.

So a couple of questions for the group, I already have a wire shark up doing
a long term capture (so I can see what is being sent at the server).
However is there a way PDNS can email/notify when it dies and does not come
back?  Also what type of information/logging should I be enabling the system
to further diagnose or troubleshoot the issue?

Any help/feedback is greatly appreciated.

Thanks

--Chris

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.powerdns.com/pipermail/pdns-users/attachments/20090622/c56fc10e/attachment.html>