[Pdns-users] Possible DNS DOS?

Mon Jun 22 23:00:00 UTC 2009

Look at using monit. It can monitor services and email or even restart
the service for you. 

Brad Dameron

(425)216-4691 Desk
(360)340-7431 Mobile
IM: serpent6877 at yahoo.com
<outbind://23-00000000029DEF604F3FF74E9CC062CD5464C7780700EBDEAF27DD61EA
40A5DF1D15EAD9735300000071DB6F00006724EAC831AA6A4FBB9C93DFAD3A3BDC000001
91157D0000/serpent6877 at yahoo.com> 

________________________________

From: pdns-users-bounces at mailman.powerdns.com
[mailto:pdns-users-bounces at mailman.powerdns.com] On Behalf Of Chris
Modesitt
Sent: Monday, June 22, 2009 3:28 PM
To: pdns-users at mailman.powerdns.com
Subject: [Pdns-users] Possible DNS DOS?

I have an interesting problem that has been happening for about 2 weeks.
First a little about my setup, currently I am running the following:

Debian 5.0 (Lenny)

Pdns-server 2.9.22-1

Pdns-backend-mysql 2.9.21.2-1

Pdns-recursor 3.1.7-1

Hardware Platform is a Dell 1850 (dual processor), 8 GIG ram running a
VMWARE virtualized environment.

We are hosting about 100 forwarding lookup domains and a lot of reverse
delegation zones (we are an ISP with about 40,000 IP addresses we
currently manage).

Our system is fairly busy but under normal traffic I very rarely see
much load on the processor/network cards.

This server is the primary server and we have a few (mysql slaves) that
replicate off of its database.  Under normal circumstances (4 or 5 days
in a row) database queue averages 0 and spikes to 2 (so the database is
keeping up just fine).

What I have been seeing recently show up in the logs is:

Jun 22 09:09:38 dns1 pdns[10948]: 5003 questions waiting for database
attention. Limit is 5000, respawning

Jun 22 09:09:39 dns1 pdns[2538]: Our pdns instance exited with code 1

Jun 22 09:09:39 dns1 pdns[2538]: Respawning

Jun 22 09:09:39 dns1 kernel: [724751.668503] UDP: bad checksum. From
71.113.153.36:61250 to 208.187.180.2:53 ulen 46

Jun 22 09:09:40 dns1 pdns[10957]: Guardian is launching an instance

Jun 22 09:09:40 dns1 pdns[10957]: Reading random entropy from
'/dev/urandom'

Jun 22 09:09:40 dns1 pdns[10957]: This is module gmysqlbackend.so
reporting

Jun 22 09:09:40 dns1 pdns[10957]: This is a guarded instance of pdns

Jun 22 09:09:40 dns1 pdns[10957]: It is advised to bind to explicit
addresses with the --local-address option

Jun 22 09:09:40 dns1 pdns[10957]: UDP server bound to 0.0.0.0:53

Jun 22 09:09:40 dns1 pdns[10957]: TCP server bound to 0.0.0.0:53

Jun 22 09:09:40 dns1 pdns[10957]: PowerDNS 2.9.22 (C) 2001-2009
PowerDNS.COM BV (Mar 22 2009, 16:58:52, gcc 4.3.2) starting up

Jun 22 09:09:40 dns1 pdns[10957]: PowerDNS comes with ABSOLUTELY NO
WARRANTY. This is free software, and you are welcome to redistribute it
according to the terms of the GPL version 2.

Jun 22 09:09:40 dns1 pdns[10957]: DNS Proxy launched, local port 24312,
remote 127.0.0.1:5300

Jun 22 09:09:40 dns1 pdns[10957]: Master/slave communicator launching

Jun 22 09:09:40 dns1 pdns[10957]: Creating backend connection for TCP

Jun 22 09:09:40 dns1 pdns[10957]: gmysql Connection succesful

Jun 22 09:09:40 dns1 pdns[10957]: gmysql Connection succesful

Jun 22 09:09:40 dns1 pdns[10957]: About to create 3 backend threads for
UDP

Jun 22 09:09:40 dns1 pdns[10957]: gmysql Connection succesful

Jun 22 09:09:40 dns1 pdns[10957]: All slave domains are fresh

Jun 22 09:09:40 dns1 pdns[10957]: gmysql Connection succesful

Jun 22 09:09:40 dns1 pdns[10957]: gmysql Connection succesful

Jun 22 09:09:40 dns1 pdns[10957]: Done launching threads, ready to
distribute questions

I will see this 11 to 12 times in less than 1 minute, network traffic
and eth0 interrupts spike quickly during this time (feeling a little
like a DNS denial of service).  After this happens about 11 times I see
the following in the logs:

Jun 22 09:09:41 dns1 pdns[10957]: 5029 questions waiting for database
attention. Limit is 5000, respawning

Jun 22 09:09:41 dns1 pdns[10957]: Got a signal 11, attempting to print
trace:

Jun 22 09:09:41 dns1 pdns[10957]: /usr/sbin/pdns_server-instance
[0x80ba397]

Jun 22 09:09:41 dns1 pdns[10957]: [0xb7f83400]

Jun 22 09:09:41 dns1 pdns[10957]:
/usr/sbin/pdns_server-instance(_ZN5boost11multi_index6detail13ordered_in
dexINS0_13composite_keyIN11PacketCache10CacheEntryENS0_6memberIS5_SsXadL
_ZNS5_5qnameEEEEENS6_IS5_tXadL_

ZNS5_5qtypeEEEEENS6_IS5_tXadL_ZNS5_5ctypeEEEEENS6_IS5_iXadL_ZNS5_6zoneID
EEEEENS6_IS5_bXadL_ZNS5_15meritsRecursionEEEEENS_6tuples9null_typeESD_SD
_SD_SD_EENS0_21composite_key_compareI24CIBackwardsStringCompareSt

4lessItESI_SH_IiESH_IbESD_SD_SD_SD_SD_EENS1_9nth_layerILi1ES5_NS0_10inde
xed_byINS0_14ordered_uniqueISE_SL_N4mpl_2naEEENS0_9sequencedINS0_3tagISQ
_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_EEEESQ_

SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_EESaIS5_EEENS_3mpl7ve
ctor0ISQ_EENS1_18ordered_unique_tagEE10link_pointERKNS0_20composite_key_
resultISE_EERNS13_9link_infoES12_+0x286) [0x809f606]

Jun 22 09:09:41 dns1 pdns[10957]:
/usr/sbin/pdns_server-instance(_ZN11PacketCache6insertERKSsRK5QTypeNS_14
CacheEntryTypeES1_jib+0x103) [0x809a3c3]

Jun 22 09:09:41 dns1 pdns[10957]:
/usr/sbin/pdns_server-instance(_ZN12UeberBackend11addNegCacheERKNS_8Ques
tionE+0x8e) [0x80c32de]

Jun 22 09:09:41 dns1 pdns[10957]:
/usr/sbin/pdns_server-instance(_ZN12UeberBackend3getER17DNSResourceRecor
d+0x12f) [0x80c351f]

After this entry PDNS is down and stays down.

So a couple of questions for the group, I already have a wire shark up
doing a long term capture (so I can see what is being sent at the
server).  However is there a way PDNS can email/notify when it dies and
does not come back?  Also what type of information/logging should I be
enabling the system to further diagnose or troubleshoot the issue?

Any help/feedback is greatly appreciated.

Thanks

--Chris

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.powerdns.com/pipermail/pdns-users/attachments/20090622/057e6ef1/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.jpg
Type: image/jpeg
Size: 1635 bytes
Desc: image002.jpg
URL: <http://mailman.powerdns.com/pipermail/pdns-users/attachments/20090622/057e6ef1/attachment-0001.jpg>