<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML xmlns="http://www.w3.org/TR/REC-html40" xmlns:v =
"urn:schemas-microsoft-com:vml" xmlns:o =
"urn:schemas-microsoft-com:office:office" xmlns:w =
"urn:schemas-microsoft-com:office:word" xmlns:x =
"urn:schemas-microsoft-com:office:excel" xmlns:p =
"urn:schemas-microsoft-com:office:powerpoint" xmlns:a =
"urn:schemas-microsoft-com:office:access" xmlns:dt =
"uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" xmlns:s =
"uuid:BDC6E3F0-6DA3-11d1-A2A3-00AA00C14882" xmlns:rs =
"urn:schemas-microsoft-com:rowset" xmlns:z = "#RowsetSchema" xmlns:b =
"urn:schemas-microsoft-com:office:publisher" xmlns:ss =
"urn:schemas-microsoft-com:office:spreadsheet" xmlns:c =
"urn:schemas-microsoft-com:office:component:spreadsheet" xmlns:odc =
"urn:schemas-microsoft-com:office:odc" xmlns:oa =
"urn:schemas-microsoft-com:office:activation" xmlns:html =
"http://www.w3.org/TR/REC-html40" xmlns:q =
"http://schemas.xmlsoap.org/soap/envelope/" xmlns:rtc =
"http://microsoft.com/officenet/conferencing" XMLNS:D = "DAV:" XMLNS:Repl =
"http://schemas.microsoft.com/repl/" xmlns:mt =
"http://schemas.microsoft.com/sharepoint/soap/meetings/" xmlns:x2 =
"http://schemas.microsoft.com/office/excel/2003/xml" xmlns:ppda =
"http://www.passport.com/NameSpace.xsd" xmlns:ois =
"http://schemas.microsoft.com/sharepoint/soap/ois/" xmlns:dir =
"http://schemas.microsoft.com/sharepoint/soap/directory/" xmlns:ds =
"http://www.w3.org/2000/09/xmldsig#" xmlns:dsp =
"http://schemas.microsoft.com/sharepoint/dsp" xmlns:udc =
"http://schemas.microsoft.com/data/udc" xmlns:xsd =
"http://www.w3.org/2001/XMLSchema" xmlns:sub =
"http://schemas.microsoft.com/sharepoint/soap/2002/1/alerts/" xmlns:ec =
"http://www.w3.org/2001/04/xmlenc#" xmlns:sp =
"http://schemas.microsoft.com/sharepoint/" xmlns:sps =
"http://schemas.microsoft.com/sharepoint/soap/" xmlns:xsi =
"http://www.w3.org/2001/XMLSchema-instance" xmlns:udcs =
"http://schemas.microsoft.com/data/udc/soap" xmlns:udcxf =
"http://schemas.microsoft.com/data/udc/xmlfile" xmlns:udcp2p =
"http://schemas.microsoft.com/data/udc/parttopart" xmlns:wf =
"http://schemas.microsoft.com/sharepoint/soap/workflow/" xmlns:dsss =
"http://schemas.microsoft.com/office/2006/digsig-setup" xmlns:dssi =
"http://schemas.microsoft.com/office/2006/digsig" xmlns:mdssi =
"http://schemas.openxmlformats.org/package/2006/digital-signature" xmlns:mver =
"http://schemas.openxmlformats.org/markup-compatibility/2006" xmlns:m =
"http://schemas.microsoft.com/office/2004/12/omml" xmlns:mrels =
"http://schemas.openxmlformats.org/package/2006/relationships" xmlns:spwp =
"http://microsoft.com/sharepoint/webpartpages" xmlns:ex12t =
"http://schemas.microsoft.com/exchange/services/2006/types" xmlns:ex12m =
"http://schemas.microsoft.com/exchange/services/2006/messages" xmlns:pptsl =
"http://schemas.microsoft.com/sharepoint/soap/SlideLibrary/" xmlns:spsl =
"http://microsoft.com/webservices/SharePointPortalServer/PublishedLinksService"
XMLNS:Z = "urn:schemas-microsoft-com:" xmlns:st = ""><HEAD>
<META http-equiv=Content-Type content="text/html; charset=us-ascii">
<META content="MSHTML 6.00.6000.16850" name=GENERATOR>
<STYLE>@font-face {
font-family: Cambria Math;
}
@font-face {
font-family: Calibri;
}
@page Section1 {size: 8.5in 11.0in; margin: 1.0in 1.0in 1.0in 1.0in; }
P.MsoNormal {
FONT-SIZE: 11pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Calibri","sans-serif"
}
LI.MsoNormal {
FONT-SIZE: 11pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Calibri","sans-serif"
}
DIV.MsoNormal {
FONT-SIZE: 11pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Calibri","sans-serif"
}
A:link {
COLOR: blue; TEXT-DECORATION: underline; mso-style-priority: 99
}
SPAN.MsoHyperlink {
COLOR: blue; TEXT-DECORATION: underline; mso-style-priority: 99
}
A:visited {
COLOR: purple; TEXT-DECORATION: underline; mso-style-priority: 99
}
SPAN.MsoHyperlinkFollowed {
COLOR: purple; TEXT-DECORATION: underline; mso-style-priority: 99
}
SPAN.EmailStyle17 {
COLOR: windowtext; FONT-FAMILY: "Calibri","sans-serif"; mso-style-type: personal
}
SPAN.EmailStyle18 {
COLOR: #1f497d; FONT-FAMILY: "Calibri","sans-serif"; mso-style-type: personal-reply
}
.MsoChpDefault {
FONT-SIZE: 10pt; mso-style-type: export-only
}
DIV.Section1 {
page: Section1
}
</STYLE>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></HEAD>
<BODY lang=EN-US vLink=purple link=blue>
<DIV dir=ltr align=left><SPAN class=783155822-22062009><FONT face=Arial
color=#0000ff size=2>Look at using monit. It can monitor services and email or
even restart the service for you. </FONT></SPAN></DIV>
<DIV> </DIV>
<DIV class=Section1><B style="mso-bidi-font-weight: normal"><SPAN
style="FONT-FAMILY: Arial">Brad Dameron<o:p></o:p></SPAN></B><BR><IMG height=27
src="cid:783155822@22062009-121F" width=91 v:shapes="_x0000_i1025">
<P class=MsoNormal style="mso-pagination: widow-orphan lines-together"><TT><SPAN
style="FONT-SIZE: 10pt">(425)216-4691 <SPAN
class=GramE>Desk</SPAN></SPAN></TT><BR><TT><SPAN
style="FONT-SIZE: 10pt">(360)340-7431 Mobile</SPAN></TT><BR><TT><SPAN
style="FONT-SIZE: 10pt">IM: <A title=blocked::serpent6877@yahoo.com
href="outbind://23-00000000029DEF604F3FF74E9CC062CD5464C7780700EBDEAF27DD61EA40A5DF1D15EAD9735300000071DB6F00006724EAC831AA6A4FBB9C93DFAD3A3BDC00000191157D0000/serpent6877@yahoo.com">serpent6877@yahoo.com</A></SPAN></TT></P></DIV>
<DIV> </DIV><BR>
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left>
<HR tabIndex=-1>
<FONT face=Tahoma size=2><B>From:</B> pdns-users-bounces@mailman.powerdns.com
[mailto:pdns-users-bounces@mailman.powerdns.com] <B>On Behalf Of </B>Chris
Modesitt<BR><B>Sent:</B> Monday, June 22, 2009 3:28 PM<BR><B>To:</B>
pdns-users@mailman.powerdns.com<BR><B>Subject:</B> [Pdns-users] Possible DNS
DOS?<BR></FONT><BR></DIV>
<DIV></DIV>
<DIV class=Section1>
<P class=MsoNormal>I have an interesting problem that has been happening for
about 2 weeks. First a little about my setup, currently I am running the
following:<o:p></o:p></P>
<P class=MsoNormal><o:p> </o:p></P>
<P class=MsoNormal>Debian 5.0 (Lenny)<o:p></o:p></P>
<P class=MsoNormal>Pdns-server 2.9.22-1<o:p></o:p></P>
<P class=MsoNormal>Pdns-backend-mysql 2.9.21.2-1<o:p></o:p></P>
<P class=MsoNormal>Pdns-recursor 3.1.7-1<o:p></o:p></P>
<P class=MsoNormal><o:p> </o:p></P>
<P class=MsoNormal>Hardware Platform is a Dell 1850 (dual processor), 8 GIG ram
running a VMWARE virtualized environment.<o:p></o:p></P>
<P class=MsoNormal><o:p> </o:p></P>
<P class=MsoNormal>We are hosting about 100 forwarding lookup domains and a lot
of reverse delegation zones (we are an ISP with about 40,000 IP addresses we
currently manage).<o:p></o:p></P>
<P class=MsoNormal><o:p> </o:p></P>
<P class=MsoNormal>Our system is fairly busy but under normal traffic I very
rarely see much load on the processor/network cards.<o:p></o:p></P>
<P class=MsoNormal><o:p> </o:p></P>
<P class=MsoNormal>This server is the primary server and we have a few (mysql
slaves) that replicate off of its database. Under normal circumstances (4
or 5 days in a row) database queue averages 0 and spikes to 2 (so the database
is keeping up just fine).<o:p></o:p></P>
<P class=MsoNormal><o:p> </o:p></P>
<P class=MsoNormal>What I have been seeing recently show up in the logs
is:<o:p></o:p></P>
<P class=MsoNormal><o:p> </o:p></P>
<P class=MsoNormal><I>Jun 22 09:09:38 dns1 pdns[10948]: 5003 questions waiting
for database attention. Limit is 5000, respawning<o:p></o:p></I></P>
<P class=MsoNormal><I>Jun 22 09:09:39 dns1 pdns[2538]: Our pdns instance exited
with code 1<o:p></o:p></I></P>
<P class=MsoNormal><I>Jun 22 09:09:39 dns1 pdns[2538]:
Respawning<o:p></o:p></I></P>
<P class=MsoNormal><I>Jun 22 09:09:39 dns1 kernel: [724751.668503] UDP: bad
checksum. From 71.113.153.36:61250 to 208.187.180.2:53 ulen
46<o:p></o:p></I></P>
<P class=MsoNormal><I>Jun 22 09:09:40 dns1 pdns[10957]: Guardian is launching an
instance<o:p></o:p></I></P>
<P class=MsoNormal><I>Jun 22 09:09:40 dns1 pdns[10957]: Reading random entropy
from '/dev/urandom'<o:p></o:p></I></P>
<P class=MsoNormal><I>Jun 22 09:09:40 dns1 pdns[10957]: This is module
gmysqlbackend.so reporting<o:p></o:p></I></P>
<P class=MsoNormal><I>Jun 22 09:09:40 dns1 pdns[10957]: This is a guarded
instance of pdns<o:p></o:p></I></P>
<P class=MsoNormal><I>Jun 22 09:09:40 dns1 pdns[10957]: It is advised to bind to
explicit addresses with the --local-address option<o:p></o:p></I></P>
<P class=MsoNormal><I>Jun 22 09:09:40 dns1 pdns[10957]: UDP server bound to
0.0.0.0:53<o:p></o:p></I></P>
<P class=MsoNormal><I>Jun 22 09:09:40 dns1 pdns[10957]: TCP server bound to
0.0.0.0:53<o:p></o:p></I></P>
<P class=MsoNormal><I>Jun 22 09:09:40 dns1 pdns[10957]: PowerDNS 2.9.22 (C)
2001-2009 PowerDNS.COM BV (Mar 22 2009, 16:58:52, gcc 4.3.2) starting
up<o:p></o:p></I></P>
<P class=MsoNormal><I>Jun 22 09:09:40 dns1 pdns[10957]: PowerDNS comes with
ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to
redistribute it according to the terms of the GPL version 2.<o:p></o:p></I></P>
<P class=MsoNormal><I>Jun 22 09:09:40 dns1 pdns[10957]: DNS Proxy launched,
local port 24312, remote 127.0.0.1:5300<o:p></o:p></I></P>
<P class=MsoNormal><I>Jun 22 09:09:40 dns1 pdns[10957]: Master/slave
communicator launching<o:p></o:p></I></P>
<P class=MsoNormal><I>Jun 22 09:09:40 dns1 pdns[10957]: Creating backend
connection for TCP<o:p></o:p></I></P>
<P class=MsoNormal><I>Jun 22 09:09:40 dns1 pdns[10957]: gmysql Connection
succesful<o:p></o:p></I></P>
<P class=MsoNormal><I>Jun 22 09:09:40 dns1 pdns[10957]: gmysql Connection
succesful<o:p></o:p></I></P>
<P class=MsoNormal><I>Jun 22 09:09:40 dns1 pdns[10957]: About to create 3
backend threads for UDP<o:p></o:p></I></P>
<P class=MsoNormal><I>Jun 22 09:09:40 dns1 pdns[10957]: gmysql Connection
succesful<o:p></o:p></I></P>
<P class=MsoNormal><I>Jun 22 09:09:40 dns1 pdns[10957]: All slave domains are
fresh<o:p></o:p></I></P>
<P class=MsoNormal><I>Jun 22 09:09:40 dns1 pdns[10957]: gmysql Connection
succesful<o:p></o:p></I></P>
<P class=MsoNormal><I>Jun 22 09:09:40 dns1 pdns[10957]: gmysql Connection
succesful<o:p></o:p></I></P>
<P class=MsoNormal><I>Jun 22 09:09:40 dns1 pdns[10957]: Done launching threads,
ready to distribute questions<o:p></o:p></I></P>
<P class=MsoNormal><I><o:p> </o:p></I></P>
<P class=MsoNormal>I will see this 11 to 12 times in less than 1 minute, network
traffic and eth0 interrupts spike quickly during this time (feeling a little
like a DNS denial of service). After this happens about 11 times I see the
following in the logs:<o:p></o:p></P>
<P class=MsoNormal><o:p> </o:p></P>
<P class=MsoNormal><I>Jun 22 09:09:41 dns1 pdns[10957]: 5029 questions waiting
for database attention. Limit is 5000, respawning<o:p></o:p></I></P>
<P class=MsoNormal><I>Jun 22 09:09:41 dns1 pdns[10957]: Got a signal 11,
attempting to print trace:<o:p></o:p></I></P>
<P class=MsoNormal><I>Jun 22 09:09:41 dns1 pdns[10957]:
/usr/sbin/pdns_server-instance [0x80ba397]<o:p></o:p></I></P>
<P class=MsoNormal><I>Jun 22 09:09:41 dns1 pdns[10957]:
[0xb7f83400]<o:p></o:p></I></P>
<P class=MsoNormal><I>Jun 22 09:09:41 dns1 pdns[10957]:
/usr/sbin/pdns_server-instance(_ZN5boost11multi_index6detail13ordered_indexINS0_13composite_keyIN11PacketCache10CacheEntryENS0_6memberIS5_SsXadL_ZNS5_5qnameEEEEENS6_IS5_tXadL_<o:p></o:p></I></P>
<P
class=MsoNormal><I>ZNS5_5qtypeEEEEENS6_IS5_tXadL_ZNS5_5ctypeEEEEENS6_IS5_iXadL_ZNS5_6zoneIDEEEEENS6_IS5_bXadL_ZNS5_15meritsRecursionEEEEENS_6tuples9null_typeESD_SD_SD_SD_EENS0_21composite_key_compareI24CIBackwardsStringCompareSt<o:p></o:p></I></P>
<P
class=MsoNormal><I>4lessItESI_SH_IiESH_IbESD_SD_SD_SD_SD_EENS1_9nth_layerILi1ES5_NS0_10indexed_byINS0_14ordered_uniqueISE_SL_N4mpl_2naEEENS0_9sequencedINS0_3tagISQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_EEEESQ_<o:p></o:p></I></P>
<P
class=MsoNormal><I>SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_EESaIS5_EEENS_3mpl7vector0ISQ_EENS1_18ordered_unique_tagEE10link_pointERKNS0_20composite_key_resultISE_EERNS13_9link_infoES12_+0x286)
[0x809f606]<o:p></o:p></I></P>
<P class=MsoNormal><I>Jun 22 09:09:41 dns1 pdns[10957]:
/usr/sbin/pdns_server-instance(_ZN11PacketCache6insertERKSsRK5QTypeNS_14CacheEntryTypeES1_jib+0x103)
[0x809a3c3]<o:p></o:p></I></P>
<P class=MsoNormal><I>Jun 22 09:09:41 dns1 pdns[10957]:
/usr/sbin/pdns_server-instance(_ZN12UeberBackend11addNegCacheERKNS_8QuestionE+0x8e)
[0x80c32de]<o:p></o:p></I></P>
<P class=MsoNormal><I>Jun 22 09:09:41 dns1 pdns[10957]:
/usr/sbin/pdns_server-instance(_ZN12UeberBackend3getER17DNSResourceRecord+0x12f)
[0x80c351f]<o:p></o:p></I></P>
<P class=MsoNormal><o:p> </o:p></P>
<P class=MsoNormal>After this entry PDNS is down and stays down.<o:p></o:p></P>
<P class=MsoNormal><o:p> </o:p></P>
<P class=MsoNormal>So a couple of questions for the group, I already have a wire
shark up doing a long term capture (so I can see what is being sent at the
server). However is there a way PDNS can email/notify when it dies and
does not come back? Also what type of information/logging should I be
enabling the system to further diagnose or troubleshoot the
issue?<o:p></o:p></P>
<P class=MsoNormal><o:p> </o:p></P>
<P class=MsoNormal>Any help/feedback is greatly appreciated.<o:p></o:p></P>
<P class=MsoNormal><o:p> </o:p></P>
<P class=MsoNormal>Thanks<o:p></o:p></P>
<P class=MsoNormal><o:p> </o:p></P>
<P class=MsoNormal>--Chris<o:p></o:p></P></DIV></BODY></HTML>