<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:p="urn:schemas-microsoft-com:office:powerpoint" xmlns:a="urn:schemas-microsoft-com:office:access" xmlns:dt="uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" xmlns:s="uuid:BDC6E3F0-6DA3-11d1-A2A3-00AA00C14882" xmlns:rs="urn:schemas-microsoft-com:rowset" xmlns:z="#RowsetSchema" xmlns:b="urn:schemas-microsoft-com:office:publisher" xmlns:ss="urn:schemas-microsoft-com:office:spreadsheet" xmlns:c="urn:schemas-microsoft-com:office:component:spreadsheet" xmlns:odc="urn:schemas-microsoft-com:office:odc" xmlns:oa="urn:schemas-microsoft-com:office:activation" xmlns:html="http://www.w3.org/TR/REC-html40" xmlns:q="http://schemas.xmlsoap.org/soap/envelope/" xmlns:rtc="http://microsoft.com/officenet/conferencing" xmlns:D="DAV:" xmlns:Repl="http://schemas.microsoft.com/repl/" xmlns:mt="http://schemas.microsoft.com/sharepoint/soap/meetings/" xmlns:x2="http://schemas.microsoft.com/office/excel/2003/xml" xmlns:ppda="http://www.passport.com/NameSpace.xsd" xmlns:ois="http://schemas.microsoft.com/sharepoint/soap/ois/" xmlns:dir="http://schemas.microsoft.com/sharepoint/soap/directory/" xmlns:ds="http://www.w3.org/2000/09/xmldsig#" xmlns:dsp="http://schemas.microsoft.com/sharepoint/dsp" xmlns:udc="http://schemas.microsoft.com/data/udc" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:sub="http://schemas.microsoft.com/sharepoint/soap/2002/1/alerts/" xmlns:ec="http://www.w3.org/2001/04/xmlenc#" xmlns:sp="http://schemas.microsoft.com/sharepoint/" xmlns:sps="http://schemas.microsoft.com/sharepoint/soap/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:udcs="http://schemas.microsoft.com/data/udc/soap" xmlns:udcxf="http://schemas.microsoft.com/data/udc/xmlfile" xmlns:udcp2p="http://schemas.microsoft.com/data/udc/parttopart" xmlns:wf="http://schemas.microsoft.com/sharepoint/soap/workflow/" xmlns:dsss="http://schemas.microsoft.com/office/2006/digsig-setup" xmlns:dssi="http://schemas.microsoft.com/office/2006/digsig" xmlns:mdssi="http://schemas.openxmlformats.org/package/2006/digital-signature" xmlns:mver="http://schemas.openxmlformats.org/markup-compatibility/2006" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns:mrels="http://schemas.openxmlformats.org/package/2006/relationships" xmlns:spwp="http://microsoft.com/sharepoint/webpartpages" xmlns:ex12t="http://schemas.microsoft.com/exchange/services/2006/types" xmlns:ex12m="http://schemas.microsoft.com/exchange/services/2006/messages" xmlns:pptsl="http://schemas.microsoft.com/sharepoint/soap/SlideLibrary/" xmlns:spsl="http://microsoft.com/webservices/SharePointPortalServer/PublishedLinksService" xmlns:Z="urn:schemas-microsoft-com:" xmlns:st="" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<style>
<!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
..MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.Section1
{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal>I have an interesting problem that has been happening for
about 2 weeks. First a little about my setup, currently I am running the
following:<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Debian 5.0 (Lenny)<o:p></o:p></p>
<p class=MsoNormal>Pdns-server 2.9.22-1<o:p></o:p></p>
<p class=MsoNormal>Pdns-backend-mysql 2.9.21.2-1<o:p></o:p></p>
<p class=MsoNormal>Pdns-recursor 3.1.7-1<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Hardware Platform is a Dell 1850 (dual processor), 8 GIG ram
running a VMWARE virtualized environment.<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>We are hosting about 100 forwarding lookup domains and a lot
of reverse delegation zones (we are an ISP with about 40,000 IP addresses we
currently manage).<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Our system is fairly busy but under normal traffic I very
rarely see much load on the processor/network cards.<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>This server is the primary server and we have a few (mysql
slaves) that replicate off of its database. Under normal circumstances (4
or 5 days in a row) database queue averages 0 and spikes to 2 (so the database
is keeping up just fine).<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>What I have been seeing recently show up in the logs is:<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal><i>Jun 22 09:09:38 dns1 pdns[10948]: 5003 questions waiting
for database attention. Limit is 5000, respawning<o:p></o:p></i></p>
<p class=MsoNormal><i>Jun 22 09:09:39 dns1 pdns[2538]: Our pdns instance exited
with code 1<o:p></o:p></i></p>
<p class=MsoNormal><i>Jun 22 09:09:39 dns1 pdns[2538]: Respawning<o:p></o:p></i></p>
<p class=MsoNormal><i>Jun 22 09:09:39 dns1 kernel: [724751.668503] UDP: bad
checksum. From 71.113.153.36:61250 to 208.187.180.2:53 ulen 46<o:p></o:p></i></p>
<p class=MsoNormal><i>Jun 22 09:09:40 dns1 pdns[10957]: Guardian is launching
an instance<o:p></o:p></i></p>
<p class=MsoNormal><i>Jun 22 09:09:40 dns1 pdns[10957]: Reading random entropy
from '/dev/urandom'<o:p></o:p></i></p>
<p class=MsoNormal><i>Jun 22 09:09:40 dns1 pdns[10957]: This is module
gmysqlbackend.so reporting<o:p></o:p></i></p>
<p class=MsoNormal><i>Jun 22 09:09:40 dns1 pdns[10957]: This is a guarded
instance of pdns<o:p></o:p></i></p>
<p class=MsoNormal><i>Jun 22 09:09:40 dns1 pdns[10957]: It is advised to bind
to explicit addresses with the --local-address option<o:p></o:p></i></p>
<p class=MsoNormal><i>Jun 22 09:09:40 dns1 pdns[10957]: UDP server bound to
0.0.0.0:53<o:p></o:p></i></p>
<p class=MsoNormal><i>Jun 22 09:09:40 dns1 pdns[10957]: TCP server bound to
0.0.0.0:53<o:p></o:p></i></p>
<p class=MsoNormal><i>Jun 22 09:09:40 dns1 pdns[10957]: PowerDNS 2.9.22 (C)
2001-2009 PowerDNS.COM BV (Mar 22 2009, 16:58:52, gcc 4.3.2) starting up<o:p></o:p></i></p>
<p class=MsoNormal><i>Jun 22 09:09:40 dns1 pdns[10957]: PowerDNS comes with
ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to
redistribute it according to the terms of the GPL version 2.<o:p></o:p></i></p>
<p class=MsoNormal><i>Jun 22 09:09:40 dns1 pdns[10957]: DNS Proxy launched,
local port 24312, remote 127.0.0.1:5300<o:p></o:p></i></p>
<p class=MsoNormal><i>Jun 22 09:09:40 dns1 pdns[10957]: Master/slave
communicator launching<o:p></o:p></i></p>
<p class=MsoNormal><i>Jun 22 09:09:40 dns1 pdns[10957]: Creating backend
connection for TCP<o:p></o:p></i></p>
<p class=MsoNormal><i>Jun 22 09:09:40 dns1 pdns[10957]: gmysql Connection
succesful<o:p></o:p></i></p>
<p class=MsoNormal><i>Jun 22 09:09:40 dns1 pdns[10957]: gmysql Connection
succesful<o:p></o:p></i></p>
<p class=MsoNormal><i>Jun 22 09:09:40 dns1 pdns[10957]: About to create 3
backend threads for UDP<o:p></o:p></i></p>
<p class=MsoNormal><i>Jun 22 09:09:40 dns1 pdns[10957]: gmysql Connection
succesful<o:p></o:p></i></p>
<p class=MsoNormal><i>Jun 22 09:09:40 dns1 pdns[10957]: All slave domains are
fresh<o:p></o:p></i></p>
<p class=MsoNormal><i>Jun 22 09:09:40 dns1 pdns[10957]: gmysql Connection
succesful<o:p></o:p></i></p>
<p class=MsoNormal><i>Jun 22 09:09:40 dns1 pdns[10957]: gmysql Connection
succesful<o:p></o:p></i></p>
<p class=MsoNormal><i>Jun 22 09:09:40 dns1 pdns[10957]: Done launching threads,
ready to distribute questions<o:p></o:p></i></p>
<p class=MsoNormal><i><o:p> </o:p></i></p>
<p class=MsoNormal>I will see this 11 to 12 times in less than 1 minute,
network traffic and eth0 interrupts spike quickly during this time (feeling a
little like a DNS denial of service). After this happens about 11 times I
see the following in the logs:<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal><i>Jun 22 09:09:41 dns1 pdns[10957]: 5029 questions waiting
for database attention. Limit is 5000, respawning<o:p></o:p></i></p>
<p class=MsoNormal><i>Jun 22 09:09:41 dns1 pdns[10957]: Got a signal 11,
attempting to print trace:<o:p></o:p></i></p>
<p class=MsoNormal><i>Jun 22 09:09:41 dns1 pdns[10957]:
/usr/sbin/pdns_server-instance [0x80ba397]<o:p></o:p></i></p>
<p class=MsoNormal><i>Jun 22 09:09:41 dns1 pdns[10957]: [0xb7f83400]<o:p></o:p></i></p>
<p class=MsoNormal><i>Jun 22 09:09:41 dns1 pdns[10957]:
/usr/sbin/pdns_server-instance(_ZN5boost11multi_index6detail13ordered_indexINS0_13composite_keyIN11PacketCache10CacheEntryENS0_6memberIS5_SsXadL_ZNS5_5qnameEEEEENS6_IS5_tXadL_<o:p></o:p></i></p>
<p class=MsoNormal><i>ZNS5_5qtypeEEEEENS6_IS5_tXadL_ZNS5_5ctypeEEEEENS6_IS5_iXadL_ZNS5_6zoneIDEEEEENS6_IS5_bXadL_ZNS5_15meritsRecursionEEEEENS_6tuples9null_typeESD_SD_SD_SD_EENS0_21composite_key_compareI24CIBackwardsStringCompareSt<o:p></o:p></i></p>
<p class=MsoNormal><i>4lessItESI_SH_IiESH_IbESD_SD_SD_SD_SD_EENS1_9nth_layerILi1ES5_NS0_10indexed_byINS0_14ordered_uniqueISE_SL_N4mpl_2naEEENS0_9sequencedINS0_3tagISQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_EEEESQ_<o:p></o:p></i></p>
<p class=MsoNormal><i>SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_EESaIS5_EEENS_3mpl7vector0ISQ_EENS1_18ordered_unique_tagEE10link_pointERKNS0_20composite_key_resultISE_EERNS13_9link_infoES12_+0x286)
[0x809f606]<o:p></o:p></i></p>
<p class=MsoNormal><i>Jun 22 09:09:41 dns1 pdns[10957]:
/usr/sbin/pdns_server-instance(_ZN11PacketCache6insertERKSsRK5QTypeNS_14CacheEntryTypeES1_jib+0x103)
[0x809a3c3]<o:p></o:p></i></p>
<p class=MsoNormal><i>Jun 22 09:09:41 dns1 pdns[10957]:
/usr/sbin/pdns_server-instance(_ZN12UeberBackend11addNegCacheERKNS_8QuestionE+0x8e)
[0x80c32de]<o:p></o:p></i></p>
<p class=MsoNormal><i>Jun 22 09:09:41 dns1 pdns[10957]:
/usr/sbin/pdns_server-instance(_ZN12UeberBackend3getER17DNSResourceRecord+0x12f)
[0x80c351f]<o:p></o:p></i></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>After this entry PDNS is down and stays down.<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>So a couple of questions for the group, I already have a
wire shark up doing a long term capture (so I can see what is being sent at the
server). However is there a way PDNS can email/notify when it dies and
does not come back? Also what type of information/logging should I be
enabling the system to further diagnose or troubleshoot the issue?<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Any help/feedback is greatly appreciated.<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Thanks<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>--Chris<o:p></o:p></p>
</div>
</body>
</html>