<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">So on twitter I was pointed to this blog post about what seems to be about a related (or the same) attack:<div><br></div><div><a href="http://blog.powerdns.com/2014/04/03/further-dos-guidance-packages-and-patches-available/">http://blog.powerdns.com/2014/04/03/further-dos-guidance-packages-and-patches-available/</a></div><div><br></div><div>Now, I'm not clear on this however. Is the mitigation described in that link a solution to this problem?</div><div><br></div><div>I updated to 3.6RC1 and the good news is that the throttled percentage has gone up from 0% to 30%.</div><div><br></div><div>However, "rec_control current-queries" still shows that these queries are going out to Google's name servers:</div><div><br></div><div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;"><div><div style="margin: 0px; font-size: 13px; font-family: Monaco; color: rgb(0, 249, 41); background-color: rgb(0, 0, 0);"><span style="color: #ff3b1d">#</span> rec_control current-queries</div><div style="margin: 0px; font-size: 13px; font-family: Monaco; color: rgb(0, 249, 41); background-color: rgb(0, 0, 0);">8 currently outstanding questions</div><div style="margin: 0px; font-size: 13px; font-family: Monaco; color: rgb(0, 249, 41); background-color: rgb(0, 0, 0);">qname qtype remote tcp chained</div><div style="margin: 0px; font-size: 13px; font-family: Monaco; color: rgb(0, 249, 41); background-color: rgb(0, 0, 0);"><a href="http://abpqefghiwkyz.www.7098.com">abpqefghiwkyz.www.7098.com</a>. A 8.8.4.4 n n</div><div style="margin: 0px; font-size: 13px; font-family: Monaco; color: rgb(0, 249, 41); background-color: rgb(0, 0, 0);"><a href="http://caqrjqahz.www.7098.com">caqrjqahz.www.7098.com</a>. A 8.8.4.4 n n</div><div style="margin: 0px; font-size: 13px; font-family: Monaco; color: rgb(0, 249, 41); background-color: rgb(0, 0, 0);"><a href="http://nxiwdmg.www.7098.com">nxiwdmg.www.7098.com</a>. A 8.8.4.4 n n</div><div style="margin: 0px; font-size: 13px; font-family: Monaco; color: rgb(0, 249, 41); background-color: rgb(0, 0, 0);"><a href="http://pfpfrpooqwo.www.7098.com">pfpfrpooqwo.www.7098.com</a>. A 8.8.4.4 n n</div><div style="margin: 0px; font-size: 13px; font-family: Monaco; color: rgb(0, 249, 41); background-color: rgb(0, 0, 0);"><a href="http://aopdestuvjklm.www.7098.com">aopdestuvjklm.www.7098.com</a>. A 8.8.8.8 n n</div><div style="margin: 0px; font-size: 13px; font-family: Monaco; color: rgb(0, 249, 41); background-color: rgb(0, 0, 0);"><a href="http://nbpdefthvwklz.www.7098.com">nbpdefthvwklz.www.7098.com</a>. A 8.8.8.8 n n</div><div style="margin: 0px; font-size: 13px; font-family: Monaco; color: rgb(0, 249, 41); background-color: rgb(0, 0, 0);"><a href="http://umochbiexah.www.7098.com">umochbiexah.www.7098.com</a>. A 8.8.8.8 n n</div><div style="margin: 0px; font-size: 13px; font-family: Monaco; color: rgb(0, 249, 41); background-color: rgb(0, 0, 0);"><a href="http://xhvefhfmm.www.7098.com">xhvefhfmm.www.7098.com</a>. A 8.8.8.8 n n</div><div style="margin: 0px; font-size: 13px; font-family: Monaco; color: rgb(0, 249, 41); background-color: rgb(0, 0, 0);"> - done</div><div style="margin: 0px; font-size: 13px; font-family: Monaco; color: rgb(0, 249, 41); background-color: rgb(0, 0, 0);">10 currently outstanding questions</div><div style="margin: 0px; font-size: 13px; font-family: Monaco; color: rgb(0, 249, 41); background-color: rgb(0, 0, 0);">qname qtype remote tcp chained</div><div style="margin: 0px; font-size: 13px; font-family: Monaco; color: rgb(0, 249, 41); background-color: rgb(0, 0, 0);"><a href="http://lbhvjpk.www.7098.com">lbhvjpk.www.7098.com</a>. A 8.8.4.4 n n</div><div style="margin: 0px; font-size: 13px; font-family: Monaco; color: rgb(0, 249, 41); background-color: rgb(0, 0, 0);"><a href="http://rsoxyll.www.7098.com">rsoxyll.www.7098.com</a>. A 8.8.4.4 n n</div><div style="margin: 0px; font-size: 13px; font-family: Monaco; color: rgb(0, 249, 41); background-color: rgb(0, 0, 0);"><a href="http://zkjlkmg.www.7098.com">zkjlkmg.www.7098.com</a>. A 8.8.4.4 n n</div><div style="margin: 0px; font-size: 13px; font-family: Monaco; color: rgb(0, 249, 41); background-color: rgb(0, 0, 0);"><a href="http://azj.www.7098.com">azj.www.7098.com</a>. A 8.8.8.8 n n</div><div style="margin: 0px; font-size: 13px; font-family: Monaco; color: rgb(0, 249, 41); background-color: rgb(0, 0, 0);"><a href="http://ckyvaayyu.www.7098.com">ckyvaayyu.www.7098.com</a>. A 8.8.8.8 n n</div><div style="margin: 0px; font-size: 13px; font-family: Monaco; color: rgb(0, 249, 41); background-color: rgb(0, 0, 0);"><a href="http://jcqrchk.www.7098.com">jcqrchk.www.7098.com</a>. A 8.8.8.8 n n</div><div style="margin: 0px; font-size: 13px; font-family: Monaco; color: rgb(0, 249, 41); background-color: rgb(0, 0, 0);"><a href="http://nbpdrsguiwklm.www.7098.com">nbpdrsguiwklm.www.7098.com</a>. A 8.8.8.8 n n</div><div style="margin: 0px; font-size: 13px; font-family: Monaco; color: rgb(0, 249, 41); background-color: rgb(0, 0, 0);"><a href="http://nbpdrsguvwxyz.www.7098.com">nbpdrsguvwxyz.www.7098.com</a>. A 8.8.8.8 n n</div><div style="margin: 0px; font-size: 13px; font-family: Monaco; color: rgb(0, 249, 41); background-color: rgb(0, 0, 0);"><a href="http://qbzuhuz.www.7098.com">qbzuhuz.www.7098.com</a>. A 8.8.8.8 n n</div><div style="margin: 0px; font-size: 13px; font-family: Monaco; color: rgb(0, 249, 41); background-color: rgb(0, 0, 0);"><a href="http://xbh.www.7098.com">xbh.www.7098.com</a>. A 8.8.8.8 n n</div><div style="margin: 0px; font-size: 13px; font-family: Monaco; color: rgb(0, 249, 41); background-color: rgb(0, 0, 0); position: static; z-index: auto;"> - done</div></div></blockquote><div><br class="webkit-block-placeholder"></div><div>Shouldn't these queries be dropped simply?</div><div><br></div><div>Running the command again shows a completely different set of queries, telling me that the queries are not being dropped as I would expect.</div><div><br></div><div>So... is mitigation working or not?</div><div><br></div><div>Thanks,</div><div>Greg</div><div>
<br class="Apple-interchange-newline"><span style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; display: inline !important; float: none;">--</span><br style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; display: inline !important; float: none;">Please do not email me anything that you are not comfortable also sharing</span><span style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; display: inline !important; float: none;"> with the NSA.</span>
</div>
<br><div><div>On May 31, 2014, at 2:23 PM, okTurtles <<a href="mailto:hi@okturtles.com">hi@okturtles.com</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><meta http-equiv="Content-Type" content="text/html charset=us-ascii"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">Hi list!<div><br></div><div># Background</div><div><br></div><div>I am actively developing a NodeJS blockchain-based DNS resolver called <a href="https://github.com/okTurtles/dnschain">DNSChain</a> (which I've recently combined to work with a locally running instance of PowerDNS recursor).</div><div><br></div><div>While running DNSChain on its own I noticed that my logs suddenly started showing a large volume of timeouts happening in what was clearly some sort of attack. I temporarily enabled logging of IP addresses (for timeouts only) to see if I could figure out what was going on.</div><div><br></div><div># Attack Details</div><div><br></div><div>Here you can see an anonymized excerpt from that log. IPs have been changed but the mapping is consistent to show the pattern:</div><div><br></div><div><a href="https://gist.github.com/taoeffect/a74d3e302b06965036bf">https://gist.github.com/taoeffect/a74d3e302b06965036bf</a></div><div><br></div><div>Some notes from that:</div><div><br></div><div><ul class="MailOutline"><li>The first 3 octets were anonymized while the last octet was left intact.</li><li>It's clear there was mainly one target in that attack, what I've labelled the "666.666.666" block</li><li>They are using randomized subdomains to avoid cached queries</li><li>They include other IPs among those attacked (probably random) to try and disguise the attack, but the main focus is on 3 IPs in the "devil's block" ;-).</li></ul><div><br></div></div><div># PDNS 3.5.3 log</div><div><br></div><div>I decided to pair up DNSChain with PowerDNS recursor thinking that maybe since it has been in development for a long time now that it more effectively deal with this problem, however, it seems that it's only marginally doing so.</div><div><br></div><div>Here's the log:</div><div><br></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;"><div>May 31 14:58:38 pdns_recursor[666]: stats: 526156 questions, 33494 cache entries, 7006 negative entries, 2% cache hits</div><div>May 31 14:58:38 pdns_recursor[666]: stats: throttle map: 1369, ns speeds: 6</div><div>May 31 14:58:38 pdns_recursor[666]: stats: outpacket/query ratio 126%, 0% throttled, 0 no-delegation drops</div><div>May 31 14:58:38 pdns_recursor[666]: stats: 25 outgoing tcp connections, 19 queries running, 561421 outgoing timeouts</div><div>May 31 14:58:38 pdns_recursor[666]: stats: 253422 packet cache entries, 25% packet cache hits</div><div>May 31 14:58:38 pdns_recursor[666]: stats: 11 qps (average over 1956 seconds)</div><div>May 31 15:01:09 pdns_recursor[666]: [411B blob data]</div><div>May 31 15:01:14 pdns_recursor[666]: [411B blob data]</div><div>May 31 15:01:19 pdns_recursor[666]: [411B blob data]</div><div>May 31 15:05:52 pdns_recursor[666]: [411B blob data]</div><div>May 31 15:05:56 pdns_recursor[666]: [411B blob data]</div><div>May 31 15:06:01 pdns_recursor[666]: [411B blob data]</div><div>May 31 15:10:36 pdns_recursor[666]: [411B blob data]</div><div>May 31 15:10:41 pdns_recursor[666]: [411B blob data]</div></blockquote><div><br></div><div>Notice:</div><div><br></div><div><ul class="MailOutline"><li>0% throttled</li><li>2% cache hits</li><li>WTF is "[411B blob data]"? That's only started happened recently after it's been running for a while. It did not show this for the first hour of running.</li></ul></div><div><br></div><div># Questions</div><div><br></div><div>I'm currently running PowerDNS recursor 3.5.3-1 on Debian, because that is the most recent version available.</div><div><br></div><div>## Question 1</div><div><br></div><div>I saw that RC1 of 3.6 was released somewhere: <a href="http://mailman.powerdns.com/pipermail/pdns-users/2014-May/010657.html">http://mailman.powerdns.com/pipermail/pdns-users/2014-May/010657.html</a></div><div><br></div><div>But it's not available for Debian. How to install for Debian "correctly" when I've already installed the Debian package?</div><div><br></div><div>## Question 2</div><div><br></div><div>Does 3.6 mitigate this attack by itself?</div><div><br></div><div>## Question 3</div><div><br></div><div>See WTF is "[411B blob data]"? above.</div><div><br></div><div># Solution</div><div><br></div><div>The IPs obviously cannot be relied on for much of anything. Instead we can rely on the following:</div><div><br></div><div><ul class="MailOutline"><li>The fact that we're getting thousands of timeouts for a particular domain</li><li>The fact that a single domain is having hundreds/thousands of its subdomains being enumerated</li></ul></div><div><div><br class="webkit-block-placeholder"></div><div>Therefore this can be fairly trivially detected.</div><div><br></div><div>I'd prefer for PDNS recursor to do the detecting and mitigating itself, but I want a solution ASAP and don't want to wait, so if it doesn't already have an answer, then maybe a simple Lua script can be made to mitigate this.</div><div><br></div><div>The algorithm can be as simple as:</div><div><br></div><div>1. Is a domain having its subdomains enumerated recently? (100+ subdomains within say the past 30 minutes)</div><div>2. (Optional) are they returning timeouts too?</div><div><br></div><div>If so, silently drop the packets, give no response.</div><div><br></div><div>Thoughts?</div><div><br></div><div>Can this be done with 3.5.3? What would the Lua code look like? (Sorry, I'm new to PDNS).</div><div><br></div><div>Thanks!</div><div><br></div><div>Greg Slepak</div><div><br></div><div>P.S. This email was not proof-read because I'm short on time. Sorry for inevitable typos and grammar mistakes!</div><div>
<br class="Apple-interchange-newline"><span style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;">--</span><br style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;">Please do not email me anything that you are not comfortable also sharing</span><span style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;"> with the NSA.</span>
</div>
<br></div></div>_______________________________________________<br>Pdns-users mailing list<br><a href="mailto:Pdns-users@mailman.powerdns.com">Pdns-users@mailman.powerdns.com</a><br>http://mailman.powerdns.com/mailman/listinfo/pdns-users<br></blockquote></div><br></div></body></html>