[Pdns-users] Mitigating / stopping recent Denial of Service Attacks
okTurtles
hi at okturtles.com
Sun Jun 1 20:58:45 UTC 2014
So on twitter I was pointed to this blog post about what seems to be about a related (or the same) attack:
http://blog.powerdns.com/2014/04/03/further-dos-guidance-packages-and-patches-available/
Now, I'm not clear on this however. Is the mitigation described in that link a solution to this problem?
I updated to 3.6RC1 and the good news is that the throttled percentage has gone up from 0% to 30%.
However, "rec_control current-queries" still shows that these queries are going out to Google's name servers:
# rec_control current-queries
8 currently outstanding questions
qname qtype remote tcp chained
abpqefghiwkyz.www.7098.com. A 8.8.4.4 n n
caqrjqahz.www.7098.com. A 8.8.4.4 n n
nxiwdmg.www.7098.com. A 8.8.4.4 n n
pfpfrpooqwo.www.7098.com. A 8.8.4.4 n n
aopdestuvjklm.www.7098.com. A 8.8.8.8 n n
nbpdefthvwklz.www.7098.com. A 8.8.8.8 n n
umochbiexah.www.7098.com. A 8.8.8.8 n n
xhvefhfmm.www.7098.com. A 8.8.8.8 n n
- done
10 currently outstanding questions
qname qtype remote tcp chained
lbhvjpk.www.7098.com. A 8.8.4.4 n n
rsoxyll.www.7098.com. A 8.8.4.4 n n
zkjlkmg.www.7098.com. A 8.8.4.4 n n
azj.www.7098.com. A 8.8.8.8 n n
ckyvaayyu.www.7098.com. A 8.8.8.8 n n
jcqrchk.www.7098.com. A 8.8.8.8 n n
nbpdrsguiwklm.www.7098.com. A 8.8.8.8 n n
nbpdrsguvwxyz.www.7098.com. A 8.8.8.8 n n
qbzuhuz.www.7098.com. A 8.8.8.8 n n
xbh.www.7098.com. A 8.8.8.8 n n
- done
Shouldn't these queries be dropped simply?
Running the command again shows a completely different set of queries, telling me that the queries are not being dropped as I would expect.
So... is mitigation working or not?
Thanks,
Greg
--
Please do not email me anything that you are not comfortable also sharing with the NSA.
On May 31, 2014, at 2:23 PM, okTurtles <hi at okturtles.com> wrote:
> Hi list!
>
> # Background
>
> I am actively developing a NodeJS blockchain-based DNS resolver called DNSChain (which I've recently combined to work with a locally running instance of PowerDNS recursor).
>
> While running DNSChain on its own I noticed that my logs suddenly started showing a large volume of timeouts happening in what was clearly some sort of attack. I temporarily enabled logging of IP addresses (for timeouts only) to see if I could figure out what was going on.
>
> # Attack Details
>
> Here you can see an anonymized excerpt from that log. IPs have been changed but the mapping is consistent to show the pattern:
>
> https://gist.github.com/taoeffect/a74d3e302b06965036bf
>
> Some notes from that:
>
> The first 3 octets were anonymized while the last octet was left intact.
> It's clear there was mainly one target in that attack, what I've labelled the "666.666.666" block
> They are using randomized subdomains to avoid cached queries
> They include other IPs among those attacked (probably random) to try and disguise the attack, but the main focus is on 3 IPs in the "devil's block" ;-).
>
> # PDNS 3.5.3 log
>
> I decided to pair up DNSChain with PowerDNS recursor thinking that maybe since it has been in development for a long time now that it more effectively deal with this problem, however, it seems that it's only marginally doing so.
>
> Here's the log:
>
> May 31 14:58:38 pdns_recursor[666]: stats: 526156 questions, 33494 cache entries, 7006 negative entries, 2% cache hits
> May 31 14:58:38 pdns_recursor[666]: stats: throttle map: 1369, ns speeds: 6
> May 31 14:58:38 pdns_recursor[666]: stats: outpacket/query ratio 126%, 0% throttled, 0 no-delegation drops
> May 31 14:58:38 pdns_recursor[666]: stats: 25 outgoing tcp connections, 19 queries running, 561421 outgoing timeouts
> May 31 14:58:38 pdns_recursor[666]: stats: 253422 packet cache entries, 25% packet cache hits
> May 31 14:58:38 pdns_recursor[666]: stats: 11 qps (average over 1956 seconds)
> May 31 15:01:09 pdns_recursor[666]: [411B blob data]
> May 31 15:01:14 pdns_recursor[666]: [411B blob data]
> May 31 15:01:19 pdns_recursor[666]: [411B blob data]
> May 31 15:05:52 pdns_recursor[666]: [411B blob data]
> May 31 15:05:56 pdns_recursor[666]: [411B blob data]
> May 31 15:06:01 pdns_recursor[666]: [411B blob data]
> May 31 15:10:36 pdns_recursor[666]: [411B blob data]
> May 31 15:10:41 pdns_recursor[666]: [411B blob data]
>
> Notice:
>
> 0% throttled
> 2% cache hits
> WTF is "[411B blob data]"? That's only started happened recently after it's been running for a while. It did not show this for the first hour of running.
>
> # Questions
>
> I'm currently running PowerDNS recursor 3.5.3-1 on Debian, because that is the most recent version available.
>
> ## Question 1
>
> I saw that RC1 of 3.6 was released somewhere: http://mailman.powerdns.com/pipermail/pdns-users/2014-May/010657.html
>
> But it's not available for Debian. How to install for Debian "correctly" when I've already installed the Debian package?
>
> ## Question 2
>
> Does 3.6 mitigate this attack by itself?
>
> ## Question 3
>
> See WTF is "[411B blob data]"? above.
>
> # Solution
>
> The IPs obviously cannot be relied on for much of anything. Instead we can rely on the following:
>
> The fact that we're getting thousands of timeouts for a particular domain
> The fact that a single domain is having hundreds/thousands of its subdomains being enumerated
>
> Therefore this can be fairly trivially detected.
>
> I'd prefer for PDNS recursor to do the detecting and mitigating itself, but I want a solution ASAP and don't want to wait, so if it doesn't already have an answer, then maybe a simple Lua script can be made to mitigate this.
>
> The algorithm can be as simple as:
>
> 1. Is a domain having its subdomains enumerated recently? (100+ subdomains within say the past 30 minutes)
> 2. (Optional) are they returning timeouts too?
>
> If so, silently drop the packets, give no response.
>
> Thoughts?
>
> Can this be done with 3.5.3? What would the Lua code look like? (Sorry, I'm new to PDNS).
>
> Thanks!
>
> Greg Slepak
>
> P.S. This email was not proof-read because I'm short on time. Sorry for inevitable typos and grammar mistakes!
>
> --
> Please do not email me anything that you are not comfortable also sharing with the NSA.
>
> _______________________________________________
> Pdns-users mailing list
> Pdns-users at mailman.powerdns.com
> http://mailman.powerdns.com/mailman/listinfo/pdns-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.powerdns.com/pipermail/pdns-users/attachments/20140601/f801165a/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mailman.powerdns.com/pipermail/pdns-users/attachments/20140601/f801165a/attachment.sig>
More information about the Pdns-users
mailing list