[Pdns-users] DDOS prevents pdns-recursor from responding regular queries

[Rygl Aleš]

Hi all,

I would like to kindly ask you for an advice. Recently we were facing a DNS oriented DDOS attack. Unfortunately it is not a usual 
amplification DDOS which could be easily handled by packet cache. The attack pattern is following:

 0.049862 62.141.28.21 -> a.b.c.d DNS Standard query A nbpqrfthvwxyz.aa.cp375.com
 0.049872 62.141.28.21 -> a.b.c.d DNS Standard query A nbpqrfthvwxyz.aa.cp375.com
 0.887789 62.141.31.87 -> a.b.c.d DNS Standard query A mvidglgdejydyxmz.aa.cp375.com
 0.887797 62.141.31.87 -> a.b.c.d DNS Standard query A mvidglgdejydyxmz.aa.cp375.com
 1.224863 62.141.18.252 -> a.b.c.d DNS Standard query A aopdesghvjkym.aa.cp375.com
 1.224868 62.141.18.252 -> a.b.c.d DNS Standard query A aopdesghvjkym.aa.cp375.com
 1.308025 62.141.23.141 -> a.b.c.d DNS Standard query A hjntnfu.aa.cp375.com
 1.308034 62.141.23.141 -> a.b.c.d DNS Standard query A hjntnfu.aa.cp375.com
 1.854975 62.141.28.25 -> a.b.c.d DNS Standard query A zdssjffzy.aa.cp375.com
 1.854983 62.141.28.25 -> a.b.c.d DNS Standard query A zdssjffzy.aa.cp375.com
 2.573946 62.141.23.231 -> a.b.c.d DNS Standard query A ljboehume.aa.cp375.com
 2.573958 62.141.23.231 -> a.b.c.d DNS Standard query A ljboehume.aa.cp375.com
 3.093115 62.141.23.231 -> a.b.c.d DNS Standard query A ljboehume.aa.cp375.com
 3.093120 62.141.23.231 -> a.b.c.d DNS Standard query A ljboehume.aa.cp375.com
 3.167982 62.141.31.201 -> a.b.c.d DNS Standard query A nocqeftuijxlz.aa.cp375.com
 3.167990 62.141.31.201 -> a.b.c.d DNS Standard query A nocqeftuijxlz.aa.cp375.com
 3.294747 62.141.18.252 -> a.b.c.d DNS Standard query A aopdesghvjkym.aa.cp375.com
 3.294754 62.141.18.252 -> a.b.c.d DNS Standard query A aopdesghvjkym.aa.cp375.com

There are random hostnames generated and the domain seemed to be existing at the time of the attack. The recursor was answering "Server 
Failure". Normally we have about 100 concurrent queries running but when this happened we had about 1000 in peaks about 2000. There is 
a pcap file of the traffic during the attack available (100kpkts). Due to random hostnames the caches were ineffective.

The problem is that while waiting for answers from superior NS the recursor was hardly able to answer regular queries (google.com) and 
we had complains from customers. Using netstat -an showed a number of opened UDP connections towards the authoritative NS. But it was 
either already down or throttling or they blocked us which was made it even worse.

Our setup consists of LVS balancer and 4 real servers running PowerDNS recursor and Unbound. We run Debian Wheezy, kernel 2.6.32-5-
amd64, Recursor 3.5.3-1., 96 GiB RAM, 16 core machines There was about 17.000 req/sec per server in the moment of the attack. Recursor 
has folowing values in the config:

max-cache-entries=200000000
max-mthreads=4096
max-negative-ttl=1800
max-packetcache-entries=30000000
network-timeout=1500
packetcache-servfail-ttl=300
quiet=yes
setgid=pdns
setuid=pdns
threads=16


Unbound is able to handle this situation without any problems and there is no impact to regular queries...

It there a way how to face such attacks with PowerDNS? 

Many thanks

Ales


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.powerdns.com/pipermail/pdns-users/attachments/20140205/df90bded/attachment.html>