<div><div><div>Hi all,</div><div><div class="gmail_quote"><div dir="ltr"><div><br></div><div>We have been using dnsdist for a couple years and it works pretty nice for us.</div><div><br></div><div>In our situation dnsdist is installed in front of a powerdns server which host our domains names (250k) as an authoritative dns server.<br></div><div><br></div><div>dnsdist is used to block unwanted request (e.g. when a domain is getting a ddos with nxdomains request, it automatically creates a whitelist for existing requests and blocks all other traffic for that domain).</div><div><br></div><div>To make things even better (and handle more requests) we are looking into the setup and try to make it perform better. We have switched to a bind backend for the domains which are getting a ddos. As far as we have seen it is better at responding to nxdomain requests because of a lack of a database query.</div><div><br></div><div>Currently we are stuck at a max of +/- 200k qps for nxdomain requests and want to be able to serve +/- 300k qps per server.</div><div><br></div><div>We have done the following:</div><div>- added multiple (6x the amount of cores) addLocal listeners for IPv4 and IPv6, with the options reusePort=true and tcpFastOpenQueueSize=100</div><div>- add multiple (2x the amount of cores) newServer to the backend, with the options tcpFastOpen=true and sockets=(2x the amount of cores)</div><div>- setMaxTCPClientThreads(1000)</div><div>- stop using connection tracking in the firewall</div><div>- run dndist + powerdns on a single machine (8 cores and 16GB ram) and a bind backend on two seperate servers (also 8 cores and 16GB ram)<br></div><div><br></div><div>And the defaults like caching requests (which doesn't work for nxdomain) and limit the amount of qps per ip (which also doens't work for nxdomain attack because they use public resolvers).</div><div><br></div><div>On a attack we have the following dnsdist rules (the abuse pool is connected to the bind backend and the all pool is the powerdns backend)<br></div><div><span style="font-family:monospace"># dnsdist --client -e 'showRules()';</span></div><div><span style="font-family:monospace"># Name Matches Rule Action<br>0 29558 qname in <attacked domain>. to pool abuse<br>1 1346 (opcode==4) || (opcode==5) || (qtype==AXFR) || (qtype==IXFR) set rcode 5<br>2 11961 IP (/32, /64) match for QPS over 100 burst 100 delay by 100 msec<br>3 44931079 All to pool all</span></div><div><br></div><div>When we simulate a nxdomain attack (with 200k qps and 500MBit of traffic) , we get a high load on the dnsdist server (50% CPU for dsndist and a lot of interrupts and context switches).</div><div><br></div><div>The network connection is not the bottleneck, it is able to do 10G of traffic, if we only use bind as a backend (so skip dnsdist) it also is able to serve those numbers.<br></div><div><br></div><div>So the question from me to you are:</div><div>- how much qps are you able to push through dnsdist using a powerdns or bind backend</div><div>- have I overlooked some tuning parameters, e.g. more kernel parameters or some dnsdist parameters</div><div>- how can we get more insights into what dnsdist is doing, which metrics are most usefull to us. I have seen the metrics (<a href="https://dnsdist.org/statistics.html" target="_blank">https://dnsdist.org/statistics.html</a>) and keeping an eye on those metrics.<br></div><div>- what is the best method of sending packets for a domain to a seperate backend, right we now we use 'addAction("<domain>", PoolAction("abuse")), but is this the least CPU intensive one? Are there better methods?<br></div><div><br></div><div>I have seen eBPF socket filtering, but as far as I have seen that is for dropping unwanted packets.</div><div><br></div><div>Please let me know if you have more questions, happy to answer them.<br></div><div><br></div><div>Kind regards,</div><div><br></div><div>Jasper Aikema<br></div></div>
</div></div>
</div>
</div>