[dnsdist] Suggestions for rules to block abusive traffic

Tue Jan 2 19:50:44 UTC 2024

Hey Dan,

Yes, the old Dyn days!

topQueries works well for an automated way to find the top offending
traffic and we also use dnstop which works very nicely for the same
thing plus a lot of other functionality to see what's going on with queries.

Salt works very well for updating dnsdist configs and reloading them. We
currently run 2 different instances of dnsdist per server because we have
multiple backends to different dns platforms. So far we haven't run into
any issues with controlling instances individually and we never reload all
at the same time in case of any config issues. There is always a test
system to update and check validity.

If you ever want to chat more on this just contact me off list and we can
talk some more about it.

Cheers,
Jeff

On Tue, Jan 2, 2024 at 10:52 AM Dan McCombs <dmccombs at digitalocean.com>
wrote:

> Hey Jeff,
>
> Thanks for the info, and it's good to hear from you since our days at Dyn
> together!
>
> That topQueries call is a good way to quickly find this kind of attack
> when it accounts for the majority of traffic like we've been seeing.
>
> We've considered using Salt to manage our DNS infra but lack of
> time/resources and internal headwinds have stopped us. I assume you've
> liked it, have you run into any notable issues and how does it handle
> limiting the number of dnsdist instances in a single DC that get restarted
> with a configuration change so they don't all restart at once?
>
> Take care,
>
> -Dan
>
>
> Dan McCombs
> Senior Engineer I - DNS
> dmccombs at digitalocean.com
>
>
> On Fri, Dec 29, 2023 at 4:39 PM Jeffrey Cronstrom <
> jeff.cronstrom at cloudfloordns.com> wrote:
>
>> Hi Dan,
>>
>> We see these types of attacks on a regular basis anywhere from 5 minute
>> bursts to days at a time and have been seeing them for quite some time. We
>> do not have anything that dynamically handles any of this yet but rather
>> update the dnsdist configs on an as needed basis. We simply add a domain
>> name to a list and run a given Salt State (we use SaltStack for config
>> management) across the DNS network and this mitigates quickly against the
>> domain name and helps with any backend impact.
>>
>> addAction("example.com.", QPSAction(50))
>>
>> On the automated front, we've started to develop a program to run against
>> the dnsdist API so we can evaluate the current QPS and upon a
>> certain deviation of that metric we can look at a domain name that may be
>> getting impacted by a flood of queries and automatically apply a temporary
>> rule. This command would be run remotely against all dnsdist instances.
>> This is only conceptual and not in use but is the start of an outline for a
>> program to help with this type of activity.
>>
>> disthost = "http://%s:8083/jsonstat?command=stats" % (IPv4)
>> https://dnsdist.org/guides/webserver.html
>>
>> From this lookup we can establish the deviation of QPS and a rule can be
>> applied locally via the dnsdist API and also removed after any offending
>> traffic has subsided. The rule would also be dropped upon a restart because
>> it wouldn't be part of the startup configuration of dnsdist.
>>
>> Some local instance calls for evaluation.
>> /usr/bin/dnsdist -e 'topQueries(10,2)' - example.com
>> /usr/bin/dnsdist -e 'topQueries(10,3)' - example.co.uk
>> /usr/bin/dnsdist -e 'showRules()'
>> /usr/bin/dnsdist -e 'addAction("example.com.", QPSAction(50))'
>> /usr/bin/dnsdist -e 'mvRuleToTop()'
>>
>> One thing to note is the domain getting hit will be affected by limiting
>> their QPS but it will help the backend from further impact.
>>
>> This may not be exactly what you are looking for but may help and someone
>> else on this list might have better suggestions however I thought I would
>> share.
>>
>> Best Regards,
>>
>> Jeff
>>
>>
>>
>> On Fri, Dec 29, 2023, 14:11 Dan McCombs via dnsdist <
>> dnsdist at mailman.powerdns.com> wrote:
>>
>>> Hi all,
>>>
>>> I'm wondering if anyone has suggestions of reasonable ways to handle
>>> this type of abusive traffic with dnsdist.
>>>
>>> We've had on and off attacks recently targeting legitimate domains
>>> delegated to our authoritative service flooding queries for random
>>> subdomains of varying length and characters/words. i.e.
>>> 12345.example.com, fred.example.com, abc178371jd.example.com, where
>>> example.com is a different domain we're authoritative for each attack.
>>>
>>> The dnsdist nodes can handle the traffic, but breaking cache and going
>>> through to our backends is having more of an impact.
>>>
>>> We have thousands of domains, so it doesn't seem reasonable to apply
>>> individual rate limits to them all, but if there is a straight forward way
>>> to do something like that I'd be happy to hear it. The source addresses are
>>> well known public resolvers that we shouldn't rate limit either.
>>>
>>> I'm wondering if there's any way to detect and apply a rule dynamically
>>> to respond to queries for one of these domains without affecting the source
>>> IP address entirely, and not require us to manually add a rule for each
>>> domain as it occurs.
>>>
>>> Any ideas would be appreciated.
>>>
>>> Take care,
>>>
>>> -Dan
>>>
>>>
>>> Dan McCombs
>>> Senior Engineer I - DNS
>>> dmccombs at digitalocean.com
>>> _______________________________________________
>>> dnsdist mailing list
>>> dnsdist at mailman.powerdns.com
>>> https://mailman.powerdns.com/mailman/listinfo/dnsdist
>>>
>>

-- 

Jeffrey Cronstrom
CloudFloorDNS <https://www.cloudfloordns.com/>, LLC
M - +1.978.835.3617
O - +1.781.819.5172

Schedule a Meeting
https://calendly.com/jeff-cronstrom
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.powerdns.com/pipermail/dnsdist/attachments/20240102/a2bbc1ce/attachment-0001.htm>