[dnsdist] Suggestions for rules to block abusive traffic

[Dan McCombs]

Hey Jeff,

Thanks for the info, and it's good to hear from you since our days at Dyn
together!

That topQueries call is a good way to quickly find this kind of attack when
it accounts for the majority of traffic like we've been seeing.

We've considered using Salt to manage our DNS infra but lack of
time/resources and internal headwinds have stopped us. I assume you've
liked it, have you run into any notable issues and how does it handle
limiting the number of dnsdist instances in a single DC that get restarted
with a configuration change so they don't all restart at once?

Take care,

-Dan


Dan McCombs
Senior Engineer I - DNS
dmccombs at digitalocean.com


On Fri, Dec 29, 2023 at 4:39 PM Jeffrey Cronstrom <
jeff.cronstrom at cloudfloordns.com> wrote:

> Hi Dan,
>
> We see these types of attacks on a regular basis anywhere from 5 minute
> bursts to days at a time and have been seeing them for quite some time. We
> do not have anything that dynamically handles any of this yet but rather
> update the dnsdist configs on an as needed basis. We simply add a domain
> name to a list and run a given Salt State (we use SaltStack for config
> management) across the DNS network and this mitigates quickly against the
> domain name and helps with any backend impact.
>
> addAction("example.com.", QPSAction(50))
>
> On the automated front, we've started to develop a program to run against
> the dnsdist API so we can evaluate the current QPS and upon a
> certain deviation of that metric we can look at a domain name that may be
> getting impacted by a flood of queries and automatically apply a temporary
> rule. This command would be run remotely against all dnsdist instances.
> This is only conceptual and not in use but is the start of an outline for a
> program to help with this type of activity.
>
> disthost = "http://%s:8083/jsonstat?command=stats" % (IPv4)
> https://dnsdist.org/guides/webserver.html
>
> From this lookup we can establish the deviation of QPS and a rule can be
> applied locally via the dnsdist API and also removed after any offending
> traffic has subsided. The rule would also be dropped upon a restart because
> it wouldn't be part of the startup configuration of dnsdist.
>
> Some local instance calls for evaluation.
> /usr/bin/dnsdist -e 'topQueries(10,2)' - example.com
> /usr/bin/dnsdist -e 'topQueries(10,3)' - example.co.uk
> /usr/bin/dnsdist -e 'showRules()'
> /usr/bin/dnsdist -e 'addAction("example.com.", QPSAction(50))'
> /usr/bin/dnsdist -e 'mvRuleToTop()'
>
> One thing to note is the domain getting hit will be affected by limiting
> their QPS but it will help the backend from further impact.
>
> This may not be exactly what you are looking for but may help and someone
> else on this list might have better suggestions however I thought I would
> share.
>
> Best Regards,
>
> Jeff
>
>
>
> On Fri, Dec 29, 2023, 14:11 Dan McCombs via dnsdist <
> dnsdist at mailman.powerdns.com> wrote:
>
>> Hi all,
>>
>> I'm wondering if anyone has suggestions of reasonable ways to handle this
>> type of abusive traffic with dnsdist.
>>
>> We've had on and off attacks recently targeting legitimate domains
>> delegated to our authoritative service flooding queries for random
>> subdomains of varying length and characters/words. i.e. 12345.example.com,
>> fred.example.com, abc178371jd.example.com, where example.com is a
>> different domain we're authoritative for each attack.
>>
>> The dnsdist nodes can handle the traffic, but breaking cache and going
>> through to our backends is having more of an impact.
>>
>> We have thousands of domains, so it doesn't seem reasonable to apply
>> individual rate limits to them all, but if there is a straight forward way
>> to do something like that I'd be happy to hear it. The source addresses are
>> well known public resolvers that we shouldn't rate limit either.
>>
>> I'm wondering if there's any way to detect and apply a rule dynamically
>> to respond to queries for one of these domains without affecting the source
>> IP address entirely, and not require us to manually add a rule for each
>> domain as it occurs.
>>
>> Any ideas would be appreciated.
>>
>> Take care,
>>
>> -Dan
>>
>>
>> Dan McCombs
>> Senior Engineer I - DNS
>> dmccombs at digitalocean.com
>> _______________________________________________
>> dnsdist mailing list
>> dnsdist at mailman.powerdns.com
>> https://mailman.powerdns.com/mailman/listinfo/dnsdist
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.powerdns.com/pipermail/dnsdist/attachments/20240102/d124d211/attachment.htm>