[Pdns-users] throttled-outqueries

Fri Oct 28 08:11:54 UTC 2016

Thanks David, 

Yes we have dnsdist in front of the recursor, and some scripts detecting those SERVFAILS and redirecting all those queries to the abuse pool(dnsdist) in which we have QPS restrictions. (all of that trying to deal with the SERVFAILs automatically )

But the fact is when I'm having throttled-outqueries my kernel logs keeps logging lots of those messages like 

Oct 28 09:54:22 rec01 kernel: [22546440.225357] net_ratelimit: 28 callbacks suppressed
Oct 28 09:54:27 rec01 kernel: [22546445.394524] net_ratelimit: 35 callbacks suppressed
Oct 28 09:54:33 rec01 kernel: [22546450.915227] net_ratelimit: 30 callbacks suppressed
Oct 28 09:54:38 rec01 kernel: [22546456.054420] net_ratelimit: 22 callbacks suppressed

Seems both terms related, because If I restart my recursor, those kernel messages stops for a while.

dnsdist                             1.1.0~beta1-1pdns.trusty
pdns-recursor                3.7.3-1

I'll try enabling full logging for the recursor ... I need information in deep.

Thanks a lot family. Any help will be much appreciated.

Ale.

-----Original Message-----
From: Pdns-users [mailto:pdns-users-bounces at mailman.powerdns.com] On Behalf Of David
Sent: martes, 11 de octubre de 2016 17:11
To: pdns-users at mailman.powerdns.com
Subject: Re: [Pdns-users] throttled-outqueries

On 2016-10-11 9:05 AM, Alejandro Adroher Mellado wrote:
> Hi all,t
>
> I'm interested in how can I investigate in deep the throttled out queries.
> We have an open recursor due to business needs and during the last 
> weeks we are answering a lot of SERVFAIL for random queries like 
> web-127.com ..... (all of the forwarded to our abuse pool server)
>

Your resolver is likely under subdomain reflection attacks. If you look at "rec_control current-queries" you will probably see lookups similar to sajasljkdasjdkl.web-127.com sdjksdjlk.web-127.com

etc. Pretty much anything that shows up on: 
https://twitter.com/dnsstream will be likely to hit your server as well.

You can either block abusive clients making these queries, or you can filter them from answering on your recursor. If all you have is powerdns and nothing like dnsdist in front you can setup forward zones for these pointing to yourself so that your recursor will respond with a servfail right away instead of doing the work trying to resolve the name. This will be a never ending cat and mouse game, though.

> From last service reboot I have this stats: (Recursor v 3.7.3)
> throttle-entries        390
> throttled-out   344055
> throttled-outqueries    344055
>
> We received 2.291.374 of SERVFAILS on last 10 days.
> The server performance it's fine!
>
> But, we are receiving on syslog a lot of entries like : kernel: 
> [21099878.651281] net_ratelimit: 153 callbacks suppressed We cannot be sure that both (SERVFAIL & kernel net_ratelimit) are related.
>
> Have anyone previous experience on this case?
>
> Thanks a lot.
>
> Alejandro
>
> _______________________________________________
> Pdns-users mailing list
> Pdns-users at mailman.powerdns.com
> https://mailman.powerdns.com/mailman/listinfo/pdns-users
>

_______________________________________________
Pdns-users mailing list
Pdns-users at mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/pdns-users