[Pdns-users] throttled-outqueries

[David]

On 2016-10-28 2:11 AM, Alejandro Adroher Mellado wrote:
> Thanks David,
>
> Yes we have dnsdist in front of the recursor, and some scripts detecting those SERVFAILS and redirecting all those queries to the abuse pool(dnsdist) in which we have QPS restrictions. (all of that trying to deal with the SERVFAILs automatically )
>
> But the fact is when I'm having throttled-outqueries my kernel logs keeps logging lots of those messages like
>
> Oct 28 09:54:22 rec01 kernel: [22546440.225357] net_ratelimit: 28 callbacks suppressed
> Oct 28 09:54:27 rec01 kernel: [22546445.394524] net_ratelimit: 35 callbacks suppressed
> Oct 28 09:54:33 rec01 kernel: [22546450.915227] net_ratelimit: 30 callbacks suppressed
> Oct 28 09:54:38 rec01 kernel: [22546456.054420] net_ratelimit: 22 callbacks suppressed

Is this "trusty" ubuntu? Is it logging to the journal or do you use 
syslog or rsyslog? I suspect one of these is limiting the amount of logs 
it's writing, but I can't say I've seen that exact message before.

>
> Seems both terms related, because If I restart my recursor, those kernel messages stops for a while.
>
> dnsdist                             1.1.0~beta1-1pdns.trusty
> pdns-recursor                3.7.3-1

IIRC log-common-errors was enabled by default on 3.7.x - which will 
produce a lot of log entries already for these abusive queries. As a 
second note, 4.x handles these much better. We used to have queries that 
would spawn 50 recursive lookups (the max) but now with 4.x they stop 
much quicker.

>
> I'll try enabling full logging for the recursor ... I need information in deep.
>



> Thanks a lot family. Any help will be much appreciated.
>
> Ale.
>
> -----Original Message-----
> From: Pdns-users [mailto:pdns-users-bounces at mailman.powerdns.com] On Behalf Of David
> Sent: martes, 11 de octubre de 2016 17:11
> To: pdns-users at mailman.powerdns.com
> Subject: Re: [Pdns-users] throttled-outqueries
>
> On 2016-10-11 9:05 AM, Alejandro Adroher Mellado wrote:
>> Hi all,t
>>
>> I'm interested in how can I investigate in deep the throttled out queries.
>> We have an open recursor due to business needs and during the last
>> weeks we are answering a lot of SERVFAIL for random queries like
>> web-127.com ..... (all of the forwarded to our abuse pool server)
>>
>
>
> Your resolver is likely under subdomain reflection attacks. If you look at "rec_control current-queries" you will probably see lookups similar to sajasljkdasjdkl.web-127.com sdjksdjlk.web-127.com
>
> etc. Pretty much anything that shows up on:
> https://twitter.com/dnsstream will be likely to hit your server as well.
>
> You can either block abusive clients making these queries, or you can filter them from answering on your recursor. If all you have is powerdns and nothing like dnsdist in front you can setup forward zones for these pointing to yourself so that your recursor will respond with a servfail right away instead of doing the work trying to resolve the name. This will be a never ending cat and mouse game, though.
>
>
>> From last service reboot I have this stats: (Recursor v 3.7.3)
>> throttle-entries        390
>> throttled-out   344055
>> throttled-outqueries    344055
>>
>> We received 2.291.374 of SERVFAILS on last 10 days.
>> The server performance it's fine!
>>
>> But, we are receiving on syslog a lot of entries like : kernel:
>> [21099878.651281] net_ratelimit: 153 callbacks suppressed We cannot be sure that both (SERVFAIL & kernel net_ratelimit) are related.
>>
>> Have anyone previous experience on this case?
>>
>> Thanks a lot.
>>
>> Alejandro
>>
>> _______________________________________________
>> Pdns-users mailing list
>> Pdns-users at mailman.powerdns.com
>> https://mailman.powerdns.com/mailman/listinfo/pdns-users
>>
>
> _______________________________________________
> Pdns-users mailing list
> Pdns-users at mailman.powerdns.com
> https://mailman.powerdns.com/mailman/listinfo/pdns-users
>