[dnsdist] dnsdist[]: While reading a TCP question: accepting new connection on socket: Too many open files

[Fredrik Pettai]

Hi Jacob,

Thanks for your input and see my answers below (inline)

> On 26 Jul 2023, at 13:50, Jacob Bunk Nielsen via dnsdist <dnsdist at mailman.powerdns.com> wrote:
> 
> Fredrik Pettai via dnsdist <dnsdist at mailman.powerdns.com> writes:
> 
>> One dnsdist instance recently got overloaded, and the message (subject + below) appeared a lot in the logs:
>> 
>> 	“dnsdist[]: While reading a TCP question: accepting new connection on socket: Too many open files"
>> 
>> Is this only related to too much DNS-traffic over TCP, or could lots
>> of DNS traffic over UDP also potentially lead to slowdown/locking
>> issues for dnsdist TCP handling?
> 
> It's not just TCP, but also UDP. There's a good chance that you got hit
> by a DDOS attack and those tend to often be UDP based because it's much
> harder to spoof the source address of a TCP connection.

Yes, it was some kind of DoS.
But I’m still on holiday, so it was my colleagues that handled that..

>> I’ve increased the amount of addLocal() + newServer() workers to be able to handle more traffic.
> 
> This probably wasn't your problem since you managed to run out of
> available file descriptors just fine with the current number of
> addLocal() and newServer().

Ok

>> Dnsdist currently gets 16k fd’s (via systemctl's dnsdist.service configuration)
>> 
>> # grep -E '^Max open files' /proc/$(pidof dnsdist)/limits
>> Max open files            16384                16384                files
>> 
>> Would it be okay to increase this 4x or so?
> 
> That depends on your specific hardware, but probably, yes.

Ok, I’ve increased it to see if that helps dnsdist in the future.

>> What other things could one do to increase dnsdist ability to handle large bursts of DNS traffic better?
> 
> Have you checked out dynamic blocks? If not, have a look at https://dnsdist.org/guides/dynblocks.html

Yes, and we already have that in place.
Still, the descriptors ran out, so I guess dnsdist didn’t manage block all the incoming bogus packets in time…

How many packets/s is dnsdist able to handle? Should dnsdist be able to handle 100K packets/s at peaks with the proper settings?

Have a nice holiday,
/P
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: Message signed with OpenPGP
URL: <http://mailman.powerdns.com/pipermail/dnsdist/attachments/20230726/45006256/attachment.sig>