[Pdns-users] PowerDNS issues
zaphodb at zaphods.net
Wed Sep 22 15:36:22 UTC 2021
September 22, 2021 3:03 PM, "Andrey Sedletsky via Pdns-users" <pdns-users at mailman.powerdns.com>
> Good afternoon!
> After restarting the pdns-recursor process, the number of "outgoing
> query timeout" and "over capacity drops" sharply increases, which leads
> to serious degradation of the service.
> This behavior manifests itself at times of high load on the server (more
> than 400 thousand requests per second). With a lower load, restarting
> the process does not lead to such consequences.
Have you considered the possibility that 400 thousand queries per second is a load that is taxing your server to the brink of resource exhaustion? That sure is a lot of queries. According to https://pc.nanog.org/static/published/meetings/NANOG77/2142/20191029_Spacek_Lightning_Talk_Dns_v2.pdf they were able to achieve a lot less than that in 2019.
> We are interested in what could be the reason for this behavior
Upon the hunch that your setup might be in an overload scenario i followed 'over-capacity-drops' in the code and ended up at
https://github.com/PowerDNS/pdns/blob/97a4cff6fc7b3da1ff44d42b950cfc17d2fd95cf/pdns/pdns_recursor.cc#L3146 so it seems that you have exhausted your thread capacity when that happens. See https://doc.powerdns.com/recursor/performance.html on how to tune the recursor however if that is not benchmark traffic but real world i would strongly suggest getting more servers installed.
The SERVFAIL response is just what i would expect in such a case. See https://www.rfc-editor.org/rfc/rfc1035.html#section-4.1.1 .
More information about the Pdns-users