[Pdns-users] How to stop automatic respawning

[Stefan Schmidt]

October 6, 2021 10:28 PM, "Klaus Darilion via Pdns-users" <pdns-users at mailman.powerdns.com> wrote:

> Hi! During random subdomain attacks we often hit the max-queue-length:

Hey there,
 
> pdns_server-customer1[51284]: 5001 questions waiting for database/backend attention. Limit is 5000,
> respawning
> 
> This happens constantly (+50K q/s). Of course we try to filter with dnsdist/iptables/... but until
> our detection and dynamic filtering kicks in, the attacks fully hits PDNS and causes constant
> respawning. I think this respawning causes aditional load, backend DOWN/UP in dnsdist, outages
> during " Service hold-off time over".
> 
> When the queue is full, instead of respawning, I would like PowerDNS to:
> - just stop reading from incoming sockets (Linux input buffer will overflow and packets will be
> dropped), or
> - read packets from incoming sockets and discard them, or
> - just flush the queue, or
> - any other ideas?
> 
> Are there any options to avoid respawning but keep the queue-length at a sane value?

We have had success with setting overload-queue-length [1] to a value below max-queue-length [2] but of course as per its description the drawback is: "If this many packets are waiting for database attention, answer any new questions strictly from the packet cache." so the answers in this overload situation might not be exactly what you would want/need them to be.

Since dnsdist is in the loop you might be able to offload queries over the capacity of your PowerDNS Server instance to another one that is backed by an in memory database that gets updated less frequently or even only has a subset of data kind of like a sorry server in the HTTP world.
Obviously the latter is especially tricky as you would want to avoid answering with NXDOMAIN or higher TTL records for where you have gaps in data coverage.

As for PowerDNS Server doing anything other than respawning
https://github.com/PowerDNS/pdns/blob/b6cfaec150374684c341cb6475fea34dd8882076/pdns/distributor.hh#L325
does not look promising i fear.

regards,

 Stefan

[1] https://doc.powerdns.com/authoritative/settings.html#overload-queue-length
[2] https://doc.powerdns.com/authoritative/settings.html#max-queue-length