[dnsdist] dnsdist[29321]: Marking downstream IP:53 as 'down'

[Remi Gacogne]

Hi,

 > We have configured dnsdist instance to handle around 500k QPS, but we
 > are seeing downstream down frequently once QPS reached above 25k. below
 > are the logs which we found to relative issue.
 >
 > dnsdist[29321]: Marking downstream server1 IP:53 as 'down'
 >
 > dnsdist[29321]: Marking downstream server2 IP:53 as 'down'

You might be able to get more information about why the health-checks 
are failing by adding setVerboseHealthChecks(true) to your configuration.

It usually happens because the backend is overwhelmed and needs to be 
tuned to handle the load, but it might also be caused by a network 
issue, like a link reaching its maximum capacity, or by dnsdist itself 
being overwhelmed and needing tuning (like increasing the number of 
newServer() directives, see [1]).

[1]: 
https://dnsdist.org/advanced/tuning.html#udp-and-incoming-dns-over-https

Best regards,
-- 
Remi Gacogne
PowerDNS.COM BV - https://www.powerdns.com/