[dnsdist] Erratic CPU Usage

[Remi Gacogne]

On 12/24/20 10:28 AM, Jahanzeb Arshad via dnsdist wrote:
> Following is the configuration file. The strace output is attached here
> 
> https://drive.google.com/file/d/1lRBNr6PB83zdbfMVtSKbzZ5ZKipy140f/view?usp=sharing 
> <https://drive.google.com/file/d/1lRBNr6PB83zdbfMVtSKbzZ5ZKipy140f/view?usp=sharing>

Thanks! I don't see anything unusual in that trace, unfortunately. Was 
it taken during the time where the CPU usage was unusually high? How is 
that CPU usage measured? Do you observe any issue during that time, like 
some queries being dropped or the latency increasing?

Looking at the metrics you are sending to our public metronome 
(https://metronome1.powerdns.com/?server=dnsdist.NTL-DNSLB-GD-01.main&beginTime=-604800), 
I don't see anything unusual and the CPU usage seems quite stable.

> -- Enable DNS query caching
> -- 300K entries with avg 512B would take about 150MB of RAM
> pc = newPacketCache(300000, {maxTTL=86400, minTTL=0, 
> temporaryFailureTTL=60, staleTTL=60, dontAge=false})
> getPool(""):setCache(pc)

Just a tuning advice, you seem to have several threads so it would be a 
good idea to use more than one shard in the packet cache to reduce 
contention between threads. You can do that by passing numberOfShards=10 
(for example) to the new newPacketCache() command, see 
https://dnsdist.org/reference/config.html#newPacketCache

It might also be a good idea to use more than one shard in the ring 
buffers, by passing 10 (for example) as a second parameter to the 
setRingBuffersSize() command:

https://dnsdist.org/reference/config.html#setRingBuffersSize

You are using the default size so setRingBuffersSize(10000, 10) would 
only change the number of shards, and will likely improve performance a bit.

Best regards,

Remi

> On Thu, 2020-12-24 at 09:56 +0100, Remi Gacogne via dnsdist wrote:
>> Hi,
>>
>> On 12/24/20 7:25 AM, Jahanzeb Arshad via dnsdist wrote:
>>> We have deployed two instances of dnsdist v1.5.1 on CentOS 7.9. After
>>> running for 7-8 days both the machines start showing erratic CPU usage
>>> pattern. The CPU usage jumps to 40% then 0 and the servers keep on doing
>>> this. If the process is restarted this is fixed. Need some help to
>>> identify and fix this issue.
>>
>> Would you mind posting the dnsdist configuration you are using, after
>> redacting passwords and API keys, of course?
>>
>> It would also be very helpful if you could do a "strace -f -p <pid of
>> dnsdist>" for a few seconds after the process has started acting out.
>> Getting a backtrace might also help, which can be done by attaching a
>> debugger via "gdb -p <pid of dnsdist>" then issuing "thread apply all bt
>> full".
>>
>> Best regards,
> 
> _______________________________________________
> dnsdist mailing list
> dnsdist at mailman.powerdns.com
> https://mailman.powerdns.com/mailman/listinfo/dnsdist
> 


-- 
Remi Gacogne
PowerDNS.COM BV - https://www.powerdns.com/