[dnsdist] Erratic CPU Usage

[Jahanzeb Arshad]

There is apparently no issue with the performance or query drops. I am
just concerned that we are increasing the load on these machines and we
might run into some problems with increased load/traffic. Currently we
are getting about 5K QPS and we are expecting about 20K QPS per server.

The CPU spikes/dips are shown in the webconsole of dnsdist. Following
are two snapshots of the servers we are using. First one is showing
this behaviour.

https://drive.google.com/file/d/1OfWUiP5G2ZKpXLOPYfMSz4NZXiUi6ZNA/view?usp=sharing

https://drive.google.com/file/d/1c6QiBqnJLuW3os5HeQSlMJw_VVrQC0SZ/view?usp=sharing


Thankyou for your tuning tips. Have applied the configuration changes
and restarted the application. Now the CPU usage is smooth.

Regards

Jahanzeb


On Thu, 2020-12-24 at 12:04 +0100, Remi Gacogne via dnsdist wrote:
> On 12/24/20 10:28 AM, Jahanzeb Arshad via dnsdist wrote:
> > Following is the configuration file. The strace output is attached
> > here
> > 
> > https://drive.google.com/file/d/1lRBNr6PB83zdbfMVtSKbzZ5ZKipy140f/view?usp=sharing
> >  
> > <https://drive.google.com/file/d/1lRBNr6PB83zdbfMVtSKbzZ5ZKipy140f/view?usp=sharing
> > >
> 
> Thanks! I don't see anything unusual in that trace, unfortunately.
> Was 
> it taken during the time where the CPU usage was unusually high? How
> is 
> that CPU usage measured? Do you observe any issue during that time,
> like 
> some queries being dropped or the latency increasing?
> 
> Looking at the metrics you are sending to our public metronome 
> (https://metronome1.powerdns.com/?server=dnsdist.NTL-DNSLB-GD-01.main&beginTime=-604800
> ), 
> I don't see anything unusual and the CPU usage seems quite stable.
> 
> > -- Enable DNS query caching
> > -- 300K entries with avg 512B would take about 150MB of RAM
> > pc = newPacketCache(300000, {maxTTL=86400, minTTL=0, 
> > temporaryFailureTTL=60, staleTTL=60, dontAge=false})
> > getPool(""):setCache(pc)
> 
> Just a tuning advice, you seem to have several threads so it would be
> a 
> good idea to use more than one shard in the packet cache to reduce 
> contention between threads. You can do that by passing
> numberOfShards=10 
> (for example) to the new newPacketCache() command, see 
> https://dnsdist.org/reference/config.html#newPacketCache
> 
> It might also be a good idea to use more than one shard in the ring 
> buffers, by passing 10 (for example) as a second parameter to the 
> setRingBuffersSize() command:
> 
> https://dnsdist.org/reference/config.html#setRingBuffersSize
> 
> You are using the default size so setRingBuffersSize(10000, 10) would
> only change the number of shards, and will likely improve performance
> a bit.
> 
> Best regards,
> 
> Remi
> 
> > On Thu, 2020-12-24 at 09:56 +0100, Remi Gacogne via dnsdist wrote:
> > > Hi,
> > > 
> > > On 12/24/20 7:25 AM, Jahanzeb Arshad via dnsdist wrote:
> > > > We have deployed two instances of dnsdist v1.5.1 on CentOS 7.9.
> > > > After
> > > > running for 7-8 days both the machines start showing erratic
> > > > CPU usage
> > > > pattern. The CPU usage jumps to 40% then 0 and the servers keep
> > > > on doing
> > > > this. If the process is restarted this is fixed. Need some help
> > > > to
> > > > identify and fix this issue.
> > > 
> > > Would you mind posting the dnsdist configuration you are using,
> > > after
> > > redacting passwords and API keys, of course?
> > > 
> > > It would also be very helpful if you could do a "strace -f -p
> > > <pid of
> > > dnsdist>" for a few seconds after the process has started acting
> > > out.
> > > Getting a backtrace might also help, which can be done by
> > > attaching a
> > > debugger via "gdb -p <pid of dnsdist>" then issuing "thread apply
> > > all bt
> > > full".
> > > 
> > > Best regards,
> > 
> > _______________________________________________
> > dnsdist mailing list
> > dnsdist at mailman.powerdns.com
> > https://mailman.powerdns.com/mailman/listinfo/dnsdist
> > 
> 
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.powerdns.com/pipermail/dnsdist/attachments/20201224/6fd2ca53/attachment-0001.htm>