<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><div><blockquote type="cite" class=""><div class="">Op 15 mrt. 2024, om 17:01 heeft Tim Burns via Pdns-users <<a href="mailto:pdns-users@mailman.powerdns.com" class="">pdns-users@mailman.powerdns.com</a>> het volgende geschreven:</div><br class="Apple-interchange-newline"><div class=""><div class="">Hello all, I’m experiencing a performance degradation while using the Recursor that I haven’t been able to root cause, and I was hoping to get some insight on what might be causing it, or some troubleshooting and diagnostics tips.<br class=""> <br class="">The observed issue is that the Recursor would start up, begin receiving and forwarding traffic for several minutes, suddenly the CPU usage would jump to 100%, rec_control started timing out attempting to talk to the program, and Prometheus metrics scrapes would stop succeeding.<br class="">The Recursor also stopped reporting statistics at the logging.statistics_interval, although if I added a debug Lua script it would still log messages from the threads.<br class=""> <br class="">I’m running the Recursor as a container in Kubernetes with the 5.0.3 image from Docker Hub. The replicas receive anywhere from 2-6k QPS nominally, but this issue seems to occur regardless of query rate. Before the metrics freeze, the instance only uses 10% of its allocated CPU and about 20% of allocated memory, and then gets pegged to 100% CPU usage and stops responding. This happens anywhere from several minutes to an hour after process start.<br class=""> <br class="">Below is the configuration used. We’re using this as a middleman for now to forward requests upstream to larger recursors and cache the results, and is only accessible from within our network.<br class=""> <br class="">recursor:<br class="">forward_zones_recurse:<br class=""> - zone: .<br class=""> forwarders:<br class=""> - 8.8.8.8<br class=""> - 1.1.1.1<br class=""> - 8.8.4.4<br class=""> - 1.0.0.1<br class=""> - 208.67.222.222<br class=""> - 208.67.220.220<br class=""> threads: 8<br class="">security_poll_suffix: ''<br class=""> server_id: ''<br class=""> version_string: ''<br class=""> any_to_tcp: true<br class="">dnssec:<br class=""> validation: process-no-validate<br class="">incoming:<br class=""> allow_from:<br class=""> - 0.0.0.0/0<br class=""> listen:<br class=""> - 0.0.0.0<br class=""> port: 2353<br class="">logging:<br class=""> disable_syslog: true<br class="">recordcache:<br class=""> max_entries: 500000<br class="">webservice:<br class=""> address: 0.0.0.0<br class=""> allow_from:<br class=""> - 10.0.0.0/8<br class=""> - 172.16.0.0/12<br class=""> - 127.0.0.1<br class=""> loglevel: none<br class=""> port: 8912<br class=""> webserver: true<br class=""> <br class=""> <br class=""> <br class="">While attempting to mitigate this live, I took a variety of actions in an attempt to root cause the issue, but none of the actions made a difference and we observed the same failure mode each time.<br class="">Actions taken:<br class="">- Disabling DNSSEC entirely with dnssec.validation: off<br class="">- Reducing threads from 8 -> 4, we had this scaled up to 8 as the default 2 would lead to a high volume of UdpInErrors after some time<br class="">- Turned off TCP, only receiving queries over UDP<br class="">- Enabled logging.common_errors to see if there was anything interesting, but nothing notable happened around the time of freeze<br class=""> <br class="">Notably, rec_control would report the error: “Timeout waiting for control channel data”, and when I attempted to turn on trace logging for SERVFAILs (logging.trace: fail), logs indicated “Could not dup trace file” with reason “Bad file descriptor”.<br class=""> <br class="">Let me know if there’s any other information I can provide. I haven’t been able to correlate this behavior to any unusual metrics before the freeze as everything appeared reasonably nominal, but I’m happy to provide the last Prometheus scrape we got if it would help. Any ideas?<br class=""> <br class="">Thank you!<br class="">Tim<br class=""></div></div></blockquote></div><br class=""></div><div class=""><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">Try running when the recursor is peggef</div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class=""><br class=""></div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>perf top</div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class=""><br class=""></div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">Or attach gdb and run </div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class=""><br class=""></div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class=""><span class="Apple-tab-span" style="white-space:pre"> </span>thread apply all bt</div><div class=""><br class=""></div><div class="">on gdb's command line</div><div class=""><br class=""></div><font color="#000000" class=""><span style="caret-color: rgb(0, 0, 0);" class="">This might not possible in a container (I don’t know that), but you might be able to install the perf tools or gdb in the container. Or otherwise try to reproduce in an ordinary VM or bare metal.</span></font><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class=""><br class=""></div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>-Otto</div></div><div class=""><br class=""></div></body></html>