<html><head></head><body><div style="color:black;font-size:14px">Hello Klaus.<br></div><div style="color: black;font-size: 14px;">Thanks for your reply.<br></div><div style="color: black;font-size: 14px;"><br></div><div style="color: black;font-size: 14px;">Unfortunately nothing appears in the mysql logs.<br></div><div style="color: black;font-size: 14px;">We noticed this issue when we started to perform bulk operations, with the APIs, which add or modify the zones.<br></div><div style="color: black;font-size: 14px;"><br></div><div style="color: black;font-size: 14px;">Under normal operation, we just see messages like:<br></div><div style="color: black;font-size: 14px;"><br></div><blockquote style="border-left: 2px solid rgb(168, 168, 168); padding-left: 10px; margin-top: 0.5rem; margin-block-start: 0px; margin-inline-start: 0px;"><div style="color: black;font-size: 14px;">Feb 17 13:39:32 xxxx.xxxxx.xxx.xxxx pdns_server[26672]: Request to queue notification for domain yyyyy.yy was processed, but no valid nameserver or ALSO-NOTIFYs found. Not notifying!<br></div></blockquote><div style="color: black;font-size: 14px;">from time to time, but no crashes.</div><div style="color: black;font-size: 14px;"><br></div><div style="color: black;font-size: 14px;">Thanks.<br></div><div style="color: black;font-size: 14px;">Riccardo <br></div><div style="color:black;font-size:14px" class="signature"><br></div><div><br></div><div>18/02/2022, 09:58 Klaus Darilion ha scritto:<br></div><blockquote style="border-left: 2px solid rgb(168, 168, 168); margin-left: 0px; padding-left: 10px; margin-top: 0.5rem; margin-block-start: 0px; margin-inline-start: 0px;"><div class="qbox-body" lang="DE-AT"><div class="WordSection1"><p class="MsoNormal"><span style="color: rgb(31, 73, 125);"><span class="font" style="font-family:Calibri, sans-serif"><span class="size" style="font-size:11pt">Look in the mysql-server log. Maybe it mentions which other transaction causes the deadlock. That my help to find the cause.</span></span></span><br></p><p> <br></p><p class="MsoNormal"><span style="color: rgb(31, 73, 125);"><span class="font" style="font-family:Calibri, sans-serif"><span class="size" style="font-size:11pt">Is this problem something new? I.e. started it after a PDNs or Mysql Upgrade?</span></span></span><br></p><p> <br></p><p class="MsoNormal"><span style="color: rgb(31, 73, 125);"><span class="font" style="font-family:Calibri, sans-serif"><span class="size" style="font-size:11pt">regards</span></span></span><br></p><p class="MsoNormal"><span style="color: rgb(31, 73, 125);"><span class="font" style="font-family:Calibri, sans-serif"><span class="size" style="font-size:11pt">Klaus</span></span></span><br></p><p> <br></p><p> <br></p><div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt"><div><div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm"><p class="MsoNormal"><b><span style="" lang="DE"><span class="font" style="font-family:Calibri, sans-serif"><span class="size" style="font-size:11pt">Von:</span></span></span></b><span style="" lang="DE"><span class="font" style="font-family:Calibri, sans-serif"><span class="size" style="font-size:11pt"> Pdns-users <<a href="mailto:pdns-users-bounces@mailman.powerdns.com">pdns-users-bounces@mailman.powerdns.com</a>> <b>Im Auftrag von </b>Riccardo Brunetti via Pdns-users<br> <b>Gesendet:</b> Freitag, 18. Februar 2022 09:55<br> <b>An:</b> <a href="mailto:pdns-users@mailman.powerdns.com">pdns-users@mailman.powerdns.com</a><br> <b>Betreff:</b> [Pdns-users] powerdns frequently crashes and restarts</span></span></span></p></div></div><p> <br></p><div><p class="MsoNormal"><span style="color: black;"><span class="size" style="font-size:10.5pt">Hello.</span></span><br></p></div><div><p class="MsoNormal"><span style="color: black;"><span class="size" style="font-size:10.5pt">We are having quite frequent crashes of our pdns server with the following syslog messages:</span></span><br></p></div><div><p> <br></p></div><blockquote style="border-top: none; border-right: none; border-bottom: none; border-image: initial; border-left: 2px solid rgb(168, 168, 168); padding: 0cm 0cm 0cm 10px; margin-bottom: 5pt; margin-top: 0.5rem; margin-block-start: 0px; margin-inline-start: 0px;"><div><p class="MsoNormal"><span style="color: black;"><span class="size" style="font-size:10.5pt">Feb 17 13:39:33 <a href="http://xxxx.xxxxx.xxx">xxxx.xxxxx.xxx</a>.xxxx pdns_server[26672]: Exiting because communicator thread died with error: GSQLBackend unable to refresh domain_id XXXXX: Could not execute mysql statement: update
domains set notified_serial=? where id=?: Deadlock found when trying to get lock; try restarting transaction</span></span><br></p></div><div><p class="MsoNormal"><span style="color: black;"><span class="size" style="font-size:10.5pt">Feb 17 13:39:34 <a href="http://xxxx.xxxxx.xxx">xxxx.xxxxx.xxx</a>.xxxx systemd[1]: pdns.service: main process exited, code=exited, status=1/FAILURE</span></span><br></p></div><div><p class="MsoNormal"><span style="color: black;"><span class="size" style="font-size:10.5pt">Feb 17 13:39:34 <a href="http://xxxx.xxxxx.xxx">xxxx.xxxxx.xxx</a>.xxxx systemd[1]: Unit pdns.service entered failed state.</span></span><br></p></div><div><p class="MsoNormal"><span style="color: black;"><span class="size" style="font-size:10.5pt">Feb 17 13:39:34 <a href="http://xxxx.xxxxx.xxx">xxxx.xxxxx.xxx</a>.xxxx systemd[1]: pdns.service failed.</span></span><br></p></div><div><p class="MsoNormal"><span style="color: black;"><span class="size" style="font-size:10.5pt">Feb 17 13:39:35 <a href="http://xxxx.xxxxx.xxx">xxxx.xxxxx.xxx</a>.xxxx systemd[1]: pdns.service holdoff time over, scheduling restart.</span></span><br></p></div><div><p class="MsoNormal"><span style="color: black;"><span class="size" style="font-size:10.5pt">Feb 17 13:39:35 <a href="http://xxxx.xxxxx.xxx">xxxx.xxxxx.xxx</a>.xxxx systemd[1]: Stopped PowerDNS Authoritative Server.</span></span><br></p></div><div><p class="MsoNormal"><span style="color: black;"><span class="size" style="font-size:10.5pt">Feb 17 13:39:35 <a href="http://xxxx.xxxxx.xxx">xxxx.xxxxx.xxx</a>.xxxx systemd[1]: Starting PowerDNS Authoritative Server...</span></span><br></p></div><div><p class="MsoNormal"><span style="color: black;"><span class="size" style="font-size:10.5pt">Feb 17 13:39:35 <a href="http://xxxx.xxxxx.xxx">xxxx.xxxxx.xxx</a>.xxxx pdns_server[25899]: Loading '/usr/lib64/pdns/libgmysqlbackend.so'</span></span><br></p></div><div><p class="MsoNormal"><span style="color: black;"><span class="size" style="font-size:10.5pt">Feb 17 13:39:35 <a href="http://xxxx.xxxxx.xxx">xxxx.xxxxx.xxx</a>.xxxx pdns_server[25899]: This is a standalone pdns</span></span><br></p></div><div><p> <br></p></div></blockquote><div><p class="MsoNormal"><span style="color: black;"><span class="size" style="font-size:10.5pt">I don't know if this can be related, but just before those crashes we see messages like:</span></span><br></p></div><div><p> <br></p></div><blockquote style="border-top: none; border-right: none; border-bottom: none; border-image: initial; border-left: 2px solid rgb(168, 168, 168); padding: 0cm 0cm 0cm 10px; margin-bottom: 5pt; margin-top: 0.5rem; margin-block-start: 0px; margin-inline-start: 0px;"><div><p class="MsoNormal"><span style="color: black;"><span class="size" style="font-size:10.5pt">Feb 17 13:39:32 <a href="http://xxxx.xxxxx.xxx">xxxx.xxxxx.xxx</a>.xxxx pdns_server[26672]: Request to queue notification for domain yyyyy.yy was processed, but no valid nameserver or ALSO-NOTIFYs found. Not notifying!</span></span><br></p></div><div><p> <br></p></div></blockquote><div><p class="MsoNormal"><span style="color: black;"><span class="size" style="font-size:10.5pt">Just to mention briefly our setup: we are running:</span></span><br></p></div><div><p class="MsoNormal"><span style="color: black;"><span class="size" style="font-size:10.5pt">PowerDNS Authoritative Server 4.5.2 (C)</span></span><br></p></div><div><p> <br></p></div><div><p class="MsoNormal"><span style="color: black;"><span class="size" style="font-size:10.5pt">in a percona mysql cluster with three nodes, each node having it's own mysql and pdns server (in order to have redundancy of database and APIs).</span></span><br></p></div><div><p> <br></p></div><div><p class="MsoNormal"><span style="color: black;"><span class="size" style="font-size:10.5pt">This cluster acts as a "backend master" for the real authoritative slaves, placed elsewhere, replicating the db.</span></span><br></p></div><div><p> <br></p></div><div><p class="MsoNormal"><span style="color: black;"><span class="size" style="font-size:10.5pt">Can you help us to understand why we have those crashes?</span></span><br></p></div><div><p> <br></p></div><div><p class="MsoNormal"><span style="color: black;"><span class="size" style="font-size:10.5pt">Thanks</span></span><br></p></div><div><p class="MsoNormal"><span style="color: black;"><span class="size" style="font-size:10.5pt">Riccardo</span></span><br></p></div><div><p class="MsoNormal"><span style="color: black;"><span class="size" style="font-size:10.5pt"> </span></span><br></p></div></div></div></div></blockquote><div style="color: black;font-size: 14px;"><br></div></body></html>