[Pdns-users] powerdns frequently crashes and restarts

[Riccardo Brunetti]

Hello Klaus.

Thanks for your reply.



Unfortunately nothing appears in the mysql logs.

We noticed this issue when we started to perform bulk operations, with the APIs, which add or modify the zones.



Under normal operation, we just see messages like:



> Feb 17 13:39:32 xxxx.xxxxx.xxx.xxxx pdns_server[26672]: Request to queue notification for domain yyyyy.yy was processed, but no valid nameserver or ALSO-NOTIFYs found. Not notifying!

from time to time, but no crashes.


Thanks.

Riccardo 





18/02/2022, 09:58 Klaus Darilion ha scritto:


> Look in the mysql-server log. Maybe it mentions which other transaction causes the deadlock. That my help to find the cause.
> 
> 
>  
> 
> 
> Is this problem something new? I.e. started it after a PDNs or Mysql Upgrade?
> 
> 
>  
> 
> 
> regards
> 
> 
> Klaus
> 
> 
>  
> 
> 
>  
> 
> 
> Von: Pdns-users <pdns-users-bounces at mailman.powerdns.com> Im Auftrag von Riccardo Brunetti via Pdns-users
> Gesendet: Freitag, 18. Februar 2022 09:55
> An: pdns-users at mailman.powerdns.com
> Betreff: [Pdns-users] powerdns frequently crashes and restarts
> 
>  
> 
> 
> Hello.
> 
> 
> We are having quite frequent crashes of our pdns server with the following syslog messages:
> 
> 
>  
> 
> 
> > Feb 17 13:39:33 xxxx.xxxxx.xxx.xxxx pdns_server[26672]: Exiting because communicator thread died with error: GSQLBackend unable to refresh domain_id XXXXX: Could not execute mysql statement: update domains set notified_serial=? where id=?: Deadlock found when trying to get lock; try restarting transaction
> > 
> > 
> > Feb 17 13:39:34 xxxx.xxxxx.xxx.xxxx systemd[1]: pdns.service: main process exited, code=exited, status=1/FAILURE
> > 
> > 
> > Feb 17 13:39:34 xxxx.xxxxx.xxx.xxxx systemd[1]: Unit pdns.service entered failed state.
> > 
> > 
> > Feb 17 13:39:34 xxxx.xxxxx.xxx.xxxx systemd[1]: pdns.service failed.
> > 
> > 
> > Feb 17 13:39:35 xxxx.xxxxx.xxx.xxxx systemd[1]: pdns.service holdoff time over, scheduling restart.
> > 
> > 
> > Feb 17 13:39:35 xxxx.xxxxx.xxx.xxxx systemd[1]: Stopped PowerDNS Authoritative Server.
> > 
> > 
> > Feb 17 13:39:35 xxxx.xxxxx.xxx.xxxx systemd[1]: Starting PowerDNS Authoritative Server...
> > 
> > 
> > Feb 17 13:39:35 xxxx.xxxxx.xxx.xxxx pdns_server[25899]: Loading '/usr/lib64/pdns/libgmysqlbackend.so'
> > 
> > 
> > Feb 17 13:39:35 xxxx.xxxxx.xxx.xxxx pdns_server[25899]: This is a standalone pdns
> > 
> > 
> >  
> 
> I don't know if this can be related, but just before those crashes we see messages like:
> 
> 
>  
> 
> 
> > Feb 17 13:39:32 xxxx.xxxxx.xxx.xxxx pdns_server[26672]: Request to queue notification for domain yyyyy.yy was processed, but no valid nameserver or ALSO-NOTIFYs found. Not notifying!
> > 
> > 
> >  
> 
> Just to mention briefly our setup: we are running:
> 
> 
> PowerDNS Authoritative Server 4.5.2 (C)
> 
> 
>  
> 
> 
> in a percona mysql cluster with three nodes, each node having it's own mysql and pdns server (in order to have redundancy of database and APIs).
> 
> 
>  
> 
> 
> This cluster acts as a "backend master" for the real authoritative slaves, placed elsewhere, replicating the db.
> 
> 
>  
> 
> 
> Can you help us to understand why we have those crashes?
> 
> 
>  
> 
> 
> Thanks
> 
> 
> Riccardo
> 
> 
>  


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.powerdns.com/pipermail/pdns-users/attachments/20220218/356b358f/attachment-0001.htm>