[Pdns-users] Backend error: Failed to execute mysql_query,perhaps connection died? Err=1: MySQL server has gone away

Tue Jan 29 19:56:00 UTC 2008

Again, this problem, for which a patch already exists, is triggered
by the DB connection being terminated abnormally by the backend
server, in this case via a connection time_limit. I do not use
MySQL as a PDNS backend, but the PostgreSQL backend does correctly
reconnect to the database should the connection die. I assume that
the PDNS will reconnect to the MySQL server when the connection
dies. It seems that increasing the timeout value in MySQL is that
best measure to ensure more robust service from PDNS.

Cheers,
Ken

On Tue, Jan 29, 2008 at 11:43:43AM -0800, Stephen Manchester wrote:
> There is currently a memory leak in 2.9.21 from gmysql backend threads 
> dying in this fashion.  The threads themselves are not being cleaned up.  
> this is only noticeable on very light loads where the backends do not get 
> enough traffic to keep the mysql connection open.  You will notice it by 
> abnormally high virtual memory usage after pdns has been running for a 
> time.  Each backend that dies leaves 10MB of virtual memory allocated (the 
> thread stack?) and this will not get cleaned up until pdns is restarted.
>
> Hopefully they will implement mysql connection reconnects in the future, 
> which solves both the memory leak and the backends failing when the 
> connection is closed.  The only solution I found was to apply a patch and 
> compile from source.
>
> ----- Original Message ----- From: "Kenneth Marshall" <ktm at rice.edu>
> To: "Catalin Constantin" <dazoot at gmail.com>
> Cc: <Pdns-users at mailman.powerdns.com>
> Sent: Tuesday, January 29, 2008 5:25 AM
> Subject: Re: [Pdns-users] Backend error: Failed to execute 
> mysql_query,perhaps connection died? Err=1: MySQL server has gone away
>
>
>> On Tue, Jan 29, 2008 at 03:04:46PM +0200, Catalin Constantin wrote:
>>> Hello,
>>>
>>> This is a funny question.
>>> Mysql is FINE and it's up and running on localhost.
>>>
>>> Here is the startup log of PDNS.
>>> Jan 29 14:59:05 k2 pdns[30294]: PowerDNS 2.9.20 (C) 2001-2006
>>> PowerDNS.COMBV (Mar 10 2007, 00:36:58, gcc
>>> 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)) starting up
>>> Jan 29 14:59:05 k2 pdns[30294]: PowerDNS comes with ABSOLUTELY NO 
>>> WARRANTY.
>>> This is free software, and you are welcome to redistribute it according 
>>> to
>>> the terms of the GPL version 2.
>>> Jan 29 14:59:05 k2 pdns[30294]: Set effective group id to 109
>>> Jan 29 14:59:05 k2 pdns[30294]: Set effective user id to 113
>>> Jan 29 14:59:05 k2 pdns[30294]: Creating backend connection for TCP
>>> Jan 29 14:59:05 k2 pdns[30294]: Master/slave communicator launching
>>> Jan 29 14:59:05 k2 pdns[30294]: gmysql Connection succesful
>>> Jan 29 14:59:05 k2 pdns[30294]: gmysql Connection succesful
>>> Jan 29 14:59:05 k2 pdns[30294]: About to create 10 backend threads for 
>>> UDP
>>> Jan 29 14:59:05 k2 pdns[30294]: gmysql Connection succesful
>>> Jan 29 14:59:05 k2 pdns[30294]: 14 slave domains need checking
>>> Jan 29 14:59:05 k2 pdns[30294]: Domain test1.com is stale, master serial
>>> 1201611548, our serial 1201610827
>>> Jan 29 14:59:05 k2 pdns[30294]: Domain test2.com is stale, master serial
>>> 1201611548, our serial 1201610827
>>> Jan 29 14:59:05 k2 pdns[30294]: Domain test3.com is stale, master serial
>>> 1201611548, our serial 1201610827
>>> Jan 29 14:59:05 k2 pdns[30294]: gmysql Connection succesful
>>> Jan 29 14:59:05 k2 pdns[30294]: AXFR started for 'test1.com', transaction
>>> started
>>> Jan 29 14:59:05 k2 pdns[30294]: gmysql Connection succesful
>>> Jan 29 14:59:05 k2 pdns[30294]: AXFR done for 'test1.com', zone committed
>>>
>>>
>>> The problem is that mysql closes the connection if there is no activity.
>>> PDNS does not make sure that MySQL is still "alive" and from time to time 
>>> it
>>> shows in the log errors like:
>>> Jan 29 11:33:57 k2 pdns[28941]: Backend error: Failed to execute
>>> mysql_query, perhaps connection died? Err=1: MySQL server has gone away
>>>
>>> Does PDNS have a timeout for backend connection "recheck" ?
>>> Can this be configured ?
>>>
>>> Is there anything we can do to make sure the connection does not go away 
>>> ?
>>>
>>> Note: the NS server activity for now is not much (we're still testing and
>>> just have a couple of domains on it).
>>>
>>> Thanks,
>>>
>> Dear Ms. Constantin,
>>
>> If a properly functioning application opens a connection to a database,
>> the connection should not drop until closed. The typical fix is to
>> increase your timeout on your MySQL server appropriately. This same
>> thread comes up periodically. In order for PDNS to notice that a
>> connection has died, it tries to use it with a query. This causes
>> the errors in your logs. The amount of errors will be proportional
>> to how long your timeout is set. I certainly would not like to see
>> a patch that added overhead to every query to check the status of
>> a connection and re-establish if needed. PDNS will re-connect if
>> the connection drops, but you will get an error. If your MySQL DB
>> is not dedicated to PDNS, then you will have to balance the needs
>> of all of the applications using the DB. It sounds like you may
>> be running some pretty poorly written ones. Good luck.
>>
>> Ken
>>
>>> On 1/29/08, Nico van Royen <nico at van-royen.nl> wrote:
>>> > Hello,
>>> >
>>> > Do you infact have a running mysql server ?  Also check the database > 
>>> that
>>> > you have specified in your pdns.conf matches your actual database
>>> > information (db name, user/password etc).
>>> >
>>> > Regards,
>>> > Nico
>>> > WARP3.COM
>>> > ________________________________
>>> > From: Catalin Constantin [mailto:dazoot at gmail.com]
>>> > To: pdns-users at mailman.powerdns.com
>>> > Sent: Tue, 29 Jan 2008 10:48:09 +0100
>>> > Subject: [Pdns-users] Backend error: Failed to execute mysql_query,
>>> perhaps
>>> > connection died? Err=1: MySQL server has gone away
>>> >
>>> >
>>> > Hello,
>>> >
>>> > We're getting quite many errors:
>>> > Backend error: Failed to execute mysql_query, perhaps connection died?
>>> > Err=1: MySQL server has gone away latelly.
>>> >
>>> > I checked the docs and the mailing lists for this error and i found
>>> > little informations about it.
>>> >
>>> > What is actually the solution for this error not to happen ?
>>> > Nice would be to have a real example.
>>> >
>>> > Note:
>>> > - we use latest pdns package from Debian Etch (stable).
>>> >
>>> > Thanks,
>>> >
>>> > --
>>> > Catalin Constantin
>>> > Dazoot Software
>>> > http://www.dazoot.eu/
>>> > _______________________________________________
>>> > Pdns-users mailing list
>>> > Pdns-users at mailman.powerdns.com
>>> > http://mailman.powerdns.com/mailman/listinfo/pdns-users
>>> >
>>> > _______________________________________________
>>> > Pdns-users mailing list
>>> > Pdns-users at mailman.powerdns.com
>>> > http://mailman.powerdns.com/mailman/listinfo/pdns-users
>>> >
>>> >
>>>
>>>
>>> -- 
>>> Catalin Constantin
>>> Dazoot Software
>>> http://www.dazoot.eu/
>>
>>> _______________________________________________
>>> Pdns-users mailing list
>>> Pdns-users at mailman.powerdns.com
>>> http://mailman.powerdns.com/mailman/listinfo/pdns-users
>>
>> _______________________________________________
>> Pdns-users mailing list
>> Pdns-users at mailman.powerdns.com
>> http://mailman.powerdns.com/mailman/listinfo/pdns-users
>
> _______________________________________________
> Pdns-users mailing list
> Pdns-users at mailman.powerdns.com
> http://mailman.powerdns.com/mailman/listinfo/pdns-users
>