[Pdns-users] Backend error: Failed to executemysql_query,perhaps connection died? Err=1: MySQL server has goneaway

Tue Jan 29 20:27:55 UTC 2008

Stephen,

You are completely correct. It is a bandaid (needed) until the
patched version is available. I am in the same situation currently
with 2.9.20 and the TCP connection handle problem and have to
restart my master on a regular basis. Monitoring is definitely
the way to go for issues such as memory leaks.

Ken

On Tue, Jan 29, 2008 at 12:18:54PM -0800, Stephen Manchester wrote:
> Increasing the wait_timeout on the mysql server will help, but it does not 
> actually solve the memory leak, as dead connections do happen for a number 
> of reasons (server failure, network failure, stateful firewalls, etc.)
>
> Until a patch can be applied though, increasing the wait_timeout value on 
> the MySQL server to something very high (say 1 day or more) is the only 
> thing you can do.  From what I've read, it looks like the default for MySQL 
> 4.x was 30s.  The default for 5.x is 28800 seconds, but if there was 
> already a my.cnf file laying around (which there often is a 4.x my.cnf file 
> by default on many distros) then you might inherit this old value which 
> will cause the problem to happen very often.  Again, this is only a work 
> around, not a true fix.  You should still monitor pdns virtual memory usage 
> from time to time to make sure it isn't getting crazy.
>
> ----- Original Message ----- From: "Kenneth Marshall" <ktm at rice.edu>
> To: "Stephen Manchester" <smanches at craftyspace.com>
> Cc: <Pdns-users at mailman.powerdns.com>
> Sent: Tuesday, January 29, 2008 11:56 AM
> Subject: Re: [Pdns-users] Backend error: Failed to 
> executemysql_query,perhaps connection died? Err=1: MySQL server has 
> goneaway
>
>
>> Again, this problem, for which a patch already exists, is triggered
>> by the DB connection being terminated abnormally by the backend
>> server, in this case via a connection time_limit. I do not use
>> MySQL as a PDNS backend, but the PostgreSQL backend does correctly
>> reconnect to the database should the connection die. I assume that
>> the PDNS will reconnect to the MySQL server when the connection
>> dies. It seems that increasing the timeout value in MySQL is that
>> best measure to ensure more robust service from PDNS.
>>
>> Cheers,
>> Ken
>>
>>
>> On Tue, Jan 29, 2008 at 11:43:43AM -0800, Stephen Manchester wrote:
>>> There is currently a memory leak in 2.9.21 from gmysql backend threads
>>> dying in this fashion.  The threads themselves are not being cleaned up.
>>> this is only noticeable on very light loads where the backends do not get
>>> enough traffic to keep the mysql connection open.  You will notice it by
>>> abnormally high virtual memory usage after pdns has been running for a
>>> time.  Each backend that dies leaves 10MB of virtual memory allocated 
>>> (the
>>> thread stack?) and this will not get cleaned up until pdns is restarted.
>>>
>>> Hopefully they will implement mysql connection reconnects in the future,
>>> which solves both the memory leak and the backends failing when the
>>> connection is closed.  The only solution I found was to apply a patch and
>>> compile from source.
>>>
>>> ----- Original Message ----- From: "Kenneth Marshall" <ktm at rice.edu>
>>> To: "Catalin Constantin" <dazoot at gmail.com>
>>> Cc: <Pdns-users at mailman.powerdns.com>
>>> Sent: Tuesday, January 29, 2008 5:25 AM
>>> Subject: Re: [Pdns-users] Backend error: Failed to execute
>>> mysql_query,perhaps connection died? Err=1: MySQL server has gone away
>>>
>>>
>>>> On Tue, Jan 29, 2008 at 03:04:46PM +0200, Catalin Constantin wrote:
>>>>> Hello,
>>>>>
>>>>> This is a funny question.
>>>>> Mysql is FINE and it's up and running on localhost.
>>>>>
>>>>> Here is the startup log of PDNS.
>>>>> Jan 29 14:59:05 k2 pdns[30294]: PowerDNS 2.9.20 (C) 2001-2006
>>>>> PowerDNS.COMBV (Mar 10 2007, 00:36:58, gcc
>>>>> 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)) starting up
>>>>> Jan 29 14:59:05 k2 pdns[30294]: PowerDNS comes with ABSOLUTELY NO
>>>>> WARRANTY.
>>>>> This is free software, and you are welcome to redistribute it according
>>>>> to
>>>>> the terms of the GPL version 2.
>>>>> Jan 29 14:59:05 k2 pdns[30294]: Set effective group id to 109
>>>>> Jan 29 14:59:05 k2 pdns[30294]: Set effective user id to 113
>>>>> Jan 29 14:59:05 k2 pdns[30294]: Creating backend connection for TCP
>>>>> Jan 29 14:59:05 k2 pdns[30294]: Master/slave communicator launching
>>>>> Jan 29 14:59:05 k2 pdns[30294]: gmysql Connection succesful
>>>>> Jan 29 14:59:05 k2 pdns[30294]: gmysql Connection succesful
>>>>> Jan 29 14:59:05 k2 pdns[30294]: About to create 10 backend threads for
>>>>> UDP
>>>>> Jan 29 14:59:05 k2 pdns[30294]: gmysql Connection succesful
>>>>> Jan 29 14:59:05 k2 pdns[30294]: 14 slave domains need checking
>>>>> Jan 29 14:59:05 k2 pdns[30294]: Domain test1.com is stale, master 
>>>>> serial
>>>>> 1201611548, our serial 1201610827
>>>>> Jan 29 14:59:05 k2 pdns[30294]: Domain test2.com is stale, master 
>>>>> serial
>>>>> 1201611548, our serial 1201610827
>>>>> Jan 29 14:59:05 k2 pdns[30294]: Domain test3.com is stale, master 
>>>>> serial
>>>>> 1201611548, our serial 1201610827
>>>>> Jan 29 14:59:05 k2 pdns[30294]: gmysql Connection succesful
>>>>> Jan 29 14:59:05 k2 pdns[30294]: AXFR started for 'test1.com', 
>>>>> transaction
>>>>> started
>>>>> Jan 29 14:59:05 k2 pdns[30294]: gmysql Connection succesful
>>>>> Jan 29 14:59:05 k2 pdns[30294]: AXFR done for 'test1.com', zone 
>>>>> committed
>>>>>
>>>>>
>>>>> The problem is that mysql closes the connection if there is no 
>>>>> activity.
>>>>> PDNS does not make sure that MySQL is still "alive" and from time to 
>>>>> time
>>>>> it
>>>>> shows in the log errors like:
>>>>> Jan 29 11:33:57 k2 pdns[28941]: Backend error: Failed to execute
>>>>> mysql_query, perhaps connection died? Err=1: MySQL server has gone away
>>>>>
>>>>> Does PDNS have a timeout for backend connection "recheck" ?
>>>>> Can this be configured ?
>>>>>
>>>>> Is there anything we can do to make sure the connection does not go 
>>>>> away
>>>>> ?
>>>>>
>>>>> Note: the NS server activity for now is not much (we're still testing 
>>>>> and
>>>>> just have a couple of domains on it).
>>>>>
>>>>> Thanks,
>>>>>
>>>> Dear Ms. Constantin,
>>>>
>>>> If a properly functioning application opens a connection to a database,
>>>> the connection should not drop until closed. The typical fix is to
>>>> increase your timeout on your MySQL server appropriately. This same
>>>> thread comes up periodically. In order for PDNS to notice that a
>>>> connection has died, it tries to use it with a query. This causes
>>>> the errors in your logs. The amount of errors will be proportional
>>>> to how long your timeout is set. I certainly would not like to see
>>>> a patch that added overhead to every query to check the status of
>>>> a connection and re-establish if needed. PDNS will re-connect if
>>>> the connection drops, but you will get an error. If your MySQL DB
>>>> is not dedicated to PDNS, then you will have to balance the needs
>>>> of all of the applications using the DB. It sounds like you may
>>>> be running some pretty poorly written ones. Good luck.
>>>>
>>>> Ken
>>>>
>>>>> On 1/29/08, Nico van Royen <nico at van-royen.nl> wrote:
>>>>> > Hello,
>>>>> >
>>>>> > Do you infact have a running mysql server ?  Also check the database 
>>>>> >  >
>>>>> that
>>>>> > you have specified in your pdns.conf matches your actual database
>>>>> > information (db name, user/password etc).
>>>>> >
>>>>> > Regards,
>>>>> > Nico
>>>>> > WARP3.COM
>>>>> > ________________________________
>>>>> > From: Catalin Constantin [mailto:dazoot at gmail.com]
>>>>> > To: pdns-users at mailman.powerdns.com
>>>>> > Sent: Tue, 29 Jan 2008 10:48:09 +0100
>>>>> > Subject: [Pdns-users] Backend error: Failed to execute mysql_query,
>>>>> perhaps
>>>>> > connection died? Err=1: MySQL server has gone away
>>>>> >
>>>>> >
>>>>> > Hello,
>>>>> >
>>>>> > We're getting quite many errors:
>>>>> > Backend error: Failed to execute mysql_query, perhaps connection > 
>>>>> died?
>>>>> > Err=1: MySQL server has gone away latelly.
>>>>> >
>>>>> > I checked the docs and the mailing lists for this error and i found
>>>>> > little informations about it.
>>>>> >
>>>>> > What is actually the solution for this error not to happen ?
>>>>> > Nice would be to have a real example.
>>>>> >
>>>>> > Note:
>>>>> > - we use latest pdns package from Debian Etch (stable).
>>>>> >
>>>>> > Thanks,
>>>>> >
>>>>> > --
>>>>> > Catalin Constantin
>>>>> > Dazoot Software
>>>>> > http://www.dazoot.eu/
>>>>> > _______________________________________________
>>>>> > Pdns-users mailing list
>>>>> > Pdns-users at mailman.powerdns.com
>>>>> > http://mailman.powerdns.com/mailman/listinfo/pdns-users
>>>>> >
>>>>> > _______________________________________________
>>>>> > Pdns-users mailing list
>>>>> > Pdns-users at mailman.powerdns.com
>>>>> > http://mailman.powerdns.com/mailman/listinfo/pdns-users
>>>>> >
>>>>> >
>>>>>
>>>>>
>>>>> -- 
>>>>> Catalin Constantin
>>>>> Dazoot Software
>>>>> http://www.dazoot.eu/
>>>>
>>>>> _______________________________________________
>>>>> Pdns-users mailing list
>>>>> Pdns-users at mailman.powerdns.com
>>>>> http://mailman.powerdns.com/mailman/listinfo/pdns-users
>>>>
>>>> _______________________________________________
>>>> Pdns-users mailing list
>>>> Pdns-users at mailman.powerdns.com
>>>> http://mailman.powerdns.com/mailman/listinfo/pdns-users
>>>
>>> _______________________________________________
>>> Pdns-users mailing list
>>> Pdns-users at mailman.powerdns.com
>>> http://mailman.powerdns.com/mailman/listinfo/pdns-users
>>>
>
> _______________________________________________
> Pdns-users mailing list
> Pdns-users at mailman.powerdns.com
> http://mailman.powerdns.com/mailman/listinfo/pdns-users
>