[Pdns-users] Backend error: Failed to executemysql_query,perhaps connection died? Err=1: MySQL server has goneaway

Tue Jan 29 20:18:54 UTC 2008

Increasing the wait_timeout on the mysql server will help, but it does not 
actually solve the memory leak, as dead connections do happen for a number 
of reasons (server failure, network failure, stateful firewalls, etc.)

Until a patch can be applied though, increasing the wait_timeout value on 
the MySQL server to something very high (say 1 day or more) is the only 
thing you can do.  From what I've read, it looks like the default for MySQL 
4.x was 30s.  The default for 5.x is 28800 seconds, but if there was already 
a my.cnf file laying around (which there often is a 4.x my.cnf file by 
default on many distros) then you might inherit this old value which will 
cause the problem to happen very often.  Again, this is only a work around, 
not a true fix.  You should still monitor pdns virtual memory usage from 
time to time to make sure it isn't getting crazy.

----- Original Message ----- 
From: "Kenneth Marshall" <ktm at rice.edu>
To: "Stephen Manchester" <smanches at craftyspace.com>
Cc: <Pdns-users at mailman.powerdns.com>
Sent: Tuesday, January 29, 2008 11:56 AM
Subject: Re: [Pdns-users] Backend error: Failed to 
executemysql_query,perhaps connection died? Err=1: MySQL server has goneaway

> Again, this problem, for which a patch already exists, is triggered
> by the DB connection being terminated abnormally by the backend
> server, in this case via a connection time_limit. I do not use
> MySQL as a PDNS backend, but the PostgreSQL backend does correctly
> reconnect to the database should the connection die. I assume that
> the PDNS will reconnect to the MySQL server when the connection
> dies. It seems that increasing the timeout value in MySQL is that
> best measure to ensure more robust service from PDNS.
>
> Cheers,
> Ken
>
>
> On Tue, Jan 29, 2008 at 11:43:43AM -0800, Stephen Manchester wrote:
>> There is currently a memory leak in 2.9.21 from gmysql backend threads
>> dying in this fashion.  The threads themselves are not being cleaned up.
>> this is only noticeable on very light loads where the backends do not get
>> enough traffic to keep the mysql connection open.  You will notice it by
>> abnormally high virtual memory usage after pdns has been running for a
>> time.  Each backend that dies leaves 10MB of virtual memory allocated 
>> (the
>> thread stack?) and this will not get cleaned up until pdns is restarted.
>>
>> Hopefully they will implement mysql connection reconnects in the future,
>> which solves both the memory leak and the backends failing when the
>> connection is closed.  The only solution I found was to apply a patch and
>> compile from source.
>>
>> ----- Original Message ----- From: "Kenneth Marshall" <ktm at rice.edu>
>> To: "Catalin Constantin" <dazoot at gmail.com>
>> Cc: <Pdns-users at mailman.powerdns.com>
>> Sent: Tuesday, January 29, 2008 5:25 AM
>> Subject: Re: [Pdns-users] Backend error: Failed to execute
>> mysql_query,perhaps connection died? Err=1: MySQL server has gone away
>>
>>
>>> On Tue, Jan 29, 2008 at 03:04:46PM +0200, Catalin Constantin wrote:
>>>> Hello,
>>>>
>>>> This is a funny question.
>>>> Mysql is FINE and it's up and running on localhost.
>>>>
>>>> Here is the startup log of PDNS.
>>>> Jan 29 14:59:05 k2 pdns[30294]: PowerDNS 2.9.20 (C) 2001-2006
>>>> PowerDNS.COMBV (Mar 10 2007, 00:36:58, gcc
>>>> 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)) starting up
>>>> Jan 29 14:59:05 k2 pdns[30294]: PowerDNS comes with ABSOLUTELY NO
>>>> WARRANTY.
>>>> This is free software, and you are welcome to redistribute it according
>>>> to
>>>> the terms of the GPL version 2.
>>>> Jan 29 14:59:05 k2 pdns[30294]: Set effective group id to 109
>>>> Jan 29 14:59:05 k2 pdns[30294]: Set effective user id to 113
>>>> Jan 29 14:59:05 k2 pdns[30294]: Creating backend connection for TCP
>>>> Jan 29 14:59:05 k2 pdns[30294]: Master/slave communicator launching
>>>> Jan 29 14:59:05 k2 pdns[30294]: gmysql Connection succesful
>>>> Jan 29 14:59:05 k2 pdns[30294]: gmysql Connection succesful
>>>> Jan 29 14:59:05 k2 pdns[30294]: About to create 10 backend threads for
>>>> UDP
>>>> Jan 29 14:59:05 k2 pdns[30294]: gmysql Connection succesful
>>>> Jan 29 14:59:05 k2 pdns[30294]: 14 slave domains need checking
>>>> Jan 29 14:59:05 k2 pdns[30294]: Domain test1.com is stale, master 
>>>> serial
>>>> 1201611548, our serial 1201610827
>>>> Jan 29 14:59:05 k2 pdns[30294]: Domain test2.com is stale, master 
>>>> serial
>>>> 1201611548, our serial 1201610827
>>>> Jan 29 14:59:05 k2 pdns[30294]: Domain test3.com is stale, master 
>>>> serial
>>>> 1201611548, our serial 1201610827
>>>> Jan 29 14:59:05 k2 pdns[30294]: gmysql Connection succesful
>>>> Jan 29 14:59:05 k2 pdns[30294]: AXFR started for 'test1.com', 
>>>> transaction
>>>> started
>>>> Jan 29 14:59:05 k2 pdns[30294]: gmysql Connection succesful
>>>> Jan 29 14:59:05 k2 pdns[30294]: AXFR done for 'test1.com', zone 
>>>> committed
>>>>
>>>>
>>>> The problem is that mysql closes the connection if there is no 
>>>> activity.
>>>> PDNS does not make sure that MySQL is still "alive" and from time to 
>>>> time
>>>> it
>>>> shows in the log errors like:
>>>> Jan 29 11:33:57 k2 pdns[28941]: Backend error: Failed to execute
>>>> mysql_query, perhaps connection died? Err=1: MySQL server has gone away
>>>>
>>>> Does PDNS have a timeout for backend connection "recheck" ?
>>>> Can this be configured ?
>>>>
>>>> Is there anything we can do to make sure the connection does not go 
>>>> away
>>>> ?
>>>>
>>>> Note: the NS server activity for now is not much (we're still testing 
>>>> and
>>>> just have a couple of domains on it).
>>>>
>>>> Thanks,
>>>>
>>> Dear Ms. Constantin,
>>>
>>> If a properly functioning application opens a connection to a database,
>>> the connection should not drop until closed. The typical fix is to
>>> increase your timeout on your MySQL server appropriately. This same
>>> thread comes up periodically. In order for PDNS to notice that a
>>> connection has died, it tries to use it with a query. This causes
>>> the errors in your logs. The amount of errors will be proportional
>>> to how long your timeout is set. I certainly would not like to see
>>> a patch that added overhead to every query to check the status of
>>> a connection and re-establish if needed. PDNS will re-connect if
>>> the connection drops, but you will get an error. If your MySQL DB
>>> is not dedicated to PDNS, then you will have to balance the needs
>>> of all of the applications using the DB. It sounds like you may
>>> be running some pretty poorly written ones. Good luck.
>>>
>>> Ken
>>>
>>>> On 1/29/08, Nico van Royen <nico at van-royen.nl> wrote:
>>>> > Hello,
>>>> >
>>>> > Do you infact have a running mysql server ?  Also check the database 
>>>> >  >
>>>> that
>>>> > you have specified in your pdns.conf matches your actual database
>>>> > information (db name, user/password etc).
>>>> >
>>>> > Regards,
>>>> > Nico
>>>> > WARP3.COM
>>>> > ________________________________
>>>> > From: Catalin Constantin [mailto:dazoot at gmail.com]
>>>> > To: pdns-users at mailman.powerdns.com
>>>> > Sent: Tue, 29 Jan 2008 10:48:09 +0100
>>>> > Subject: [Pdns-users] Backend error: Failed to execute mysql_query,
>>>> perhaps
>>>> > connection died? Err=1: MySQL server has gone away
>>>> >
>>>> >
>>>> > Hello,
>>>> >
>>>> > We're getting quite many errors:
>>>> > Backend error: Failed to execute mysql_query, perhaps connection 
>>>> > died?
>>>> > Err=1: MySQL server has gone away latelly.
>>>> >
>>>> > I checked the docs and the mailing lists for this error and i found
>>>> > little informations about it.
>>>> >
>>>> > What is actually the solution for this error not to happen ?
>>>> > Nice would be to have a real example.
>>>> >
>>>> > Note:
>>>> > - we use latest pdns package from Debian Etch (stable).
>>>> >
>>>> > Thanks,
>>>> >
>>>> > --
>>>> > Catalin Constantin
>>>> > Dazoot Software
>>>> > http://www.dazoot.eu/
>>>> > _______________________________________________
>>>> > Pdns-users mailing list
>>>> > Pdns-users at mailman.powerdns.com
>>>> > http://mailman.powerdns.com/mailman/listinfo/pdns-users
>>>> >
>>>> > _______________________________________________
>>>> > Pdns-users mailing list
>>>> > Pdns-users at mailman.powerdns.com
>>>> > http://mailman.powerdns.com/mailman/listinfo/pdns-users
>>>> >
>>>> >
>>>>
>>>>
>>>> -- 
>>>> Catalin Constantin
>>>> Dazoot Software
>>>> http://www.dazoot.eu/
>>>
>>>> _______________________________________________
>>>> Pdns-users mailing list
>>>> Pdns-users at mailman.powerdns.com
>>>> http://mailman.powerdns.com/mailman/listinfo/pdns-users
>>>
>>> _______________________________________________
>>> Pdns-users mailing list
>>> Pdns-users at mailman.powerdns.com
>>> http://mailman.powerdns.com/mailman/listinfo/pdns-users
>>
>> _______________________________________________
>> Pdns-users mailing list
>> Pdns-users at mailman.powerdns.com
>> http://mailman.powerdns.com/mailman/listinfo/pdns-users
>>
> 