[Pdns-users] Backend error: Failed to executemysql_query,perhaps connection died? Err=1: MySQL server has goneaway

Catalin Constantin dazoot at gmail.com
Wed Jan 30 01:03:13 UTC 2008


Hello Kenneth

Having to "restart" the master on a regular basis!
That's quite odd.

Is this a good scenario ?

>>>> If your MySQL DB
>>>> is not dedicated to PDNS, then you will have to balance the needs
>>>> of all of the applications using the DB. It sounds like you may
>>>> be running some pretty poorly written ones. Good luck.

PDNS has it's own dedicated Mysql server and for the future
i would appreciate if we stick to the subject and not talk nonsense
about how our running applications are written.

Cheers,

On Jan 29, 2008 10:27 PM, Kenneth Marshall <ktm at rice.edu> wrote:

> Stephen,
>
> You are completely correct. It is a bandaid (needed) until the
> patched version is available. I am in the same situation currently
> with 2.9.20 and the TCP connection handle problem and have to
> restart my master on a regular basis. Monitoring is definitely
> the way to go for issues such as memory leaks.
>
> Ken
>
> On Tue, Jan 29, 2008 at 12:18:54PM -0800, Stephen Manchester wrote:
> > Increasing the wait_timeout on the mysql server will help, but it does
> not
> > actually solve the memory leak, as dead connections do happen for a
> number
> > of reasons (server failure, network failure, stateful firewalls, etc.)
> >
> > Until a patch can be applied though, increasing the wait_timeout value
> on
> > the MySQL server to something very high (say 1 day or more) is the only
> > thing you can do.  From what I've read, it looks like the default for
> MySQL
> > 4.x was 30s.  The default for 5.x is 28800 seconds, but if there was
> > already a my.cnf file laying around (which there often is a 4.x my.cnffile
> > by default on many distros) then you might inherit this old value which
> > will cause the problem to happen very often.  Again, this is only a work
> > around, not a true fix.  You should still monitor pdns virtual memory
> usage
> > from time to time to make sure it isn't getting crazy.
> >
> > ----- Original Message ----- From: "Kenneth Marshall" <ktm at rice.edu>
> > To: "Stephen Manchester" <smanches at craftyspace.com>
> > Cc: <Pdns-users at mailman.powerdns.com>
> > Sent: Tuesday, January 29, 2008 11:56 AM
> > Subject: Re: [Pdns-users] Backend error: Failed to
> > executemysql_query,perhaps connection died? Err=1: MySQL server has
> > goneaway
> >
> >
> >> Again, this problem, for which a patch already exists, is triggered
> >> by the DB connection being terminated abnormally by the backend
> >> server, in this case via a connection time_limit. I do not use
> >> MySQL as a PDNS backend, but the PostgreSQL backend does correctly
> >> reconnect to the database should the connection die. I assume that
> >> the PDNS will reconnect to the MySQL server when the connection
> >> dies. It seems that increasing the timeout value in MySQL is that
> >> best measure to ensure more robust service from PDNS.
> >>
> >> Cheers,
> >> Ken
> >>
> >>
> >> On Tue, Jan 29, 2008 at 11:43:43AM -0800, Stephen Manchester wrote:
> >>> There is currently a memory leak in 2.9.21 from gmysql backend threads
> >>> dying in this fashion.  The threads themselves are not being cleaned
> up.
> >>> this is only noticeable on very light loads where the backends do not
> get
> >>> enough traffic to keep the mysql connection open.  You will notice it
> by
> >>> abnormally high virtual memory usage after pdns has been running for a
> >>> time.  Each backend that dies leaves 10MB of virtual memory allocated
> >>> (the
> >>> thread stack?) and this will not get cleaned up until pdns is
> restarted.
> >>>
> >>> Hopefully they will implement mysql connection reconnects in the
> future,
> >>> which solves both the memory leak and the backends failing when the
> >>> connection is closed.  The only solution I found was to apply a patch
> and
> >>> compile from source.
> >>>
> >>> ----- Original Message ----- From: "Kenneth Marshall" <ktm at rice.edu>
> >>> To: "Catalin Constantin" <dazoot at gmail.com>
> >>> Cc: <Pdns-users at mailman.powerdns.com>
> >>> Sent: Tuesday, January 29, 2008 5:25 AM
> >>> Subject: Re: [Pdns-users] Backend error: Failed to execute
> >>> mysql_query,perhaps connection died? Err=1: MySQL server has gone away
> >>>
> >>>
> >>>> On Tue, Jan 29, 2008 at 03:04:46PM +0200, Catalin Constantin wrote:
> >>>>> Hello,
> >>>>>
> >>>>> This is a funny question.
> >>>>> Mysql is FINE and it's up and running on localhost.
> >>>>>
> >>>>> Here is the startup log of PDNS.
> >>>>> Jan 29 14:59:05 k2 pdns[30294]: PowerDNS 2.9.20 (C) 2001-2006
> >>>>> PowerDNS.COMBV (Mar 10 2007, 00:36:58, gcc
> >>>>> 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)) starting up
> >>>>> Jan 29 14:59:05 k2 pdns[30294]: PowerDNS comes with ABSOLUTELY NO
> >>>>> WARRANTY.
> >>>>> This is free software, and you are welcome to redistribute it
> according
> >>>>> to
> >>>>> the terms of the GPL version 2.
> >>>>> Jan 29 14:59:05 k2 pdns[30294]: Set effective group id to 109
> >>>>> Jan 29 14:59:05 k2 pdns[30294]: Set effective user id to 113
> >>>>> Jan 29 14:59:05 k2 pdns[30294]: Creating backend connection for TCP
> >>>>> Jan 29 14:59:05 k2 pdns[30294]: Master/slave communicator launching
> >>>>> Jan 29 14:59:05 k2 pdns[30294]: gmysql Connection succesful
> >>>>> Jan 29 14:59:05 k2 pdns[30294]: gmysql Connection succesful
> >>>>> Jan 29 14:59:05 k2 pdns[30294]: About to create 10 backend threads
> for
> >>>>> UDP
> >>>>> Jan 29 14:59:05 k2 pdns[30294]: gmysql Connection succesful
> >>>>> Jan 29 14:59:05 k2 pdns[30294]: 14 slave domains need checking
> >>>>> Jan 29 14:59:05 k2 pdns[30294]: Domain test1.com is stale, master
> >>>>> serial
> >>>>> 1201611548, our serial 1201610827
> >>>>> Jan 29 14:59:05 k2 pdns[30294]: Domain test2.com is stale, master
> >>>>> serial
> >>>>> 1201611548, our serial 1201610827
> >>>>> Jan 29 14:59:05 k2 pdns[30294]: Domain test3.com is stale, master
> >>>>> serial
> >>>>> 1201611548, our serial 1201610827
> >>>>> Jan 29 14:59:05 k2 pdns[30294]: gmysql Connection succesful
> >>>>> Jan 29 14:59:05 k2 pdns[30294]: AXFR started for 'test1.com',
> >>>>> transaction
> >>>>> started
> >>>>> Jan 29 14:59:05 k2 pdns[30294]: gmysql Connection succesful
> >>>>> Jan 29 14:59:05 k2 pdns[30294]: AXFR done for 'test1.com', zone
> >>>>> committed
> >>>>>
> >>>>>
> >>>>> The problem is that mysql closes the connection if there is no
> >>>>> activity.
> >>>>> PDNS does not make sure that MySQL is still "alive" and from time to
> >>>>> time
> >>>>> it
> >>>>> shows in the log errors like:
> >>>>> Jan 29 11:33:57 k2 pdns[28941]: Backend error: Failed to execute
> >>>>> mysql_query, perhaps connection died? Err=1: MySQL server has gone
> away
> >>>>>
> >>>>> Does PDNS have a timeout for backend connection "recheck" ?
> >>>>> Can this be configured ?
> >>>>>
> >>>>> Is there anything we can do to make sure the connection does not go
> >>>>> away
> >>>>> ?
> >>>>>
> >>>>> Note: the NS server activity for now is not much (we're still
> testing
> >>>>> and
> >>>>> just have a couple of domains on it).
> >>>>>
> >>>>> Thanks,
> >>>>>
> >>>> Dear Ms. Constantin,
> >>>>
> >>>> If a properly functioning application opens a connection to a
> database,
> >>>> the connection should not drop until closed. The typical fix is to
> >>>> increase your timeout on your MySQL server appropriately. This same
> >>>> thread comes up periodically. In order for PDNS to notice that a
> >>>> connection has died, it tries to use it with a query. This causes
> >>>> the errors in your logs. The amount of errors will be proportional
> >>>> to how long your timeout is set. I certainly would not like to see
> >>>> a patch that added overhead to every query to check the status of
> >>>> a connection and re-establish if needed. PDNS will re-connect if
> >>>> the connection drops, but you will get an error. If your MySQL DB
> >>>> is not dedicated to PDNS, then you will have to balance the needs
> >>>> of all of the applications using the DB. It sounds like you may
> >>>> be running some pretty poorly written ones. Good luck.
> >>>>
> >>>> Ken
> >>>>
> >>>>> On 1/29/08, Nico van Royen <nico at van-royen.nl> wrote:
> >>>>> > Hello,
> >>>>> >
> >>>>> > Do you infact have a running mysql server ?  Also check the
> database
> >>>>> >  >
> >>>>> that
> >>>>> > you have specified in your pdns.conf matches your actual database
> >>>>> > information (db name, user/password etc).
> >>>>> >
> >>>>> > Regards,
> >>>>> > Nico
> >>>>> > WARP3.COM
> >>>>> > ________________________________
> >>>>> > From: Catalin Constantin [mailto:dazoot at gmail.com]
> >>>>> > To: pdns-users at mailman.powerdns.com
> >>>>> > Sent: Tue, 29 Jan 2008 10:48:09 +0100
> >>>>> > Subject: [Pdns-users] Backend error: Failed to execute
> mysql_query,
> >>>>> perhaps
> >>>>> > connection died? Err=1: MySQL server has gone away
> >>>>> >
> >>>>> >
> >>>>> > Hello,
> >>>>> >
> >>>>> > We're getting quite many errors:
> >>>>> > Backend error: Failed to execute mysql_query, perhaps connection >
> >>>>> died?
> >>>>> > Err=1: MySQL server has gone away latelly.
> >>>>> >
> >>>>> > I checked the docs and the mailing lists for this error and i
> found
> >>>>> > little informations about it.
> >>>>> >
> >>>>> > What is actually the solution for this error not to happen ?
> >>>>> > Nice would be to have a real example.
> >>>>> >
> >>>>> > Note:
> >>>>> > - we use latest pdns package from Debian Etch (stable).
> >>>>> >
> >>>>> > Thanks,
> >>>>> >
> >>>>> > --
> >>>>> > Catalin Constantin
> >>>>> > Dazoot Software
> >>>>> > http://www.dazoot.eu/
> >>>>> > _______________________________________________
> >>>>> > Pdns-users mailing list
> >>>>> > Pdns-users at mailman.powerdns.com
> >>>>> > http://mailman.powerdns.com/mailman/listinfo/pdns-users
> >>>>> >
> >>>>> > _______________________________________________
> >>>>> > Pdns-users mailing list
> >>>>> > Pdns-users at mailman.powerdns.com
> >>>>> > http://mailman.powerdns.com/mailman/listinfo/pdns-users
> >>>>> >
> >>>>> >
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Catalin Constantin
> >>>>> Dazoot Software
> >>>>> http://www.dazoot.eu/
> >>>>
> >>>>> _______________________________________________
> >>>>> Pdns-users mailing list
> >>>>> Pdns-users at mailman.powerdns.com
> >>>>> http://mailman.powerdns.com/mailman/listinfo/pdns-users
> >>>>
> >>>> _______________________________________________
> >>>> Pdns-users mailing list
> >>>> Pdns-users at mailman.powerdns.com
> >>>> http://mailman.powerdns.com/mailman/listinfo/pdns-users
> >>>
> >>> _______________________________________________
> >>> Pdns-users mailing list
> >>> Pdns-users at mailman.powerdns.com
> >>> http://mailman.powerdns.com/mailman/listinfo/pdns-users
> >>>
> >
> > _______________________________________________
> > Pdns-users mailing list
> > Pdns-users at mailman.powerdns.com
> > http://mailman.powerdns.com/mailman/listinfo/pdns-users
> >
> _______________________________________________
> Pdns-users mailing list
> Pdns-users at mailman.powerdns.com
> http://mailman.powerdns.com/mailman/listinfo/pdns-users
>



-- 
Catalin Constantin
Dazoot Software
http://www.dazoot.eu/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.powerdns.com/pipermail/pdns-users/attachments/20080130/a4740e24/attachment-0001.html>


More information about the Pdns-users mailing list