[Pdns-users] pdns_control hangs

[Vincent Hoffman]

On 09/01/2013 12:19, Peter van Dijk wrote:
> Hello Vince,
>
> On Jan 7, 2013, at 16:56 , Vincent Hoffman wrote:
>
>>    We are using powerdns pdns-server-3.1 on centos 6 x86_64 with the
>> mydns backend and so far its been working well.
>> However 3 or 4 times now our statistics gathered by munin seem to have
>> stopped working, each time when this happens pdns_control just hangs
>> when run with an argument leaving us with many many instances of it
>> running.
> First off: if this makes you end up with many hanging instances, your statistics 
> gathering script needs some locking.
Quite agree, I'm using munin with the powerdns plugins from their git repo
(https://github.com/munin-monitoring/contrib/tree/master/plugins/network/dns)

I'll have a dig and see what will be needed to make them a little more sane.

>
> On the real issue with pdns_control: we get reports of this every few months,
> but we have not been able to reproduce on our end.
>
> Is there anything in the pdns_server log around the time it starts failing?
Its hard to be precise as it happened over the Christmas break while I
was away. I think looking at when the logging stopped then this pretty
much coincides though

Dec 27 08:26:43 delphi pdns[17139]: 5044 questions waiting for database
attention. Limit is 5000, respawning
Dec 27 08:26:43 delphi pdns[10652]: Our pdns instance exited with code 1
Dec 27 08:26:43 delphi pdns[10652]: Respawning
Dec 27 08:26:44 delphi pdns[19302]: Guardian is launching an instance
Dec 27 08:26:44 delphi pdns[19302]: Reading random entropy from
'/dev/urandom'
Dec 27 08:26:44 delphi pdns[19302]: This is a guarded instance of pdns
Dec 27 08:26:44 delphi pdns[19302]: UDP server bound to 85.xx.xx.xx:53
Dec 27 08:26:44 delphi pdns[19302]: TCP server bound to 85.xx.xx.xx:53
Dec 27 08:26:44 delphi pdns[19302]: PowerDNS 3.1 (C) 2001-2012
PowerDNS.COM BV (Oct 22 2012, 13:07:30, gcc 4.4.6 20120305 (Red Hat
4.4.6-4)) starting up
Dec 27 08:26:44 delphi pdns[19302]: PowerDNS comes with ABSOLUTELY NO
WARRANTY. This is free software, and you are welcome to redistribute it
according to the terms of the GPL version 2.Dec 27 08:26:44 delphi
pdns[19302]: Creating backend connection for TCP
Dec 27 08:26:44 delphi pdns[19302]: [MyDNSbackend] Connection successful
Dec 27 08:26:44 delphi pdns[19302]: [MyDNSbackend] Connection successful
Dec 27 08:26:44 delphi pdns[19302]: About to create 4 backend threads
for UDP
Dec 27 08:26:44 delphi pdns[19302]: [MyDNSbackend] Connection successful
Dec 27 08:26:44 delphi pdns[19302]: [MyDNSbackend] Connection successful
Dec 27 08:26:44 delphi pdns[19302]: [MyDNSbackend] Connection successful
Dec 27 08:26:44 delphi pdns[19302]: [MyDNSbackend] Connection successful
Dec 27 08:26:44 delphi pdns[19302]: [MyDNSbackend] Connection successful
Dec 27 08:26:44 delphi pdns[19302]: [MyDNSbackend] Connection successful
Dec 27 08:26:44 delphi pdns[19302]: [MyDNSbackend] Connection successful
Dec 27 08:26:44 delphi pdns[19302]: [MyDNSbackend] Connection successful
Dec 27 08:26:44 delphi pdns[19302]: Done launching threads, ready to
distribute questions
Dec 27 08:26:47 delphi pdns[19302]: 5046 questions waiting for database
attention. Limit is 5000, respawning
Dec 27 08:26:47 delphi pdns[10652]: Our pdns instance exited with code 1
Dec 27 08:26:47 delphi pdns[10652]: RespawningDec 27 08:26:48 delphi
pdns[19311]: Guardian is launching an instance
Dec 27 08:26:48 delphi pdns[19311]: Reading random entropy from
'/dev/urandom'
Dec 27 08:26:48 delphi pdns[19311]: This is a guarded instance of pdns
Dec 27 08:26:48 delphi pdns[19311]: UDP server bound to 85.xx.xx.xx:53
Dec 27 08:26:48 delphi pdns[19311]: TCP server bound to 85.xx.xx.xx:53
Dec 27 08:26:48 delphi pdns[19311]: PowerDNS 3.1 (C) 2001-2012
PowerDNS.COM BV (Oct 22 2012, 13:07:30, gcc 4.4.6 20120305 (Red Hat
4.4.6-4)) starting up
Dec 27 08:26:48 delphi pdns[19311]: PowerDNS comes with ABSOLUTELY NO
WARRANTY. This is free software, and you are welcome to redistribute it
according to the terms of the GPL version 2.
Dec 27 08:26:48 delphi pdns[19311]: Creating backend connection for TCP
Dec 27 08:26:48 delphi pdns[19311]: [MyDNSbackend] Connection successful
Dec 27 08:26:48 delphi pdns[19311]: [MyDNSbackend] Connection successful
Dec 27 08:26:48 delphi pdns[19311]: About to create 4 backend threads
for UDP
Dec 27 08:26:48 delphi pdns[19311]: [MyDNSbackend] Connection successful
Dec 27 08:26:48 delphi pdns[19311]: [MyDNSbackend] Connection successful
Dec 27 08:26:48 delphi pdns[19311]: [MyDNSbackend] Connection successful
Dec 27 08:26:48 delphi pdns[19311]: [MyDNSbackend] Connection successful
Dec 27 08:26:48 delphi pdns[19311]: [MyDNSbackend] Connection successful
Dec 27 08:26:48 delphi pdns[19311]: [MyDNSbackend] Connection successful
Dec 27 08:26:48 delphi pdns[19311]: [MyDNSbackend] Connection successful
Dec 27 08:26:48 delphi pdns[19311]: [MyDNSbackend] Connection successful
Dec 27 08:26:48 delphi pdns[19311]: Done launching threads, ready to
distribute questions


looking though the log lines such as
Jan 10 07:17:13 delphi pdns[31017]: 5046 questions waiting for database
attention. Limit is 5000, respawning

seem to be disturbingly common, any tuning suggestions? this a pretty
powerful box with reasonably tuned mysql but plenty of spare resources.

Vince

> Kind regards,