[Pdns-users] Problems with PowerDNS
Giuseppe Ravasio
giuseppe_ravasio at egontek.net
Wed Nov 11 09:28:49 UTC 2015
Are You doing recursion with authoritative name server?
Do you really need to have both recursor and authoritative together?
You would absolutely achieve better performance with two distinct services!
With this load you should consider that:
"To make sure that the local authoritative database overrides recursive
information, PowerDNS first tries to answer a question from its own
database. If that succeeds, the answer packet is sent back immediately
without involving the recursor in any way. This means that for questions
for which there is no answer, PowerDNS will consult the recursor for an
recursive query, even if PowerDNS is authoritative for a domain! This
will only cause problems if you 'fake' domains which don't really exist.
This also means that if you delegate a subzone to another set or
authoritative servers, when a request comes in for that sub-zone,
PowerDNS will respond with a delegation response (as that is the answer
from the authoritative perspective) and will not involve the recursor."
https://doc.powerdns.com/md/authoritative/recursion/
Giuseppe
On 11/11/2015 10:07 AM, Nadir M. Aliyev wrote:
> Dear Aki,
>
> Yes I have indexes.
>
> But pdns sends everytime bulk query for every dns query even not exists in db. Sometimes cache hits sometimes miss.
>
> query-cache-ttl=18600
> cache-ttl=18600
> default-ttl=7200
> soa-expire-default=18600
> soa-minimum-ttl=3600
> soa-refresh-default=10800
> soa-retry-default=3600
> max-cache-entries=10000000
>
> For ex. Nslookup google.com
>
> Nov 11 13:01:51 ns01 pdns[7180]: Remote 127.0.0.1 wants 'google.com|A', do = 0, bufsize = 512: packetcache MISS
> Nov 11 13:01:51 ns01 pdns[7180]: Query: SELECT content,ttl,prio,type,domain_id,disabled,name,auth FROM records WHERE disabled=0 and type='SOA' and name='google.com'
> Nov 11 13:01:51 ns01 pdns[7180]: Query: SELECT content,ttl,prio,type,domain_id,disabled,name,auth FROM records WHERE disabled=0 and type='SOA' and name='com'
> Nov 11 13:01:51 ns01 pdns[7180]: Query: SELECT content,ttl,prio,type,domain_id,disabled,name,auth FROM records WHERE disabled=0 and type='SOA' and name=''
> Remote 127.0.0.1 wants 'google.com|A', do = 0, bufsize = 512: packetcache MISS
> Nov 11 13:03:05 ns01 pdns[7180]: Query: SELECT content,ttl,prio,type,domain_id,disabled,name,auth FROM records WHERE disabled=0 and type='SOA' and name='google.com'
> Nov 11 13:03:05 ns01 pdns[7180]: Query: SELECT content,ttl,prio,type,domain_id,disabled,name,auth FROM records WHERE disabled=0 and type='SOA' and name='com'
> Nov 11 13:03:05 ns01 pdns[7180]: Query: SELECT content,ttl,prio,type,domain_id,disabled,name,auth FROM records WHERE disabled=0 and type='SOA' and name=''
> Nov 11 13:03:10 ns01 pdns[7180]: Remote 127.0.0.1 wants 'google.com|A', do = 0, bufsize = 512: packetcache HIT
>
>
> I think that community need to do some works on pdns.
> For example caching db zones from db, it will decrase server load. And flush it with rec_control anytime like rec_control purge dbcache :)
>
> Strange that, when I try nslookup to google.com after service restarts it works. After some time I got serv refused. Cache life too high. But why domain google.com does not hit in cache?
>
> It happens with many domains but not all. Google.com example for this.
>
> Named.root file I downloaded from IANA website. And up to date.
>
> # rec_control get cache-entries packetcache-entries
> 457723
> 284453
>
> # rec_control get-all
> all-outqueries 672092
> answers-slow 17055
> answers0-1 73094
> answers1-10 3696
> answers10-100 221413
> answers100-1000 174393
> cache-entries 458482
> cache-hits 57934
> cache-misses 431721
> case-mismatches 0
> chain-resends 32382
> client-parse-errors 0
> concurrent-queries 0
> dlg-only-drops 0
> dont-outqueries 132
> edns-ping-matches 0
> edns-ping-mismatches 0
> failed-host-entries 4346
> ipv6-outqueries 0
> ipv6-questions 0
> malloc-bytes 0
> max-mthread-stack 39560
> negcache-entries 91062
> no-packet-error 1026559
> noedns-outqueries 672122
> noerror-answers 927841
> noping-outqueries 0
> nsset-invalidations 2573
> nsspeeds-entries 41285
> nxdomain-answers 125690
> outgoing-timeouts 23292
> over-capacity-drops 0
> packetcache-entries 284954
> packetcache-hits 599140
> packetcache-misses 489559
> policy-drops 0
> qa-latency 75260
> questions 1088780
> resource-limits 0
> security-status 0
> server-parse-errors 0
> servfail-answers 35171
> spoof-prevents 0
> sys-msec 74879
> tcp-client-overflow 0
> tcp-clients 0
> tcp-outqueries 177
> tcp-questions 96
> throttle-entries 4454
> throttled-out 38905
> throttled-outqueries 38905
> too-old-drops 0
> unauthorized-tcp 0
> unauthorized-udp 0
> unexpected-packets 0
> unreachables 1505
> uptime 1277
> user-msec 145122
>
> -----Original Message-----
> From: Aki Tuomi [mailto:cmouse at youzen.ext.b2.fi]
> Sent: 11 noyabr 2015, çərşənbə 12:22
> To: Nadir M. Aliyev <admin at bakinter.net>
> Cc: 'Patrick Domack' <patrickdk at patrickdk.com>; pdns-users at mailman.powerdns.com
> Subject: Re: [Pdns-users] Problems with PowerDNS
>
> Does your database have indexes? We have had few cases before where the user had forgotten to add indexes to the database.
>
> Aki
>
> On Wed, Nov 11, 2015 at 12:10:17PM +0400, Nadir M. Aliyev wrote:
>> Dear Patrick,
>>
>> I tried to set
>>
>> gmysql-dnssec="no"
>> distributor-threads=10
>> receiver-threads=5
>>
>> Now:
>> Mysql 110%
>> Pdns_server 90 %
>> Pdns_recursor 25%
>>
>>
>> But after 10-15 minutes again I got from some domains SERVFAIL..
>>
>> [root at ns01 ~]# nslookup google.com
>> Server: 127.0.0.1
>> Address: 127.0.0.1#53
>>
>> ** server can't find google.com: REFUSED
>>
>> And logs:
>> Nov 11 12:08:59 ns01 pdns_recursor[4559]: Sending SERVFAIL to
>> 127.0.0.1 during resolve of 'ad.bb800.com.' because: Too much time
>> waiting for ad.6gg.cn.|A, timeouts: 5, throttles: 0, queries: 6,
>> 7506msec Nov 11 12:09:04 ns01 pdns_recursor[4559]: Sending SERVFAIL to
>> 127.0.0.1 during resolve of 'ad.bb800.com.' because: Too much time
>> waiting for ad.6gg.cn.|A, timeouts: 5, throttles: 5, queries: 6,
>> 7503msec Nov 11 12:09:09 ns01 pdns_recursor[4559]: Sending SERVFAIL to
>> 127.0.0.1 during resolve of 'wx.qq.com.' because: Too much time
>> waiting for wx1.qq.com.|A, timeouts: 5, throttles: 0, queries: 8,
>> 8219msec Nov 11 12:09:34 ns01 pdns_recursor[4559]: Sending SERVFAIL to
>> 127.0.0.1 during resolve of 'dev.voicecloud.cn.' because: Too much
>> time waiting for dev.voicecloud.cn.|A, timeouts: 4, throttles: 0,
>> queries: 9, 7087msec Nov 11 12:09:38 ns01 pdns_recursor[4559]: Sending
>> SERVFAIL to 127.0.0.1 during resolve of '79.208.218.41.in-addr.arpa.'
>> because: Too much time waiting for 79.208.218.41.in-addr.arpa.|PTR,
>> timeouts: 4, throttles: 0, queries: 13, 7007msec Nov 11 12:09:43 ns01
>> pdns_recursor[4559]: Sending SERVFAIL to 127.0.0.1 during resolve of
>> '61.29.19.113.in-addr.arpa.' because: Too much time waiting for
>> 61.29.19.113.in-addr.arpa.|PTR, timeouts: 4, throttles: 0, queries:
>> 11, 7928msec Nov 11 12:09:49 ns01 pdns_recursor[4559]: Sending
>> SERVFAIL to 127.0.0.1 during resolve of '50.25.36.204.in-addr.arpa.'
>> because: Too much time waiting for 50.25.36.204.in-addr.arpa.|PTR,
>> timeouts: 5, throttles: 0, queries: 7, 7587msec
>>
>>
>> -----Original Message-----
>> From: pdns-users-bounces at mailman.powerdns.com
>> [mailto:pdns-users-bounces at mailman.powerdns.com] On Behalf Of Patrick
>> Domack
>> Sent: 11 noyabr 2015, çərşənbə 01:08
>> To: pdns-users at mailman.powerdns.com
>> Subject: Re: [Pdns-users] Problems with PowerDNS
>>
>> I suppose sense you have dnssec=yes, you are using dnssec, This will cause a lot of sql queries.
>>
>> pdns is using 100% cpu of a single core, did you try adjusting receiver-threads >1 probably for that box set it to 4 and test, maybe higher even.
>>
>> Since I don't know much about what your pdns server is doing (and I haven't had issues on mine), I assume the dnssec dynamic signing is eating your cpu, and it only has one worker thread to do it with, limiting it to a single core.
>>
>> I could be completely wrong.
>>
>>
>> Quoting "Nadir M. Aliyev" <admin at bakinter.net>:
>>
>>> Dear Peter van Dijk, my connection link is 1000Gbps, server hardware
>>> from cisco ucs. There is no problem with hardware. But mysql uses
>>> huge resources even not zone in db it sends 4-5 queries to the db.
>>>
>>> I used percone tools to optimize mysql configuration. But it
>>> decreased cpu usage only 10%. I have 10.000 query per second.
>>>
>>> Maybe I need do some tuning on TTLs?
>>>
>>> -----Original Message-----
>>> From: pdns-users-bounces at mailman.powerdns.com
>>> [mailto:pdns-users-bounces at mailman.powerdns.com] On Behalf Of Peter
>>> van Dijk
>>> Sent: 10 noyabr 2015, çərşənbə axşamı 16:58
>>> To: pdns-users at mailman.powerdns.com
>>> Subject: Re: [Pdns-users] Problems with PowerDNS
>>>
>>> Hello Nadir,
>>>
>>> based on the logs, it looks like your powerdns has trouble reaching
>>> the Internet at all. Are you on a slow or congested link? Note that
>>> in general your machine looks quite busy!
>>>
>>> Kind regards,
>>> --
>>> Peter van Dijk
>>> PowerDNS.COM BV - https://www.powerdns.com/
>>>
>>> On 10 Nov 2015, at 13:01, Nadir M. Aliyev wrote:
>>>
>>>> Hi everyone!
>>>>
>>>>
>>>>
>>>> I have problems with some domains
>>>>
>>>>
>>>>
>>>> For ex. When I do google.com sometimes I get ns records but
>>>> sometimes I get SERFVAIL also it happens basically with google.
>>>> When I restrart pdns it works normally for 5 minutes. Then again SERVFAIL.
>>>>
>>>>
>>>>
>>>> Strange, some domains works some not works.. Even if cache hits.
>>>>
>>>> I increased cache ttls not helped.
>>>>
>>>>
>>>>
>>>> Server details: 8 core cpu, 8 GB of Ram.
>>>>
>>>> Load: pdns 100%, mysql 120%, pdns-recursor 30%, network 40 mbps.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Some logs:
>>>>
>>>> Nov 10 15:33:08 ns01 pdns_recursor[15237]: Sending SERVFAIL to
>>>> 127.0.0.1
>>>> during resolve of 'gm-realm.net.' because: Too much time waiting
>>>> for gm-realm.net.|A, timeouts: 5, throttles: 1, queries: 6,
>>>> 7578msec
>>>>
>>>> Nov 10 15:33:09 ns01 pdns_recursor[15237]: Sending SERVFAIL to
>>>> 127.0.0.1
>>>> during resolve of 'gm-realm.net.' because: Too much time waiting
>>>> for gm-realm.net.|A, timeouts: 5, throttles: 2, queries: 6,
>>>> 7504msec
>>>>
>>>> Nov 10 15:33:12 ns01 pdns_recursor[15237]: Sending SERVFAIL to
>>>> 127.0.0.1
>>>> during resolve of 'gm-realm.net.' because: Too much time waiting
>>>> for gm-realm.net.|A, timeouts: 5, throttles: 3, queries: 6,
>>>> 7502msec
>>>>
>>>> Nov 10 15:33:13 ns01 pdns_recursor[15237]: Sending SERVFAIL to
>>>> 127.0.0.1
>>>> during resolve of 'us.micardapi.micloud.xiaomi.net.' because: Too
>>>> much time waiting for us.api.micloud.mi.com.|A, timeouts: 5,
>>>> throttles: 0,
>>>> queries: 7,
>>>> 7709msec
>>>>
>>>> Nov 10 15:33:18 ns01 pdns_recursor[15237]: Sending SERVFAIL to
>>>> 127.0.0.1
>>>> during resolve of 'www.coocent.net.' because: Too much time waiting
>>>> for s-149179.abc188.com.|A, timeouts: 5, throttles: 0, queries: 8,
>>>> 8093msec
>>>>
>>>> Nov 10 15:33:18 ns01 pdns_recursor[15237]: Sending SERVFAIL to
>>>> 127.0.0.1
>>>> during resolve of 'www.6ud1.com.' because: Too much time waiting
>>>> for www.6ud1.com.|A, timeouts: 5, throttles: 0, queries: 6,
>>>> 7502msec
>>>>
>>>> Nov 10 15:42:52 ns01 pdns_recursor[15237]: stats: 1787915
>>>> questions,
>>>> 497334
>>>> cache entries, 86066 negative entries, 11% cache hits
>>>>
>>>> Nov 10 15:42:52 ns01 pdns_recursor[15237]: stats: throttle map:
>>>> 6856, ns
>>>> speeds: 29645
>>>>
>>>> Nov 10 15:42:52 ns01 pdns_recursor[15237]: stats: outpacket/query
>>>> ratio 49%, 11% throttled, 0 no-delegation drops
>>>>
>>>> Nov 10 15:42:52 ns01 pdns_recursor[15237]: stats: 211 outgoing tcp
>>>> connections, 1 queries running, 50712 outgoing timeouts
>>>>
>>>> Nov 10 15:42:52 ns01 pdns_recursor[15237]: stats: 322566 packet
>>>> cache entries, 61% packet cache hits
>>>>
>>>> Nov 10 15:42:52 ns01 pdns_recursor[15237]: stats: 926 qps (average
>>>> over 1930
>>>> seconds)
>>>>
>>>>
>>>>
>>>> Config:
>>>>
>>>>
>>>>
>>>> I have one master server which replicates db to the four slave server.
>>>>
>>>>
>>>>
>>>> # cat recursor.conf
>>>>
>>>> ..
>>>>
>>>> hint-file=/etc/pdns/named.root
>>>>
>>>> allow-from=127.0.0.0/8
>>>>
>>>> local-address=127.0.0.1
>>>>
>>>> local-port=5353
>>>>
>>>> version-string=Bind Recursor
>>>>
>>>> ..
>>>>
>>>>
>>>>
>>>> # cat /etc/pdns/pdns.conf
>>>>
>>>> ..
>>>>
>>>> launch=gmysql
>>>>
>>>> gmysql-host=127.0.0.1
>>>>
>>>> gmysql-port=3306
>>>>
>>>> gmysql-user=p_owerdns
>>>>
>>>> gmysql-password=verysecretpassword
>>>>
>>>> gmysql-dbname=p_ owerdns
>>>>
>>>> gmysql-dnssec="yes"
>>>>
>>>>
>>>>
>>>> #allow to customers
>>>>
>>>> allow-recursion=127.0.0.1/8, 172.16.0.0/16, 10.0.0.0/8,
>>>> xxx.xxx.xxx.xxx/16
>>>>
>>>>
>>>>
>>>> #master
>>>>
>>>> #allow-axfr-ips=172.16.6.30
>>>>
>>>>
>>>>
>>>> local-address=0.0.0.0
>>>>
>>>> local-port=53
>>>>
>>>>
>>>>
>>>> control-console=no
>>>>
>>>>
>>>>
>>>> query-cache-ttl=18600
>>>>
>>>> cache-ttl=18600
>>>>
>>>> default-ttl=7200
>>>>
>>>> soa-expire-default=18600
>>>>
>>>> soa-minimum-ttl=3600
>>>>
>>>> soa-refresh-default=10800
>>>>
>>>> soa-retry-default=3600
>>>>
>>>>
>>>>
>>>> daemon=yes
>>>>
>>>>
>>>>
>>>> default-soa-name=ns.master.mydomain.net
>>>>
>>>>
>>>>
>>>> distributor-threads=18
>>>>
>>>>
>>>>
>>>> guardian=yes
>>>>
>>>>
>>>>
>>>> #lazy-recursion=yes
>>>>
>>>>
>>>>
>>>> master=no
>>>>
>>>> slave=yes
>>>>
>>>> slave-cycle-interval=600
>>>>
>>>>
>>>>
>>>> max-tcp-connections=100
>>>>
>>>> max-queue-length=50000
>>>>
>>>>
>>>>
>>>> recursor=127.0.0.1:5353
>>>>
>>>>
>>>>
>>>> out-of-zone-additional-processing=yes
>>>>
>>>>
>>>>
>>>> webserver=yes
>>>>
>>>> webserver-address=172.16.6.34
>>>>
>>>> webserver-password=adminadminadmin
>>>>
>>>> webserver-port=8081
>>>>
>>>> webserver-print-arguments=yes
>>>>
>>>>
>>>>
>>>> #loglevel=9
>>>>
>>>> #log-dns-details=yes
>>>>
>>>> #log-dns-queries=yes
>>>>
>>>> #query-logging=yes
>>>>
>>>>
>>>>
>>>> version-string=Bind Resolver
>>>>
>>>> ..
>>>>
>>>> _______________________________________________
>>>> Pdns-users mailing list
>>>> Pdns-users at mailman.powerdns.com
>>>> http://mailman.powerdns.com/mailman/listinfo/pdns-users
>>>
>>> _______________________________________________
>>> Pdns-users mailing list
>>> Pdns-users at mailman.powerdns.com
>>> http://mailman.powerdns.com/mailman/listinfo/pdns-users
>>>
>>>
>>> _______________________________________________
>>> Pdns-users mailing list
>>> Pdns-users at mailman.powerdns.com
>>> http://mailman.powerdns.com/mailman/listinfo/pdns-users
>>
>>
>>
>>
>> _______________________________________________
>> Pdns-users mailing list
>> Pdns-users at mailman.powerdns.com
>> http://mailman.powerdns.com/mailman/listinfo/pdns-users
>>
>>
>> _______________________________________________
>> Pdns-users mailing list
>> Pdns-users at mailman.powerdns.com
>> http://mailman.powerdns.com/mailman/listinfo/pdns-users
>
>
> _______________________________________________
> Pdns-users mailing list
> Pdns-users at mailman.powerdns.com
> http://mailman.powerdns.com/mailman/listinfo/pdns-users
>
More information about the Pdns-users
mailing list