[Pdns-users] Problems with PowerDNS
Nadir M. Aliyev
admin at bakinter.net
Wed Nov 11 09:07:36 UTC 2015
Dear Aki,
Yes I have indexes.
But pdns sends everytime bulk query for every dns query even not exists in db. Sometimes cache hits sometimes miss.
query-cache-ttl=18600
cache-ttl=18600
default-ttl=7200
soa-expire-default=18600
soa-minimum-ttl=3600
soa-refresh-default=10800
soa-retry-default=3600
max-cache-entries=10000000
For ex. Nslookup google.com
Nov 11 13:01:51 ns01 pdns[7180]: Remote 127.0.0.1 wants 'google.com|A', do = 0, bufsize = 512: packetcache MISS
Nov 11 13:01:51 ns01 pdns[7180]: Query: SELECT content,ttl,prio,type,domain_id,disabled,name,auth FROM records WHERE disabled=0 and type='SOA' and name='google.com'
Nov 11 13:01:51 ns01 pdns[7180]: Query: SELECT content,ttl,prio,type,domain_id,disabled,name,auth FROM records WHERE disabled=0 and type='SOA' and name='com'
Nov 11 13:01:51 ns01 pdns[7180]: Query: SELECT content,ttl,prio,type,domain_id,disabled,name,auth FROM records WHERE disabled=0 and type='SOA' and name=''
Remote 127.0.0.1 wants 'google.com|A', do = 0, bufsize = 512: packetcache MISS
Nov 11 13:03:05 ns01 pdns[7180]: Query: SELECT content,ttl,prio,type,domain_id,disabled,name,auth FROM records WHERE disabled=0 and type='SOA' and name='google.com'
Nov 11 13:03:05 ns01 pdns[7180]: Query: SELECT content,ttl,prio,type,domain_id,disabled,name,auth FROM records WHERE disabled=0 and type='SOA' and name='com'
Nov 11 13:03:05 ns01 pdns[7180]: Query: SELECT content,ttl,prio,type,domain_id,disabled,name,auth FROM records WHERE disabled=0 and type='SOA' and name=''
Nov 11 13:03:10 ns01 pdns[7180]: Remote 127.0.0.1 wants 'google.com|A', do = 0, bufsize = 512: packetcache HIT
I think that community need to do some works on pdns.
For example caching db zones from db, it will decrase server load. And flush it with rec_control anytime like rec_control purge dbcache :)
Strange that, when I try nslookup to google.com after service restarts it works. After some time I got serv refused. Cache life too high. But why domain google.com does not hit in cache?
It happens with many domains but not all. Google.com example for this.
Named.root file I downloaded from IANA website. And up to date.
# rec_control get cache-entries packetcache-entries
457723
284453
# rec_control get-all
all-outqueries 672092
answers-slow 17055
answers0-1 73094
answers1-10 3696
answers10-100 221413
answers100-1000 174393
cache-entries 458482
cache-hits 57934
cache-misses 431721
case-mismatches 0
chain-resends 32382
client-parse-errors 0
concurrent-queries 0
dlg-only-drops 0
dont-outqueries 132
edns-ping-matches 0
edns-ping-mismatches 0
failed-host-entries 4346
ipv6-outqueries 0
ipv6-questions 0
malloc-bytes 0
max-mthread-stack 39560
negcache-entries 91062
no-packet-error 1026559
noedns-outqueries 672122
noerror-answers 927841
noping-outqueries 0
nsset-invalidations 2573
nsspeeds-entries 41285
nxdomain-answers 125690
outgoing-timeouts 23292
over-capacity-drops 0
packetcache-entries 284954
packetcache-hits 599140
packetcache-misses 489559
policy-drops 0
qa-latency 75260
questions 1088780
resource-limits 0
security-status 0
server-parse-errors 0
servfail-answers 35171
spoof-prevents 0
sys-msec 74879
tcp-client-overflow 0
tcp-clients 0
tcp-outqueries 177
tcp-questions 96
throttle-entries 4454
throttled-out 38905
throttled-outqueries 38905
too-old-drops 0
unauthorized-tcp 0
unauthorized-udp 0
unexpected-packets 0
unreachables 1505
uptime 1277
user-msec 145122
-----Original Message-----
From: Aki Tuomi [mailto:cmouse at youzen.ext.b2.fi]
Sent: 11 noyabr 2015, çərşənbə 12:22
To: Nadir M. Aliyev <admin at bakinter.net>
Cc: 'Patrick Domack' <patrickdk at patrickdk.com>; pdns-users at mailman.powerdns.com
Subject: Re: [Pdns-users] Problems with PowerDNS
Does your database have indexes? We have had few cases before where the user had forgotten to add indexes to the database.
Aki
On Wed, Nov 11, 2015 at 12:10:17PM +0400, Nadir M. Aliyev wrote:
> Dear Patrick,
>
> I tried to set
>
> gmysql-dnssec="no"
> distributor-threads=10
> receiver-threads=5
>
> Now:
> Mysql 110%
> Pdns_server 90 %
> Pdns_recursor 25%
>
>
> But after 10-15 minutes again I got from some domains SERVFAIL..
>
> [root at ns01 ~]# nslookup google.com
> Server: 127.0.0.1
> Address: 127.0.0.1#53
>
> ** server can't find google.com: REFUSED
>
> And logs:
> Nov 11 12:08:59 ns01 pdns_recursor[4559]: Sending SERVFAIL to
> 127.0.0.1 during resolve of 'ad.bb800.com.' because: Too much time
> waiting for ad.6gg.cn.|A, timeouts: 5, throttles: 0, queries: 6,
> 7506msec Nov 11 12:09:04 ns01 pdns_recursor[4559]: Sending SERVFAIL to
> 127.0.0.1 during resolve of 'ad.bb800.com.' because: Too much time
> waiting for ad.6gg.cn.|A, timeouts: 5, throttles: 5, queries: 6,
> 7503msec Nov 11 12:09:09 ns01 pdns_recursor[4559]: Sending SERVFAIL to
> 127.0.0.1 during resolve of 'wx.qq.com.' because: Too much time
> waiting for wx1.qq.com.|A, timeouts: 5, throttles: 0, queries: 8,
> 8219msec Nov 11 12:09:34 ns01 pdns_recursor[4559]: Sending SERVFAIL to
> 127.0.0.1 during resolve of 'dev.voicecloud.cn.' because: Too much
> time waiting for dev.voicecloud.cn.|A, timeouts: 4, throttles: 0,
> queries: 9, 7087msec Nov 11 12:09:38 ns01 pdns_recursor[4559]: Sending
> SERVFAIL to 127.0.0.1 during resolve of '79.208.218.41.in-addr.arpa.'
> because: Too much time waiting for 79.208.218.41.in-addr.arpa.|PTR,
> timeouts: 4, throttles: 0, queries: 13, 7007msec Nov 11 12:09:43 ns01
> pdns_recursor[4559]: Sending SERVFAIL to 127.0.0.1 during resolve of
> '61.29.19.113.in-addr.arpa.' because: Too much time waiting for
> 61.29.19.113.in-addr.arpa.|PTR, timeouts: 4, throttles: 0, queries:
> 11, 7928msec Nov 11 12:09:49 ns01 pdns_recursor[4559]: Sending
> SERVFAIL to 127.0.0.1 during resolve of '50.25.36.204.in-addr.arpa.'
> because: Too much time waiting for 50.25.36.204.in-addr.arpa.|PTR,
> timeouts: 5, throttles: 0, queries: 7, 7587msec
>
>
> -----Original Message-----
> From: pdns-users-bounces at mailman.powerdns.com
> [mailto:pdns-users-bounces at mailman.powerdns.com] On Behalf Of Patrick
> Domack
> Sent: 11 noyabr 2015, çərşənbə 01:08
> To: pdns-users at mailman.powerdns.com
> Subject: Re: [Pdns-users] Problems with PowerDNS
>
> I suppose sense you have dnssec=yes, you are using dnssec, This will cause a lot of sql queries.
>
> pdns is using 100% cpu of a single core, did you try adjusting receiver-threads >1 probably for that box set it to 4 and test, maybe higher even.
>
> Since I don't know much about what your pdns server is doing (and I haven't had issues on mine), I assume the dnssec dynamic signing is eating your cpu, and it only has one worker thread to do it with, limiting it to a single core.
>
> I could be completely wrong.
>
>
> Quoting "Nadir M. Aliyev" <admin at bakinter.net>:
>
> > Dear Peter van Dijk, my connection link is 1000Gbps, server hardware
> > from cisco ucs. There is no problem with hardware. But mysql uses
> > huge resources even not zone in db it sends 4-5 queries to the db.
> >
> > I used percone tools to optimize mysql configuration. But it
> > decreased cpu usage only 10%. I have 10.000 query per second.
> >
> > Maybe I need do some tuning on TTLs?
> >
> > -----Original Message-----
> > From: pdns-users-bounces at mailman.powerdns.com
> > [mailto:pdns-users-bounces at mailman.powerdns.com] On Behalf Of Peter
> > van Dijk
> > Sent: 10 noyabr 2015, çərşənbə axşamı 16:58
> > To: pdns-users at mailman.powerdns.com
> > Subject: Re: [Pdns-users] Problems with PowerDNS
> >
> > Hello Nadir,
> >
> > based on the logs, it looks like your powerdns has trouble reaching
> > the Internet at all. Are you on a slow or congested link? Note that
> > in general your machine looks quite busy!
> >
> > Kind regards,
> > --
> > Peter van Dijk
> > PowerDNS.COM BV - https://www.powerdns.com/
> >
> > On 10 Nov 2015, at 13:01, Nadir M. Aliyev wrote:
> >
> >> Hi everyone!
> >>
> >>
> >>
> >> I have problems with some domains
> >>
> >>
> >>
> >> For ex. When I do google.com sometimes I get ns records but
> >> sometimes I get SERFVAIL also it happens basically with google.
> >> When I restrart pdns it works normally for 5 minutes. Then again SERVFAIL.
> >>
> >>
> >>
> >> Strange, some domains works some not works.. Even if cache hits.
> >>
> >> I increased cache ttls not helped.
> >>
> >>
> >>
> >> Server details: 8 core cpu, 8 GB of Ram.
> >>
> >> Load: pdns 100%, mysql 120%, pdns-recursor 30%, network 40 mbps.
> >>
> >>
> >>
> >>
> >>
> >> Some logs:
> >>
> >> Nov 10 15:33:08 ns01 pdns_recursor[15237]: Sending SERVFAIL to
> >> 127.0.0.1
> >> during resolve of 'gm-realm.net.' because: Too much time waiting
> >> for gm-realm.net.|A, timeouts: 5, throttles: 1, queries: 6,
> >> 7578msec
> >>
> >> Nov 10 15:33:09 ns01 pdns_recursor[15237]: Sending SERVFAIL to
> >> 127.0.0.1
> >> during resolve of 'gm-realm.net.' because: Too much time waiting
> >> for gm-realm.net.|A, timeouts: 5, throttles: 2, queries: 6,
> >> 7504msec
> >>
> >> Nov 10 15:33:12 ns01 pdns_recursor[15237]: Sending SERVFAIL to
> >> 127.0.0.1
> >> during resolve of 'gm-realm.net.' because: Too much time waiting
> >> for gm-realm.net.|A, timeouts: 5, throttles: 3, queries: 6,
> >> 7502msec
> >>
> >> Nov 10 15:33:13 ns01 pdns_recursor[15237]: Sending SERVFAIL to
> >> 127.0.0.1
> >> during resolve of 'us.micardapi.micloud.xiaomi.net.' because: Too
> >> much time waiting for us.api.micloud.mi.com.|A, timeouts: 5,
> >> throttles: 0,
> >> queries: 7,
> >> 7709msec
> >>
> >> Nov 10 15:33:18 ns01 pdns_recursor[15237]: Sending SERVFAIL to
> >> 127.0.0.1
> >> during resolve of 'www.coocent.net.' because: Too much time waiting
> >> for s-149179.abc188.com.|A, timeouts: 5, throttles: 0, queries: 8,
> >> 8093msec
> >>
> >> Nov 10 15:33:18 ns01 pdns_recursor[15237]: Sending SERVFAIL to
> >> 127.0.0.1
> >> during resolve of 'www.6ud1.com.' because: Too much time waiting
> >> for www.6ud1.com.|A, timeouts: 5, throttles: 0, queries: 6,
> >> 7502msec
> >>
> >> Nov 10 15:42:52 ns01 pdns_recursor[15237]: stats: 1787915
> >> questions,
> >> 497334
> >> cache entries, 86066 negative entries, 11% cache hits
> >>
> >> Nov 10 15:42:52 ns01 pdns_recursor[15237]: stats: throttle map:
> >> 6856, ns
> >> speeds: 29645
> >>
> >> Nov 10 15:42:52 ns01 pdns_recursor[15237]: stats: outpacket/query
> >> ratio 49%, 11% throttled, 0 no-delegation drops
> >>
> >> Nov 10 15:42:52 ns01 pdns_recursor[15237]: stats: 211 outgoing tcp
> >> connections, 1 queries running, 50712 outgoing timeouts
> >>
> >> Nov 10 15:42:52 ns01 pdns_recursor[15237]: stats: 322566 packet
> >> cache entries, 61% packet cache hits
> >>
> >> Nov 10 15:42:52 ns01 pdns_recursor[15237]: stats: 926 qps (average
> >> over 1930
> >> seconds)
> >>
> >>
> >>
> >> Config:
> >>
> >>
> >>
> >> I have one master server which replicates db to the four slave server.
> >>
> >>
> >>
> >> # cat recursor.conf
> >>
> >> ..
> >>
> >> hint-file=/etc/pdns/named.root
> >>
> >> allow-from=127.0.0.0/8
> >>
> >> local-address=127.0.0.1
> >>
> >> local-port=5353
> >>
> >> version-string=Bind Recursor
> >>
> >> ..
> >>
> >>
> >>
> >> # cat /etc/pdns/pdns.conf
> >>
> >> ..
> >>
> >> launch=gmysql
> >>
> >> gmysql-host=127.0.0.1
> >>
> >> gmysql-port=3306
> >>
> >> gmysql-user=p_owerdns
> >>
> >> gmysql-password=verysecretpassword
> >>
> >> gmysql-dbname=p_ owerdns
> >>
> >> gmysql-dnssec="yes"
> >>
> >>
> >>
> >> #allow to customers
> >>
> >> allow-recursion=127.0.0.1/8, 172.16.0.0/16, 10.0.0.0/8,
> >> xxx.xxx.xxx.xxx/16
> >>
> >>
> >>
> >> #master
> >>
> >> #allow-axfr-ips=172.16.6.30
> >>
> >>
> >>
> >> local-address=0.0.0.0
> >>
> >> local-port=53
> >>
> >>
> >>
> >> control-console=no
> >>
> >>
> >>
> >> query-cache-ttl=18600
> >>
> >> cache-ttl=18600
> >>
> >> default-ttl=7200
> >>
> >> soa-expire-default=18600
> >>
> >> soa-minimum-ttl=3600
> >>
> >> soa-refresh-default=10800
> >>
> >> soa-retry-default=3600
> >>
> >>
> >>
> >> daemon=yes
> >>
> >>
> >>
> >> default-soa-name=ns.master.mydomain.net
> >>
> >>
> >>
> >> distributor-threads=18
> >>
> >>
> >>
> >> guardian=yes
> >>
> >>
> >>
> >> #lazy-recursion=yes
> >>
> >>
> >>
> >> master=no
> >>
> >> slave=yes
> >>
> >> slave-cycle-interval=600
> >>
> >>
> >>
> >> max-tcp-connections=100
> >>
> >> max-queue-length=50000
> >>
> >>
> >>
> >> recursor=127.0.0.1:5353
> >>
> >>
> >>
> >> out-of-zone-additional-processing=yes
> >>
> >>
> >>
> >> webserver=yes
> >>
> >> webserver-address=172.16.6.34
> >>
> >> webserver-password=adminadminadmin
> >>
> >> webserver-port=8081
> >>
> >> webserver-print-arguments=yes
> >>
> >>
> >>
> >> #loglevel=9
> >>
> >> #log-dns-details=yes
> >>
> >> #log-dns-queries=yes
> >>
> >> #query-logging=yes
> >>
> >>
> >>
> >> version-string=Bind Resolver
> >>
> >> ..
> >>
> >> _______________________________________________
> >> Pdns-users mailing list
> >> Pdns-users at mailman.powerdns.com
> >> http://mailman.powerdns.com/mailman/listinfo/pdns-users
> >
> > _______________________________________________
> > Pdns-users mailing list
> > Pdns-users at mailman.powerdns.com
> > http://mailman.powerdns.com/mailman/listinfo/pdns-users
> >
> >
> > _______________________________________________
> > Pdns-users mailing list
> > Pdns-users at mailman.powerdns.com
> > http://mailman.powerdns.com/mailman/listinfo/pdns-users
>
>
>
>
> _______________________________________________
> Pdns-users mailing list
> Pdns-users at mailman.powerdns.com
> http://mailman.powerdns.com/mailman/listinfo/pdns-users
>
>
> _______________________________________________
> Pdns-users mailing list
> Pdns-users at mailman.powerdns.com
> http://mailman.powerdns.com/mailman/listinfo/pdns-users
More information about the Pdns-users
mailing list