[Pdns-users] Problems with PowerDNS
leen at consolejunkie.net
leen at consolejunkie.net
Wed Nov 11 09:26:15 UTC 2015
On 2015-11-11 10:07, Nadir M. Aliyev wrote:
> Dear Aki,
>
Hi Nadir,
Tip: What most people running PowerDNS do is use 2 seperate IPs voor
PowerDNS recursor and PowerDNS Authoritative Server.
So for domains the server is authoritative for it will receive them on
the Authoritative Server and the recursive queries it can receive them
on the recursor.
This means queries voor other domains don't need to go to the database.
Because PowerDNS Authoritative server would need to check the database
first to see if it's authoritative for a domain before it can ask the
PowerDNS recursor.
If you have 40 Mbps as you mentioned maybe most of those queries are
for other domains not stored in the database.
Could it be the cache is to small to keep all those queries/domains in
cache ?
(note: I'm not a PowerDNS developer, they can answer such things)
Hope that helps,
Leen.
> Yes I have indexes.
>
> But pdns sends everytime bulk query for every dns query even not
> exists in db. Sometimes cache hits sometimes miss.
>
> query-cache-ttl=18600
> cache-ttl=18600
> default-ttl=7200
> soa-expire-default=18600
> soa-minimum-ttl=3600
> soa-refresh-default=10800
> soa-retry-default=3600
> max-cache-entries=10000000
>
> For ex. Nslookup google.com
>
> Nov 11 13:01:51 ns01 pdns[7180]: Remote 127.0.0.1 wants
> 'google.com|A', do = 0, bufsize = 512: packetcache MISS
> Nov 11 13:01:51 ns01 pdns[7180]: Query: SELECT
> content,ttl,prio,type,domain_id,disabled,name,auth FROM records WHERE
> disabled=0 and type='SOA' and name='google.com'
> Nov 11 13:01:51 ns01 pdns[7180]: Query: SELECT
> content,ttl,prio,type,domain_id,disabled,name,auth FROM records WHERE
> disabled=0 and type='SOA' and name='com'
> Nov 11 13:01:51 ns01 pdns[7180]: Query: SELECT
> content,ttl,prio,type,domain_id,disabled,name,auth FROM records WHERE
> disabled=0 and type='SOA' and name=''
> Remote 127.0.0.1 wants 'google.com|A', do = 0, bufsize = 512:
> packetcache MISS
> Nov 11 13:03:05 ns01 pdns[7180]: Query: SELECT
> content,ttl,prio,type,domain_id,disabled,name,auth FROM records WHERE
> disabled=0 and type='SOA' and name='google.com'
> Nov 11 13:03:05 ns01 pdns[7180]: Query: SELECT
> content,ttl,prio,type,domain_id,disabled,name,auth FROM records WHERE
> disabled=0 and type='SOA' and name='com'
> Nov 11 13:03:05 ns01 pdns[7180]: Query: SELECT
> content,ttl,prio,type,domain_id,disabled,name,auth FROM records WHERE
> disabled=0 and type='SOA' and name=''
> Nov 11 13:03:10 ns01 pdns[7180]: Remote 127.0.0.1 wants
> 'google.com|A', do = 0, bufsize = 512: packetcache HIT
>
>
> I think that community need to do some works on pdns.
> For example caching db zones from db, it will decrase server load.
> And flush it with rec_control anytime like rec_control purge dbcache
> :)
>
> Strange that, when I try nslookup to google.com after service
> restarts it works. After some time I got serv refused. Cache life too
> high. But why domain google.com does not hit in cache?
>
> It happens with many domains but not all. Google.com example for
> this.
>
> Named.root file I downloaded from IANA website. And up to date.
>
> # rec_control get cache-entries packetcache-entries
> 457723
> 284453
>
> # rec_control get-all
> all-outqueries 672092
> answers-slow 17055
> answers0-1 73094
> answers1-10 3696
> answers10-100 221413
> answers100-1000 174393
> cache-entries 458482
> cache-hits 57934
> cache-misses 431721
> case-mismatches 0
> chain-resends 32382
> client-parse-errors 0
> concurrent-queries 0
> dlg-only-drops 0
> dont-outqueries 132
> edns-ping-matches 0
> edns-ping-mismatches 0
> failed-host-entries 4346
> ipv6-outqueries 0
> ipv6-questions 0
> malloc-bytes 0
> max-mthread-stack 39560
> negcache-entries 91062
> no-packet-error 1026559
> noedns-outqueries 672122
> noerror-answers 927841
> noping-outqueries 0
> nsset-invalidations 2573
> nsspeeds-entries 41285
> nxdomain-answers 125690
> outgoing-timeouts 23292
> over-capacity-drops 0
> packetcache-entries 284954
> packetcache-hits 599140
> packetcache-misses 489559
> policy-drops 0
> qa-latency 75260
> questions 1088780
> resource-limits 0
> security-status 0
> server-parse-errors 0
> servfail-answers 35171
> spoof-prevents 0
> sys-msec 74879
> tcp-client-overflow 0
> tcp-clients 0
> tcp-outqueries 177
> tcp-questions 96
> throttle-entries 4454
> throttled-out 38905
> throttled-outqueries 38905
> too-old-drops 0
> unauthorized-tcp 0
> unauthorized-udp 0
> unexpected-packets 0
> unreachables 1505
> uptime 1277
> user-msec 145122
>
> -----Original Message-----
> From: Aki Tuomi [mailto:cmouse at youzen.ext.b2.fi]
> Sent: 11 noyabr 2015, çərşənbə 12:22
> To: Nadir M. Aliyev <admin at bakinter.net>
> Cc: 'Patrick Domack' <patrickdk at patrickdk.com>;
> pdns-users at mailman.powerdns.com
> Subject: Re: [Pdns-users] Problems with PowerDNS
>
> Does your database have indexes? We have had few cases before where
> the user had forgotten to add indexes to the database.
>
> Aki
>
> On Wed, Nov 11, 2015 at 12:10:17PM +0400, Nadir M. Aliyev wrote:
>> Dear Patrick,
>>
>> I tried to set
>>
>> gmysql-dnssec="no"
>> distributor-threads=10
>> receiver-threads=5
>>
>> Now:
>> Mysql 110%
>> Pdns_server 90 %
>> Pdns_recursor 25%
>>
>>
>> But after 10-15 minutes again I got from some domains SERVFAIL..
>>
>> [root at ns01 ~]# nslookup google.com
>> Server: 127.0.0.1
>> Address: 127.0.0.1#53
>>
>> ** server can't find google.com: REFUSED
>>
>> And logs:
>> Nov 11 12:08:59 ns01 pdns_recursor[4559]: Sending SERVFAIL to
>> 127.0.0.1 during resolve of 'ad.bb800.com.' because: Too much time
>> waiting for ad.6gg.cn.|A, timeouts: 5, throttles: 0, queries: 6,
>> 7506msec Nov 11 12:09:04 ns01 pdns_recursor[4559]: Sending SERVFAIL
>> to
>> 127.0.0.1 during resolve of 'ad.bb800.com.' because: Too much time
>> waiting for ad.6gg.cn.|A, timeouts: 5, throttles: 5, queries: 6,
>> 7503msec Nov 11 12:09:09 ns01 pdns_recursor[4559]: Sending SERVFAIL
>> to
>> 127.0.0.1 during resolve of 'wx.qq.com.' because: Too much time
>> waiting for wx1.qq.com.|A, timeouts: 5, throttles: 0, queries: 8,
>> 8219msec Nov 11 12:09:34 ns01 pdns_recursor[4559]: Sending SERVFAIL
>> to
>> 127.0.0.1 during resolve of 'dev.voicecloud.cn.' because: Too much
>> time waiting for dev.voicecloud.cn.|A, timeouts: 4, throttles: 0,
>> queries: 9, 7087msec Nov 11 12:09:38 ns01 pdns_recursor[4559]:
>> Sending
>> SERVFAIL to 127.0.0.1 during resolve of
>> '79.208.218.41.in-addr.arpa.'
>> because: Too much time waiting for 79.208.218.41.in-addr.arpa.|PTR,
>> timeouts: 4, throttles: 0, queries: 13, 7007msec Nov 11 12:09:43
>> ns01
>> pdns_recursor[4559]: Sending SERVFAIL to 127.0.0.1 during resolve of
>> '61.29.19.113.in-addr.arpa.' because: Too much time waiting for
>> 61.29.19.113.in-addr.arpa.|PTR, timeouts: 4, throttles: 0, queries:
>> 11, 7928msec Nov 11 12:09:49 ns01 pdns_recursor[4559]: Sending
>> SERVFAIL to 127.0.0.1 during resolve of '50.25.36.204.in-addr.arpa.'
>> because: Too much time waiting for 50.25.36.204.in-addr.arpa.|PTR,
>> timeouts: 5, throttles: 0, queries: 7, 7587msec
>>
>>
>> -----Original Message-----
>> From: pdns-users-bounces at mailman.powerdns.com
>> [mailto:pdns-users-bounces at mailman.powerdns.com] On Behalf Of
>> Patrick
>> Domack
>> Sent: 11 noyabr 2015, çərşənbə 01:08
>> To: pdns-users at mailman.powerdns.com
>> Subject: Re: [Pdns-users] Problems with PowerDNS
>>
>> I suppose sense you have dnssec=yes, you are using dnssec, This will
>> cause a lot of sql queries.
>>
>> pdns is using 100% cpu of a single core, did you try adjusting
>> receiver-threads >1 probably for that box set it to 4 and test, maybe
>> higher even.
>>
>> Since I don't know much about what your pdns server is doing (and I
>> haven't had issues on mine), I assume the dnssec dynamic signing is
>> eating your cpu, and it only has one worker thread to do it with,
>> limiting it to a single core.
>>
>> I could be completely wrong.
>>
>>
>> Quoting "Nadir M. Aliyev" <admin at bakinter.net>:
>>
>> > Dear Peter van Dijk, my connection link is 1000Gbps, server
>> hardware
>> > from cisco ucs. There is no problem with hardware. But mysql uses
>> > huge resources even not zone in db it sends 4-5 queries to the db.
>> >
>> > I used percone tools to optimize mysql configuration. But it
>> > decreased cpu usage only 10%. I have 10.000 query per second.
>> >
>> > Maybe I need do some tuning on TTLs?
>> >
>> > -----Original Message-----
>> > From: pdns-users-bounces at mailman.powerdns.com
>> > [mailto:pdns-users-bounces at mailman.powerdns.com] On Behalf Of
>> Peter
>> > van Dijk
>> > Sent: 10 noyabr 2015, çərşənbə axşamı 16:58
>> > To: pdns-users at mailman.powerdns.com
>> > Subject: Re: [Pdns-users] Problems with PowerDNS
>> >
>> > Hello Nadir,
>> >
>> > based on the logs, it looks like your powerdns has trouble
>> reaching
>> > the Internet at all. Are you on a slow or congested link? Note
>> that
>> > in general your machine looks quite busy!
>> >
>> > Kind regards,
>> > --
>> > Peter van Dijk
>> > PowerDNS.COM BV - https://www.powerdns.com/
>> >
>> > On 10 Nov 2015, at 13:01, Nadir M. Aliyev wrote:
>> >
>> >> Hi everyone!
>> >>
>> >>
>> >>
>> >> I have problems with some domains
>> >>
>> >>
>> >>
>> >> For ex. When I do google.com sometimes I get ns records but
>> >> sometimes I get SERFVAIL also it happens basically with google.
>> >> When I restrart pdns it works normally for 5 minutes. Then again
>> SERVFAIL.
>> >>
>> >>
>> >>
>> >> Strange, some domains works some not works.. Even if cache hits.
>> >>
>> >> I increased cache ttls not helped.
>> >>
>> >>
>> >>
>> >> Server details: 8 core cpu, 8 GB of Ram.
>> >>
>> >> Load: pdns 100%, mysql 120%, pdns-recursor 30%, network 40 mbps.
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> Some logs:
>> >>
>> >> Nov 10 15:33:08 ns01 pdns_recursor[15237]: Sending SERVFAIL to
>> >> 127.0.0.1
>> >> during resolve of 'gm-realm.net.' because: Too much time waiting
>> >> for gm-realm.net.|A, timeouts: 5, throttles: 1, queries: 6,
>> >> 7578msec
>> >>
>> >> Nov 10 15:33:09 ns01 pdns_recursor[15237]: Sending SERVFAIL to
>> >> 127.0.0.1
>> >> during resolve of 'gm-realm.net.' because: Too much time waiting
>> >> for gm-realm.net.|A, timeouts: 5, throttles: 2, queries: 6,
>> >> 7504msec
>> >>
>> >> Nov 10 15:33:12 ns01 pdns_recursor[15237]: Sending SERVFAIL to
>> >> 127.0.0.1
>> >> during resolve of 'gm-realm.net.' because: Too much time waiting
>> >> for gm-realm.net.|A, timeouts: 5, throttles: 3, queries: 6,
>> >> 7502msec
>> >>
>> >> Nov 10 15:33:13 ns01 pdns_recursor[15237]: Sending SERVFAIL to
>> >> 127.0.0.1
>> >> during resolve of 'us.micardapi.micloud.xiaomi.net.' because: Too
>> >> much time waiting for us.api.micloud.mi.com.|A, timeouts: 5,
>> >> throttles: 0,
>> >> queries: 7,
>> >> 7709msec
>> >>
>> >> Nov 10 15:33:18 ns01 pdns_recursor[15237]: Sending SERVFAIL to
>> >> 127.0.0.1
>> >> during resolve of 'www.coocent.net.' because: Too much time
>> waiting
>> >> for s-149179.abc188.com.|A, timeouts: 5, throttles: 0, queries:
>> 8,
>> >> 8093msec
>> >>
>> >> Nov 10 15:33:18 ns01 pdns_recursor[15237]: Sending SERVFAIL to
>> >> 127.0.0.1
>> >> during resolve of 'www.6ud1.com.' because: Too much time waiting
>> >> for www.6ud1.com.|A, timeouts: 5, throttles: 0, queries: 6,
>> >> 7502msec
>> >>
>> >> Nov 10 15:42:52 ns01 pdns_recursor[15237]: stats: 1787915
>> >> questions,
>> >> 497334
>> >> cache entries, 86066 negative entries, 11% cache hits
>> >>
>> >> Nov 10 15:42:52 ns01 pdns_recursor[15237]: stats: throttle map:
>> >> 6856, ns
>> >> speeds: 29645
>> >>
>> >> Nov 10 15:42:52 ns01 pdns_recursor[15237]: stats: outpacket/query
>> >> ratio 49%, 11% throttled, 0 no-delegation drops
>> >>
>> >> Nov 10 15:42:52 ns01 pdns_recursor[15237]: stats: 211 outgoing
>> tcp
>> >> connections, 1 queries running, 50712 outgoing timeouts
>> >>
>> >> Nov 10 15:42:52 ns01 pdns_recursor[15237]: stats: 322566 packet
>> >> cache entries, 61% packet cache hits
>> >>
>> >> Nov 10 15:42:52 ns01 pdns_recursor[15237]: stats: 926 qps
>> (average
>> >> over 1930
>> >> seconds)
>> >>
>> >>
>> >>
>> >> Config:
>> >>
>> >>
>> >>
>> >> I have one master server which replicates db to the four slave
>> server.
>> >>
>> >>
>> >>
>> >> # cat recursor.conf
>> >>
>> >> ..
>> >>
>> >> hint-file=/etc/pdns/named.root
>> >>
>> >> allow-from=127.0.0.0/8
>> >>
>> >> local-address=127.0.0.1
>> >>
>> >> local-port=5353
>> >>
>> >> version-string=Bind Recursor
>> >>
>> >> ..
>> >>
>> >>
>> >>
>> >> # cat /etc/pdns/pdns.conf
>> >>
>> >> ..
>> >>
>> >> launch=gmysql
>> >>
>> >> gmysql-host=127.0.0.1
>> >>
>> >> gmysql-port=3306
>> >>
>> >> gmysql-user=p_owerdns
>> >>
>> >> gmysql-password=verysecretpassword
>> >>
>> >> gmysql-dbname=p_ owerdns
>> >>
>> >> gmysql-dnssec="yes"
>> >>
>> >>
>> >>
>> >> #allow to customers
>> >>
>> >> allow-recursion=127.0.0.1/8, 172.16.0.0/16, 10.0.0.0/8,
>> >> xxx.xxx.xxx.xxx/16
>> >>
>> >>
>> >>
>> >> #master
>> >>
>> >> #allow-axfr-ips=172.16.6.30
>> >>
>> >>
>> >>
>> >> local-address=0.0.0.0
>> >>
>> >> local-port=53
>> >>
>> >>
>> >>
>> >> control-console=no
>> >>
>> >>
>> >>
>> >> query-cache-ttl=18600
>> >>
>> >> cache-ttl=18600
>> >>
>> >> default-ttl=7200
>> >>
>> >> soa-expire-default=18600
>> >>
>> >> soa-minimum-ttl=3600
>> >>
>> >> soa-refresh-default=10800
>> >>
>> >> soa-retry-default=3600
>> >>
>> >>
>> >>
>> >> daemon=yes
>> >>
>> >>
>> >>
>> >> default-soa-name=ns.master.mydomain.net
>> >>
>> >>
>> >>
>> >> distributor-threads=18
>> >>
>> >>
>> >>
>> >> guardian=yes
>> >>
>> >>
>> >>
>> >> #lazy-recursion=yes
>> >>
>> >>
>> >>
>> >> master=no
>> >>
>> >> slave=yes
>> >>
>> >> slave-cycle-interval=600
>> >>
>> >>
>> >>
>> >> max-tcp-connections=100
>> >>
>> >> max-queue-length=50000
>> >>
>> >>
>> >>
>> >> recursor=127.0.0.1:5353
>> >>
>> >>
>> >>
>> >> out-of-zone-additional-processing=yes
>> >>
>> >>
>> >>
>> >> webserver=yes
>> >>
>> >> webserver-address=172.16.6.34
>> >>
>> >> webserver-password=adminadminadmin
>> >>
>> >> webserver-port=8081
>> >>
>> >> webserver-print-arguments=yes
>> >>
>> >>
>> >>
>> >> #loglevel=9
>> >>
>> >> #log-dns-details=yes
>> >>
>> >> #log-dns-queries=yes
>> >>
>> >> #query-logging=yes
>> >>
>> >>
>> >>
>> >> version-string=Bind Resolver
More information about the Pdns-users
mailing list