[dnsdist] dnsdist Drops, revisited

[Fredrik Pettai]

Hi list,

I’m curious on the “high" amount of Drops I see on one dnsdist 1.4.0 (debian derived packages) frontend compared to other(s)
And I’m guessing the main reason is workload, which is different (services/servers use this resolver that Drops more).

I don’t find the “high” Drops numbers satisfying, but perhaps these numbers are about normal average? 
Anyway, I'd would like to improve those numbers if possible. Here are some stats from two dnsdist frontends:

> showServers()
#   Name                 Address                       State     Qps    Qlim Ord Wt    Queries   Drops Drate   Lat Outstanding Pools
0   worker1              127.0.0.1:53                     up    73.7       0   1  1     565950     278   0.0   0.5           0
1   worker2              [::1]:53                         up    55.7       0   1  1     584273     294   0.0   1.1           0

While one of our bigger servers doesn’t perform as well (in terms of Drops ratio):

> showServers()
#   Name                 Address                       State     Qps    Qlim Ord Wt    Queries   Drops Drate   Lat Outstanding Pools
0   worker1              127.0.0.1:53                     up    43.8       0   1  1    1054047   12728   0.0  31.1           4
1   worker2              127.0.0.1:53                     up    43.8       0   1  1    1064823   12823   0.0  17.5           4
2   worker3              [::1]:53                         up    20.9       0   1  1    1054548   12773   0.0  38.5           2
3   worker4              [::1]:53                         up    35.8       0   1  1    1081502   12854   0.0  48.9           3

FW & DNSdist rules are almost none, and the same configuration on both the above systems 
(actually more active rules and even Lua-code on the “fast” dnsdist-system)

I just found one earlier thread on the topic, and it didn’t describe a way to improve the situation,
just how to possibly look to see what the underlying issues might be...

http://powerdns.13854.n7.nabble.com/dnsdist-drops-packet-td11974.html
(https://mailman.powerdns.com/pipermail/dnsdist/2016-January/000052.html)

dumpStats from the above server

> dumpStats()
acl-drops              	          0    latency0-1             	    3620405
cache-hits             	          0    latency1-10            	      59808
cache-misses           	          0    latency10-50           	     132513
cpu-sys-msec           	     749565    latency100-1000        	     386909
cpu-user-msec          	     470696    latency50-100          	     101861
downstream-send-errors 	          0    no-policy              	          0
downstream-timeouts    	      52571    noncompliant-queries   	          0
dyn-block-nmg-size     	          0    noncompliant-responses 	          0
dyn-blocked            	          0    queries                	    4382032
empty-queries          	          0    rdqueries              	    4382007
fd-usage               	         42    real-memory-usage      	  315129856
frontend-noerror       	    3254422    responses              	    4329454
frontend-nxdomain      	     902996    rule-drop              	          0
frontend-servfail      	     172012    rule-nxdomain          	          0
latency-avg100         	      41936.3  rule-refused           	          0
latency-avg1000        	      44165.7  rule-servfail          	          0
latency-avg10000       	      43366.6  security-status        	          0
latency-avg1000000     	      41994.4  self-answered          	          1
latency-count          	    4329455    servfail-responses     	     172012
latency-slow           	      27681    special-memory-usage   	   95940608
latency-sum            	  172860695    trunc-failures         	          0

> topSlow(10, 1000)
   1  uyrg.com.                                  69 46.9%
   2  115.61.96.156.in-addr.arpa.                19 12.9%
   3  nhu.edu.tw.                                 9  6.1%
   4  nbkailan.com.                               8  5.4%
   5  aikesi.com.                                 8  5.4%
   6  168.122.238.45.in-addr.arpa.                6  4.1%
   7  45-179-252-62-dynamic.proxyar.com.          4  2.7%
   8  callforarticle.com.                         3  2.0%
   9  default._domainkey.nhu.edu.tw.              3  2.0%
  10  205.78.127.180.in-addr.arpa.                3  2.0%
  11  Rest                                       15 10.2%

(Many are probably spammy relay IPs, sending domains, etc) 

Is there a way to optimise the dnsdist configuration, for instance making a slow path?
either for the slow queries, or possibly the clients that ask those queries?

(Also, It’s unbound in the backend of all dnsdist frontend, and it’s caching heavily, also expired answers).

Re,
/P