[Pdns-users] Recursor: behaviour of packet cache

Thu Mar 16 12:00:29 UTC 2017

I've just observed something strange with pdns-recursor.

Having added a name to the authoritative server, some clients querying 
the *same* pdns recursor consistently saw the new name, but others 
consistently saw NXDOMAIN!  It was as if there were different views for 
different clients.  (Sample dig exchanges at end of mail).

The only explanation I can think of is the packet cache: 
https://doc.powerdns.com/md/recursor/performance/#recursor-caches

I am guessing that clients which had queried for this name *before* it 
was added to the authoritative servers continued to see NXDOMAIN, and 
clients which first queried it afterwards got the right response.

However, this raises some questions.

Question 1: does this mean that the packet cache includes the source IP 
address when deciding whether this is the "same" query or not? Or was it 
other subtle differences between the query content, e.g. flags, which 
made them be treated differently? (Different clients have different 
versions of "dig")

Question 2: how long before the packet cache entries expire, especially 
with regards to negative caching?  Is it controlled by the SOA record?

;; AUTHORITY SECTION:
int.xxxxxxxxxx.net.    3600    IN    SOA wrn-dc1.ad.xxxxxxxxxx.net. 
hostmaster.ad.xxxxxxxxxx.net. 292 900 600 86400 3600

That is, does the packet cache honour the negative cache of 3600 seconds 
here?

I'm afraid I ended up restarting the pdns-recursor to clear the issue as 
I couldn't wait an hour to see if it cleared by itself, but I captured 
some traffic first.

Regards,

Brian.

Client 1 sees NXDOMAIN:

$ dig @192.168.5.53 storage12.int.xxxxxxxxxx.net. a

; <<>> DiG 9.8.3-P1 <<>> @192.168.5.53 storage12.int.xxxxxxxxxx.net. a
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 58696
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;storage12.int.xxxxxxxxxx.net.  IN      A

;; Query time: 81 msec
;; SERVER: 192.168.5.53#53(192.168.5.53)
;; WHEN: Thu Mar 16 11:23:56 2017
;; MSG SIZE  rcvd: 46

Client 2 sees the correct answer:

# dig @192.168.5.53 storage12.int.xxxxxxxxxx.net. a

; <<>> DiG 9.10.3-P4-Ubuntu <<>> @192.168.5.53 
storage12.int.xxxxxxxxxx.net. a
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3795
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;storage12.int.xxxxxxxxxx.net.    IN    A

;; ANSWER SECTION:
storage12.int.xxxxxxxxxx.net. 186 IN    A    192.168.5.81

;; Query time: 0 msec
;; SERVER: 192.168.5.53#53(192.168.5.53)
;; WHEN: Thu Mar 16 11:23:31 GMT 2017
;; MSG SIZE  rcvd: 73

tcpdump at the pdns recursor:

11:23:25.950250 IP 10.26.1.246.60226 > 192.168.5.53.53: 58696+ A? 
storage12.int.xxxxxxxxxx.net. (46)
     0x0000:  4500 004a 7a8f 0000 3f11 2f27 0a1a 01f6 E..Jz...?./'....
     0x0010:  c0a8 0535 eb42 0035 0036 XXXX e548 0100 ...5.B.5.6xx.H..
     0x0020:  0001 0000 0000 0000 0973 746f 7261 6765 .........storage
     0x0030:  3132 0369 6e74 0aXX XXXX XXXX XXXX XXXX 12.int.xxxxxxxxx
     0x0040:  XX03 6e65 7400 0001 0001 x.net.....
11:23:25.950362 IP 192.168.5.53.53 > 10.26.1.246.60226: 58696 NXDomain 
0/0/0 (46)
     0x0000:  4500 004a 5735 4000 4011 1181 c0a8 0535 E..JW5 at .@......5
     0x0010:  0a1a 01f6 0035 eb42 0036 XXXX e548 8183 .....5.B.6xx.H..
     0x0020:  0001 0000 0000 0000 0973 746f 7261 6765 .........storage
     0x0030:  3132 0369 6e74 0aXX XXXX XXXX XXXX XXXX 12.int.xxxxxxxxx
     0x0040:  XX03 6e65 7400 0001 0001 x.net.....
11:23:31.188823 IP 192.168.5.54.34744 > 192.168.5.53.53: 3795+ [1au] A? 
storage12.int.xxxxxxxxxx.net. (57)
     0x0000:  4500 0055 a141 0000 4011 4d9b c0a8 0536 E..U.A.. at .M....6
     0x0010:  c0a8 0535 87b8 0035 0041 XXXX 0ed3 0120 ...5...5.Axx....
     0x0020:  0001 0000 0000 0001 0973 746f 7261 6765 .........storage
     0x0030:  3132 0369 6e74 0aXX XXXX XXXX XXXX XXXX 12.int.xxxxxxxxx
     0x0040:  XX03 6e65 7400 0001 0001 0000 2910 0000 x.net.......)...
     0x0050:  0000 0000 00 .....
11:23:31.188939 IP 192.168.5.53.53 > 192.168.5.54.34744: 3795 1/0/1 A 
192.168.5.81 (73)
     0x0000:  4500 0065 3933 4000 4011 7599 c0a8 0535 E..e93 at .@.u....5
     0x0010:  c0a8 0536 0035 87b8 0051 XXXX 0ed3 8180 ...6.5...Qxx....
     0x0020:  0001 0001 0000 0001 0973 746f 7261 6765 .........storage
     0x0030:  3132 0369 6e74 0aXX XXXX XXXX XXXX XXXX 12.int.xxxxxxxxx
     0x0040:  XX03 6e65 7400 0001 0001 c00c 0001 0001 x.net...........
     0x0050:  0000 00ba 0004 c0a8 0551 0000 2910 0000 .........Q..)...
     0x0060:  0000 0000 00 .....

(Apologies for minor obfuscation, but only the domain and UDP checksum 
have been masked - everything else including IP addresses and ports is 
unchanged)