[Pdns-users] Reply from unexpected source: ip#267, expected ip#53

[Nick Williams]

I have three identically-configured Power DNS 2.9.21 servers. Server 1 and 2 are on Centos5 "Linux version 2.6.18-028stab064.7 (root at rhel5-64-build) (gcc version 4.1.2 20070626 (Red Hat 4.1.2-14)) #1 SMP Wed Aug 26 13:11:07 MSD 2009" and "Linux version 2.6.18-028stab070.5 (root at rhel5-build-x64) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)) #1 SMP Fri Sep 17 15:37:23 MSD 2010" and server 3 is on SuSE "Linux version 2.6.22.18-0.2-default (geeko at buildhost) (gcc version 4.2.1 (SUSE Linux)) #1 SMP 2008-06-09 13:53:20 +0200". Up until two weeks ago, everything was working the same. Then, suddenly, Server 3 responses changed without my making any configuration changes.

Server 1 Response
$ host ip2.dns2.my-domain-1.com dns1.my-domain-1.com
Using domain server:
Name: dns1.my-domain-1.com
Address: [my-ip-address-1]#53
Aliases: 

ip2.dns2.my-domain-1.com has address [my-ip-address-4]

Server 1 Log
Oct 02 10:30:37 About to create 3 backend threads for UDP
Oct 02 10:30:37 gmysql Connection succesful
Oct 02 10:30:37 gmysql Connection succesful
Oct 02 10:30:37 gmysql Connection succesful
Oct 02 10:30:37 Done launching threads, ready to distribute questions
Oct 02 10:30:44 Distributor misses a thread (4<3), spawning new one
Oct 02 10:30:44 gmysql Connection succesful
Oct 02 10:30:44 Authoritative empty NO ERROR to [my-ip-address-home] for 'ip2.dns2.my-domain-1.com' (AAAA), other types do exist
Oct 02 10:30:44 Authoritative empty NO ERROR to [my-ip-address-home] for 'ip2.dns2.my-domain-1.com' (MX), other types do exist
Oct 02 10:31:23 Authoritative empty NO ERROR to [my-ip-address-home] for 'ip2.dns2.my-domain-1.com' (AAAA), other types do exist
Oct 02 10:31:23 Authoritative empty NO ERROR to [my-ip-address-home] for 'ip2.dns2.my-domain-1.com' (MX), other types do exist

Server 2 Response
$ host ip2.dns2.my-domain-1.com dns2.my-domain-1.com
Using domain server:
Name: dns2.my-domain-1.com
Address: [my-ip-address-1]#53
Aliases: 

ip2.dns2.my-domain-1.com has address [my-ip-address-4]

Server 2 Log
Oct 02 10:33:24 About to create 3 backend threads for UDP
Oct 02 10:33:24 gmysql Connection succesful
Oct 02 10:33:24 gmysql Connection succesful
Oct 02 10:33:24 gmysql Connection succesful
Oct 02 10:33:24 Done launching threads, ready to distribute questions
Oct 02 10:33:30 Distributor misses a thread (4<3), spawning new one
Oct 02 10:33:30 gmysql Connection succesful
Oct 02 10:33:31 Authoritative empty NO ERROR to [my-ip-address-home] for 'ip2.dns2.my-domain-1.com' (AAAA), other types do exist
Oct 02 10:33:31 Authoritative empty NO ERROR to [my-ip-address-home] for 'ip2.dns2.my-domain-1.com' (MX), other types do exist
Oct 02 10:34:12 Authoritative empty NO ERROR to [my-ip-address-home] for 'ip2.dns2.my-domain-1.com' (AAAA), other types do exist
Oct 02 10:34:12 Authoritative empty NO ERROR to [my-ip-address-home] for 'ip2.dns2.my-domain-1.com' (MX), other types do exist

So far, everything looks right. But here's server 3...

Server 3 Response
$ host ip2.dns2.my-domain-1.com dns3.my-domain-1.com
;; reply from unexpected source: [my-ip-address-3]#267, expected [my-ip-address-3]#53
;; reply from unexpected source: [my-ip-address-3]#267, expected [my-ip-address-3]#53
;; connection timed out; no servers could be reached
$ host my-domain-2.com dns3.my-domain-1.com
;; reply from unexpected source: [my-ip-address-3]#267, expected [my-ip-address-3]#53
;; reply from unexpected source: [my-ip-address-3]#267, expected [my-ip-address-3]#53
;; connection timed out; no servers could be reached

Server 3 Log
Oct 02 10:24:00 About to create 3 backend threads for UDP
Oct 02 10:24:00 gmysql Connection succesful
Oct 02 10:24:00 gmysql Connection succesful
Oct 02 10:24:00 gmysql Connection succesful
Oct 02 10:24:00 Done launching threads, ready to distribute questions
Oct 02 10:24:09 Distributor misses a thread (4<3), spawning new one
Oct 02 10:24:09 gmysql Connection succesful
Oct 02 10:35:29 Not authoritative for 'my-domain-2.com', sending servfail to [my-ip-address-home] (recursion was desired)

I've examined my ports in use with nmap and lsof and I've examined my firewall rules. Nothing has changed and everything looks correct. pdns_serv is listening on port 53. I can telnet in to port 53 from the same machine as the queries above were executed from. I get the same results with both host and dig.

Anybody have any ideas?

Thanks,

Nick
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.powerdns.com/pipermail/pdns-users/attachments/20101002/5355d184/attachment.html>