[Pdns-users] SNAT and notify messages

[ch-and-pdns-users at ch.pkts.ca]

Hi PDNS users, I need your advice:  (Note: there's a TL;DR: section at
the bottom) 

We've set up three nameservers (ns1, ns2 and ns3) with two ip addresses
each: an internal 10.x.x.x and an external public ip. 

Each server runs dnsdist, pdns_recursor, and two copies of pdns_server,
like this: 

    dnsdist -> pdns_recursor -> pdns_server [internal]
            \
              pdns_server [external] 

 	* Dnsdist is listening on 127.0.0.1, 10.x.x.x, and the external public
ip, all on port 53.
 	* pdns_recursor is listening on 127.0.0.2
 	* pdns_server (internal) is listening on 127.0.0.3
 	* pdns_server (external) is listening on 127.0.0.4
 	* systemd_resolved is listening on 127.0.0.53 to satisfy local dns
requests (Ubuntu 20.04 default configuration, with 127.0.0.2 as its
resolver)

Requests come in to dnsdist, and based on their source address they're
forwarded to 127.0.0.2 (internal) or 127.0.0.4 (external). 
Pdns_recursor forwards requests to 127.0.0.3 (pdns_server internal) for
zones listed in the forward zones file and recursively answers the rest.


So far, so good; requests come in, replies go out, and external clients
can't abuse our recursor or see our internal dns entries, yayyy! 

===== 

Here's where things start go off the rails: 

Our internal network grew "organically", and for reasons lost in the
mists of time, we've got NAT gateways in our network, one of which is
between ns3 and the other two name servers. 

 	* When a zone is updated on ns1, it sends a notify to ns3, but the
source ip is changed by the NAT gateway. 
 	* The notify-acknowledgement is sent back through the NAT gateway and
ns1 gets it, since the gateway expected the response.
 	* Ns3 then sends an SOA request to the NAT gateway (as that's where
the notify came from), but it's lost because it's not related to the
previous conversation.
 	* Pdns_recursor seems to eat Notify and AXFR messages, so we've told
Dnsdist to direct those to 127.0.0.3

We've temporarily worked around the problem by adding an iptables rule
to ns3 that says 'redirect packets sent to "port 53 on the gateway" to
ns1', and it works. 

====== 

Things are now going further off the rails here: 

 	* Now we've added a bunch of independent sub-zones on small name
servers in different parts of our company, and they're going to be using
ns1, ns2 and ns3 as their secondary servers, whichever is closest. 
 	* Notifies are going to be flying all over, and packets from some of
them will be going through the same NAT gateway that ns1 uses to get to
ns3.
 	* Because of that, the iptables rule is going to mess things up, as it
assumes outgoing DNS requests to the NAT gateway should really go to
ns1's internal address.

Is there a way to get the SOA request from ns3 to go to the right place
using pdns or dnsdist? 

====== 

Now we're so far off the rails, we're in the middle of a cornfield: 

 	* The internal and external pdns servers have different zone files
(internal/external ip addresses, some hosts not listed in the external
zones).
 	* We're using the internal ips on ns1,2,3 for transferring internal
zones, and the external ips for external ones.
 	* We're attempting to use NetmaskGroupRule [1] with src=false to have
dnsdist direct requests internally/externally based on that.
 	* The independent sub-zone name servers will be notifying the ns1,2,3
on both internal and external ips, and the NAT is going to mess up the
source ips, iptables can't handle this.

Can we use DoH/DoT to establish a TCP connection for the NOTIFY and
reuse it for the SOA and AXFR?  The NAT respects open TCP connections
much more than UDP conversations. 

Should we manually add entries to the zone metadata to specify where
zones are really hosted?  I really wanted to use the auto-secondary
feature, but sometimes we can't have nice things.  :-/ 

Oh god, please don't tell me I have to set up a VPN between all the name
servers.... :-( 

Anyhow, I'd love to hear someone write a happy ending to this story. 

====== 

TL;DR: How do you get notifies + zone transfers to work when the source
ip addresses of NOTIFY packets are unreliable?

Thanks! 

-- 
CH (ch-and-pdns-users at ch.pkts.ca) 

Links:
------
[1] https://dnsdist.org/rules-actions.html#NetmaskGroupRule
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.powerdns.com/pipermail/pdns-users/attachments/20221117/7b0458dd/attachment.htm>