[Pdns-users] auth multi-backend (bind+dnssec and gsqlite3) domain metadata mixed up

Sun Apr 7 21:59:43 UTC 2024

Hi,

Recently I've upgraded from PowerDNS Authoritative 4.6.2 to 4.9.0 and 
noticed a change in behaviour and I got it to work again, but it feels 
very fragile and I'd like some advice.

My instance runs a 'hidden primary' role where I have two backends (BIND 
with DNSSEC SQLite db and SQLite3) enabled like shown below.
The purpose is to have a few mostly static signed zones served from 
files using the bindbackend, and a dynamic zone in SQLite3 (unsigned for 
now).

This has been working fine on 4.2.x, 4.3.x, 4.6.2, but breaks on 4.9.0 
(didn't try 4.7.x, 4.8.x).
What I noticed after the upgrade is that the domain metadata for zones 
in the SQLite3 backend aren't recognized anymore and as a result 
DNSUPDATEs aren't accepted for that dynamic zone.

My configuration:

     launch=bind
     bind-config=/etc/powerdns/bindbackend.conf
     bind-dnssec-db=/var/lib/powerdns/bind-dnssec/bind-dnssec-db.sqlite3
     primary=yes
     secondary=no
     disable-axfr=no
     # Two secondaries, allow and notify both and only those two.
     allow-axfr-ips=10.51.0.3,10.52.0.3
     only-notify=10.51.0.3,10.52.0.3
     also-notify=10.51.0.3,10.52.0.3
     allow-notify-from=
     # AXFR signed zones, new signatures weekly.
     default-soa-edit-signed=INCREMENT-WEEKS

     launch+=gsqlite3
     gsqlite3-database=/var/lib/powerdns/gsqlite3-backend/sqlite3.db
     # DNSSEC on but only unsigned zones in there for now
     gsqlite3-dnssec=on
     dnsupdate=yes
     allow-dnsupdate-from=10.50.3.2

Zones are listed fine using pdnsutil, from both backends (i6t.nl = 
static BIND zone, dynamic.i6t.nl is in SQLite3):

     $ pdnsutil list-all-zones
     Apr 07 19:02:04 [bindbackend] Done parsing domains, 0 rejected,
     7 new, 0 removed
     i6t.nl
     [...]
     dynamic.i6t.nl

The domain metadata in SQLite3 is there:

     $ sqlite3 /var/lib/powerdns/gsqlite3-backend/sqlite3.db \
       "select * from domains;"
     1|dynamic.i6t.nl|||MASTER|2024040701|||

     $ sqlite3 /var/lib/powerdns/gsqlite3-backend/sqlite3.db \
       "select * from domainmetadata;"
     1|1|SOA-EDIT-DNSUPDATE|SOA-EDIT-INCREASE
     2|1|NOTIFY-DNSUPDATE|1
     3|1|TSIG-ALLOW-DNSUPDATE|dynamic-i6t-nl

     $ sqlite3 /var/lib/powerdns/gsqlite3-backend/sqlite3.db \
       "select * from tsigkeys;"
     1|dynamic-i6t-nl|hmac-sha256|[key omitted]

But, the metadata isn't recognized (no lines after Metadata...):

     $ pdnsutil get-meta dynamic.i6t.nl
     Apr 07 19:03:38 [bindbackend] Done parsing domains, 0 rejected,
     7 new, 0 removed
     Metadata for 'dynamic.i6t.nl'

Similarly, 'pdnsutil list-tsig-keys' shows an empty output too.

And, knowing this, it's no surprise DNSUPDATEs with TSIG aren't working 
anymore:

     Packet for domain 'dynamic.i6t.nl' denied: can't find TSIG key with
     name 'dynamic-i6t-nl' and algorithm '(empty)'

After some debugging I found the following; reverse the order of the 
enabled backends to be loaded by removing the two launch= lines with a 
single one (order matters!):

     launch=gsqlite3,bind

     $ pdnsutil get-meta dynamic.i6t.nl
     Apr 07 19:06:27 [bindbackend] Done parsing domains, 0 rejected,
     7 new, 0 removed
     Metadata for 'dynamic.i6t.nl'
     NOTIFY-DNSUPDATE = 1
     SOA-EDIT-DNSUPDATE = SOA-EDIT-INCREASE
     TSIG-ALLOW-DNSUPDATE = dynamic-i6t-nl

Things *appeared* to work again with DNSUPDATEs using TSIG.

... but now my signed BIND zones are broken; the DNSSEC metadata isn't 
recognized (keys, NSEC3, etc.) and shows unsigned:

     $ pdnsutil show-zone i6t.nl
     Apr 07 19:45:21 [bindbackend] Done parsing domains, 0 rejected,
     7 new, 0 removed
     This is a Master zone
     Last SOA serial number we notified: [...]
     Zone is not actively secured
     Metadata items: None
     No keys for zone 'i6t.nl'.

Reverting the order reversal it works for the BIND zones (but I have my 
initial problem again):

     $ pdnsutil show-zone i6t.nl
     Apr 07 19:42:48 [bindbackend] Done parsing domains, 0 rejected,
     7 new, 0 removed
     This is a Master zone
     Last SOA serial number we notified: [...]
     Metadata items:
         NSEC3PARAM      1 0 0 -
     Zone has hashed NSEC3 semantics, configuration: 1 0 0 -
     keys: [...omitted...]

Using pdnsutil's set-meta command I found out - to my surprise - the 
SQLite3 domain metadata now ends up in the BIND DNSSEC db (uhhhh?):

     $ pdnsutil set-meta dynamic.i6t.nl NOTIFY-DNSUPDATE 1
     $ sqlite3 /var/lib/powerdns/bind-dnssec/bind-dnssec-db.sqlite3 \
       "select * from domainmetadata where domain='dynamic.i6t.nl';"
     15|dynamic.i6t.nl|NOTIFY-DNSUPDATE|1

So I have moved the domain metadata to the BIND DNSSEC db and that 
actually *appears* to work, but then I'm facing the problem that the 
dynamic zone can't be AXFR'd with the following error on the secondaries 
on NOTIFY:

     While checking domain freshness: Query to '10.50.1.4' for SOA of
     'dynamic.i6t.nl' did not return a SOA
     Received serial number updates for 0 zones, had 1 timeout

Which I have worked-around with the settings on the hidden primary:

     zone-cache-refresh-interval=0
     zone-metadata-cache-ttl=0

(Those two settings I needed before on 4.5.x and 4.6.x for any zone 
transfer to complete on my instance, but I could remove after the 4.9.0 
upgrade it *seemed*.)

My main question is: is it something I've missed in the upgrade notes 
regarding multi-backend or metadata behaviour, was this a 
misconfiguration in the first place that worked by accident on 4.2.x., 
4.3.x and 4.6.x or am I looking at a bug right now with 4.9.0?

The main confusion here for me is how it can be loading the zones and 
records from both backends correctly, but not its metadata? My upgrade 
passed the smoke test at first because of that...

With the note from the FAQs [1]:

 > Do note that multi-backend behaviour is not specified and might
 > change between versions. This is especially true when DNSSEC is
 > involved.

I suddenly feel like I should never have had a multi-backend setup to 
begin with and I would be better off splitting them up into two 
instances. WDYT?

If it's really that broken I'd suggest to include a big fat 'unsupported 
configuration' warning in the logs when launching multiple backends.

Side question: suppose I want to enable DNSSEC on my dynamic zone, how 
do I combine any of the 'INCREASE' variants for SOA-EDIT-DNSUPDATE [2] 
with INCREMENT-WEEKS [3] to assure it triggers an AXFR with new weekly 
signatures if there are no updates? This is keeping me from enabling 
DNSSEC on the dynamic (sub)zone, but perhaps I'm missing something.

Thanks!

Gert

[1]: https://doc.powerdns.com/authoritative/appendices/FAQ.html
[2]: 
https://doc.powerdns.com/authoritative/dnsupdate.html#soa-edit-dnsupdate-settings
[3]: 
https://doc.powerdns.com/authoritative/dnssec/operational.html?highlight=increment%20weeks#increment-weeks