[dnsdist] DoH issues after 1.8.3 -> 1.9.0 upgrade

[Remi Gacogne]

Hi Christoph,

In addition to the issue mentioned by Otto, it might also be that the 
monitoring does not support HTTP/2. The new nghttp2 provider for 
incoming DNS over HTTPS does not support HTTP/1.1. In 1.9.x it's still 
possible to switch back to the legacy h2o provider but note that it will 
likely go away in the next major version of DNSdist. In our testing the 
lack of HTTP/1.1 support was not an issue for actual DNS over HTTPS 
clients, with most of HTTP/1.1 queries coming from crawlers/bots, but of 
course we will reconsider if you find out that legitimate DoH clients 
are impacted.

Best regards,

Remi

On 17/03/2024 19:12, Otto Moerbeek via dnsdist wrote:
> On Sun, Mar 17, 2024 at 06:41:13PM +0100, Christoph via dnsdist wrote:
> 
>> Hi,
>>
>> in February we upgraded our test DoH/DoT server from 1.8.3 to 1.9.0
>> but we did not notice any problems so we upgraded our production server
>> from 1.8.3 to 1.9.0 yesterday.
>>
>> Immediately after upgrading our monitoring claimed our DoH service is
>> unavailable (HTTP 400) but we were unable to reproduce it using firefox.
>>
>> A closer look confirmed that there is some issue because we see about 50%
>> less DoH requests in our grafana graphs showing DoH request rates.
>>
>> Having a look at the request rates per HTTP method suggests that we "loose"
>> almost all GET requests but also a significant fraction of POST DoH
>> requests.
>>
>> sum by (method) (irate(dnsdist_frontend_doh_http_method_queries{job="$job"}[$__rate_interval]))
>>
>> After looking at the TLS versions graph I noticed a clear correlation
>> but then I realized that all our DoH requests are TLS version 1.3
>> because we set minTLSVersion='tls1.3' - so this might be irrelevant.
>>
>> irate(dnsdist_frontend_tlsqueries{job="$job"}[$__rate_interval])
>>
>> 2024-03-16 20:57:59 dnsdist upgraded: 1.8.3_1 -> 1.9.0
>> 2024-03-16 20:59 monitoring says DoH is down (HTTP 400 - Bad Request)
>> monitoring requests this: https://doh.applied-privacy.net/query?dns=l1sBAAABAAAAAAAAA3d3dw1rbm90LXJlc29sdmVyAmN6AAAcAAE
>> Mar 17 02:40:45 bender-dpriv1 kernel: pid 77544 (dnsdist), jid 0, uid 208:
>> exited on signal 11 -> also interesting put likely unrelated?
>>
>> Today we downgraded to 1.8.3, and everything went back to normal.
>>
>> Is anyone else observing similar issues on dnsdist 1.9.0?
>>
>> DoT does not appear to be affected.
>>
>> best regards,
>> Christoph
>>
>> OS: FreeBSD 13.2
>> dnsdist installed via pkg
>>
>> our dnsdist config:
>>
>> newServer({address="109.70.100.136", maxInFlight=1000, sockets=32,
>> name="clamps"})
>> newServer({address="109.70.100.140", maxInFlight=1000, sockets=32,
>> name="roberto"})
>> --newServer({address="109.70.100.133", sockets=4, name="titanius-dpriv1"})
>> setServerPolicy(leastOutstanding)
>>
>> addTLSLocal("0.0.0.0",
>> "/usr/local/etc/ssl/lego/certificates/doh.applied-privacy.net.crt",
>> "/usr/local/etc/ssl/lego/certificates/doh.applied-privacy.net.key", {ciphers='ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256',
>> minTLSVersion='tls1.2', tcpFastOpenQueueSize=1000, maxInFlight=1000 })
>> addTLSLocal("[::]",
>> "/usr/local/etc/ssl/lego/certificates/doh.applied-privacy.net.crt",
>> "/usr/local/etc/ssl/lego/certificates/doh.applied-privacy.net.key", {ciphers='ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256',
>> minTLSVersion='tls1.2', tcpFastOpenQueueSize=1000, maxInFlight=1000 })
>>
>> addDOHLocal("0.0.0.0:444",
>> "/usr/local/etc/ssl/lego/certificates/doh.applied-privacy.net.crt",
>> "/usr/local/etc/ssl/lego/certificates/doh.applied-privacy.net.key",
>> "/query", {minTLSVersion='tls1.3', serverTokens='doh',
>> tcpFastOpenQueueSize=1000, tcpListenQueueSize=4096 })
>> addDOHLocal("[::]:444",
>> "/usr/local/etc/ssl/lego/certificates/doh.applied-privacy.net.crt",
>> "/usr/local/etc/ssl/lego/certificates/doh.applied-privacy.net.key",
>> "/query", {minTLSVersion='tls1.3', serverTokens='doh',
>> tcpFastOpenQueueSize=1000, tcpListenQueueSize=4096 })
>>
>> setACL({'0.0.0.0/0', '::/0'})
>> controlSocket('127.0.0.1:5199')
>> setConsoleACL('127.0.0.1/8')
>>
>> setKey(....)
>>
>> pc = newPacketCache(50000, {maxTTL=86400, minTTL=3, temporaryFailureTTL=60,
>> staleTTL=60, dontAge=false})
>> getPool(""):setCache(pc)
>>
>> webserver("127.0.0.1:8083")
>> setWebserverConfig({...})
>> setVerboseHealthChecks(true)
>> addAction(QTypeRule(65535), RCodeAction(DNSRCode.NOTIMP))
> 
> 
> This might be related: https://github.com/PowerDNS/pdns/issues/13850,
> not backported yet
> 
> 	-Otto
> 
> _______________________________________________
> dnsdist mailing list
> dnsdist at mailman.powerdns.com
> https://mailman.powerdns.com/mailman/listinfo/dnsdist

-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://mailman.powerdns.com/pipermail/dnsdist/attachments/20240318/b24cca63/attachment.sig>