[dnsdist] DoH issues after 1.8.3 -> 1.9.0 upgrade

[Christoph]

Hi,

in February we upgraded our test DoH/DoT server from 1.8.3 to 1.9.0
but we did not notice any problems so we upgraded our production server
from 1.8.3 to 1.9.0 yesterday.

Immediately after upgrading our monitoring claimed our DoH service is 
unavailable (HTTP 400) but we were unable to reproduce it using firefox.

A closer look confirmed that there is some issue because we see about 
50% less DoH requests in our grafana graphs showing DoH request rates.

Having a look at the request rates per HTTP method suggests that we 
"loose" almost all GET requests but also a significant fraction of POST 
DoH requests.

sum by (method) 
(irate(dnsdist_frontend_doh_http_method_queries{job="$job"}[$__rate_interval]))

After looking at the TLS versions graph I noticed a clear correlation
but then I realized that all our DoH requests are TLS version 1.3
because we set minTLSVersion='tls1.3' - so this might be irrelevant.

irate(dnsdist_frontend_tlsqueries{job="$job"}[$__rate_interval])

2024-03-16 20:57:59 dnsdist upgraded: 1.8.3_1 -> 1.9.0
2024-03-16 20:59 monitoring says DoH is down (HTTP 400 - Bad Request)
monitoring requests this: 
https://doh.applied-privacy.net/query?dns=l1sBAAABAAAAAAAAA3d3dw1rbm90LXJlc29sdmVyAmN6AAAcAAE
Mar 17 02:40:45 bender-dpriv1 kernel: pid 77544 (dnsdist), jid 0, uid 
208: exited on signal 11 -> also interesting put likely unrelated?

Today we downgraded to 1.8.3, and everything went back to normal.

Is anyone else observing similar issues on dnsdist 1.9.0?

DoT does not appear to be affected.

best regards,
Christoph

OS: FreeBSD 13.2
dnsdist installed via pkg

our dnsdist config:

newServer({address="109.70.100.136", maxInFlight=1000, sockets=32, 
name="clamps"})
newServer({address="109.70.100.140", maxInFlight=1000, sockets=32, 
name="roberto"})
--newServer({address="109.70.100.133", sockets=4, name="titanius-dpriv1"})
setServerPolicy(leastOutstanding)

addTLSLocal("0.0.0.0", 
"/usr/local/etc/ssl/lego/certificates/doh.applied-privacy.net.crt", 
"/usr/local/etc/ssl/lego/certificates/doh.applied-privacy.net.key", 
{ciphers='ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256', 
minTLSVersion='tls1.2', tcpFastOpenQueueSize=1000, maxInFlight=1000 })
addTLSLocal("[::]", 
"/usr/local/etc/ssl/lego/certificates/doh.applied-privacy.net.crt", 
"/usr/local/etc/ssl/lego/certificates/doh.applied-privacy.net.key", 
{ciphers='ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256', 
minTLSVersion='tls1.2', tcpFastOpenQueueSize=1000, maxInFlight=1000 })

addDOHLocal("0.0.0.0:444", 
"/usr/local/etc/ssl/lego/certificates/doh.applied-privacy.net.crt", 
"/usr/local/etc/ssl/lego/certificates/doh.applied-privacy.net.key", 
"/query", {minTLSVersion='tls1.3', serverTokens='doh', 
tcpFastOpenQueueSize=1000, tcpListenQueueSize=4096 })
addDOHLocal("[::]:444", 
"/usr/local/etc/ssl/lego/certificates/doh.applied-privacy.net.crt", 
"/usr/local/etc/ssl/lego/certificates/doh.applied-privacy.net.key", 
"/query", {minTLSVersion='tls1.3', serverTokens='doh', 
tcpFastOpenQueueSize=1000, tcpListenQueueSize=4096 })

setACL({'0.0.0.0/0', '::/0'})
controlSocket('127.0.0.1:5199')
setConsoleACL('127.0.0.1/8')

setKey(....)

pc = newPacketCache(50000, {maxTTL=86400, minTTL=3, 
temporaryFailureTTL=60, staleTTL=60, dontAge=false})
getPool(""):setCache(pc)

webserver("127.0.0.1:8083")
setWebserverConfig({...})
setVerboseHealthChecks(true)
addAction(QTypeRule(65535), RCodeAction(DNSRCode.NOTIMP))