[Pdns-users] Pdns-users Digest, Vol 268, Issue 6

Mon May 12 14:36:25 UTC 2025

Hi Robby,

A valid question, non-stop forwarding and failover for voice services. Normally the session sync is fast enough for most traffic. However, with powerdns I appear to be finding the limits of how fast a session can sync. Which seems like a luxury issue have IMHO. It’s not a major problem, client will rerequest a DNS entry if the first one fails, and it’s only a tiny amount of traffic that’s erroneously dropped.

I think for now I’ll just permit the egress traffic based on the source port. If it was inbound traffic I would be less inclined to do so.

-- 
Best regards,
Djerk Geurts

> On 12 May 2025, at 15:28, pdns-users-request at mailman.powerdns.com wrote:
> 
> Send Pdns-users mailing list submissions to
> 	pdns-users at mailman.powerdns.com
> 
> To subscribe or unsubscribe via the World Wide Web, visit
> 	https://mailman.powerdns.com/mailman/listinfo/pdns-users
> or, via email, send a message with subject or body 'help' to
> 	pdns-users-request at mailman.powerdns.com
> 
> You can reach the person managing the list at
> 	pdns-users-owner at mailman.powerdns.com
> 
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Pdns-users digest..."
> 
> 
> Today's Topics:
> 
>   1. Recursor too fast? (Djerk Geurts)
>   2. Re: Recursor too fast? (Robby Pedrica)
>   3. Re: No response from pdns-recursor for some clients
>      (Robby Pedrica)
> 
> 
> ----------------------------------------------------------------------
> 
> Message: 1
> Date: Mon, 12 May 2025 15:04:44 +0100
> From: Djerk Geurts <djerk at maizymoo.com>
> To: pdns-users at mailman.powerdns.com
> Subject: [Pdns-users] Recursor too fast?
> Message-ID: <6CCD76D8-CC96-4E2A-BE86-B8F486C2E703 at maizymoo.com>
> Content-Type: text/plain; charset="utf-8"
> 
> An odd statement possibly, but I?m looking for a way to solve a problem (even if it?s a temporary solution).
> 
> The DC firewalls have changed and the recursors are located in a DMZ behind two HA firewalls in active/active mode. So far so good. The firewalls sync their state tables, so asymmetric return traffic works fine. Except when the recursor replies so quickly that the sync hasn?t updated the state table yet for the return packets. As a result we?re seeing a few drops among a lot of perfectly fine traffic.
> 
> I have a few things I can do:
> 
> 1) permit all outbound traffic with source udp/53 from the recursors. Not ideal, but low risk.
> 2) raise a support ticket with the firewall vendor. Will do this, but not holding my breath for a solution (if any)
> 3) delay DNS replies a millisecond or so. Not ideal as this introduces delay.
> 
> Thoughts?
> 
> -- 
> Best regards,
> Djerk Geurts
> 
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://mailman.powerdns.com/pipermail/pdns-users/attachments/20250512/62791407/attachment-0001.htm>
> 
> ------------------------------
> 
> Message: 2
> Date: Mon, 12 May 2025 15:26:57 +0100
> From: Robby Pedrica <rpedrica at gmail.com>
> To: pdns-users at mailman.powerdns.com
> Subject: Re: [Pdns-users] Recursor too fast?
> Message-ID: <1d8adb52-6f8e-42e1-8473-f4591bf50aa0 at gmail.com>
> Content-Type: text/plain; charset="utf-8"; Format="flowed"
> 
> Question Djerk: why are you running your firewalls in active/active? 
> This is an unusual configuration that has many challenges, including the 
> one you've just mentioned.
> 
> Regards
> 
> Robby
> 
> 
> On 2025/05/12 15:04, Djerk Geurts via Pdns-users wrote:
>> An odd statement possibly, but I?m looking for a way to solve a 
>> problem (even if it?s a temporary solution).
>> 
>> The DC firewalls have changed and the recursors are located in a DMZ 
>> behind two HA firewalls in active/active mode. So far so good. The 
>> firewalls sync their state tables, so asymmetric return traffic works 
>> fine. Except when the recursor replies so quickly that the sync hasn?t 
>> updated the state table yet for the return packets. As a result we?re 
>> seeing a few drops among a lot of perfectly fine traffic.
>> 
>> I have a few things I can do:
>> 
>> 1) permit all outbound traffic with source udp/53 from the recursors. 
>> Not ideal, but low risk.
>> 2) raise a support ticket with the firewall vendor. Will do this, but 
>> not holding my breath for a solution (if any)
>> 3) delay DNS replies a millisecond or so. Not ideal as this introduces 
>> delay.
>> 
>> Thoughts?
>> 
>> -- 
>> Best regards,
>> *Djerk Geurts*
>> 
>> 
>> _______________________________________________
>> Pdns-users mailing list
>> Pdns-users at mailman.powerdns.com
>> https://mailman.powerdns.com/mailman/listinfo/pdns-users
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://mailman.powerdns.com/pipermail/pdns-users/attachments/20250512/e21e6a6b/attachment-0001.htm>
> 
> ------------------------------
> 
> Message: 3
> Date: Mon, 12 May 2025 15:28:53 +0100
> From: Robby Pedrica <rpedrica at gmail.com>
> To: Otto Moerbeek <otto at drijf.net>
> Cc: All about using and deploying powerdns
> 	<pdns-users at mailman.powerdns.com>
> Subject: Re: [Pdns-users] No response from pdns-recursor for some
> 	clients
> Message-ID: <d96d5bed-1292-409a-a56e-f51fadba5a40 at gmail.com>
> Content-Type: text/plain; charset="utf-8"; Format="flowed"
> 
> 
> On 2025/05/08 15:25, Otto Moerbeek wrote:
>> The logs (in your original post) are redacted. So we cannot correlate
>> the log lines with your config. If posting unredacted logs is not
>> possible we cannot help you here.
>> 
>> 	-Otto
> Thanks Otto, I'll check internally if we can share otherwise, thanks for 
> your help and assume this request is closed.
> 
> Regards
> 
> Robby
>> 
>> On Thu, May 08, 2025 at 03:00:37PM +0100, Robby Pedrica wrote:
>> 
>>> 
>>> On 2025/04/30 12:41, Otto Moerbeek wrote:
>>>> On Tue, Apr 29, 2025 at 03:18:44PM +0100, Robby Pedrica via Pdns-users wrote:
>>>> 
>>>>> Hi pdns community
>>>>> 
>>>>> I've got an odd issue where some clients do not get a response from either
>>>>> of my 2 recursors. Both are v5.1.4 deployed via docker with fairly std
>>>>> configs. Generally the logs will indicate if something is not in the
>>>>> allowed-from list but these clients don't show there. For all intents and
>>>>> purposes, the recursors work normally and well for all my other clients.
>>>>> 
>>>> Since you left out specifics, it's not possible for us to see what is
>>>> going wrong. Please read
>>>> https://blog.powerdns.com/2016/01/18/open-source-support-out-in-the-open
>>>> and try again with no information edited except secrets like
>>>> passwords.
>>>> 
>>>> 	-Otto
>>> Hi Otto
>>> 
>>> 1 - thank you very much for your reply
>>> 
>>> 2 - my apologies for the delayed response however I've been travelling the
>>> last week
>>> 
>>> 3 - I intimately understand the requirement to provide as much information
>>> as possible as I provide support myself; in this case, I spent a significant
>>> amount of time troubleshooting and collecting information on the issue - |I
>>> thought I had provided everything relevant but it's clear from your reply
>>> that that is not the case; however what is not clear, is what I left out
>>> (and the provided link does not assist with specifics either).
>>> 
>>> I'm going to make the assumption that you are referring to the recursor.yml
>>> configuration file, and therefore provide that here in full (minus secrets):
>>> 
>>> ///
>>> ######### SECTION incoming #########
>>> incoming:
>>> ? listen:
>>> ? - 0.0.0.0
>>> ? - '::'
>>> ? allow_from:
>>> #??? - 0.0.0.0/0
>>> ? - 127.0.0.1
>>> ? - 172.0.0.0/8 # docker networks
>>> ? - 10.10.10.0/24 # client subnet
>>> 
>>> ##### The load factor used when PowerDNS is distributing queries to worker
>>> threads
>>> #?? distribution_load_factor: 0.0
>>> ##### Launch this number of distributor threads, distributing queries to
>>> other threads
>>> #?? distributor_threads: 0
>>> ? port: 53
>>> ? proxy_protocol_from: [105.55.55.33/32]
>>> ? use_incoming_edns_subnet: true
>>> ##### Maximum number of requests handled concurrently per TCP connection
>>> #?? max_concurrent_requests_per_tcp_connection: 10
>>> ##### Maximum number of simultaneous TCP clients
>>> ? max_tcp_clients: 128
>>> 
>>> ######### SECTION logging #########
>>> logging:
>>> ? common_errors: true
>>> ? disable_syslog: false
>>> #?? facility: ''
>>> ? loglevel: 6
>>> ##### Suppress logging of questions and answers
>>> ? quiet: false
>>> 
>>> ######### SECTION nod #########
>>> nod:
>>> ##### Log newly observed domains.
>>> ? log: true
>>> ##### Track newly observed domains (i.e. never seen before).
>>> #?? tracking: false
>>> 
>>> ######### SECTION outgoing #########
>>> outgoing:
>>> ? edns_subnet_allow_list: !override
>>> ? - 0.0.0.0/0.
>>> ? max_busy_dot_probes: 50
>>> 
>>> ######### SECTION packetcache #########
>>> packetcache:
>>> ##### Disable packetcache
>>> #?? disable: false
>>> 
>>> ######### SECTION recursor #########
>>> recursor:
>>> ? daemon: false
>>> ? etc_hosts_file: /etc/hosts
>>> ? hint_file: /etc/named.root.txt
>>> ? lua_config_file: /etc/proxy-map.lua
>>> ##### Launch this number of threads listening for and processing TCP queries
>>> #?? tcp_threads: 1
>>> ##### Launch this number of threads
>>> ? threads: 4
>>> ##### string reported on version.pdns or version.bind
>>> #?? version_string: '*runtime determined*'
>>> ? write_pid: true
>>> 
>>> ######### SECTION webservice #########
>>> webservice:
>>> ? address: 0.0.0.0
>>> ? allow_from: !override
>>> ? - 10.10.11.0/24
>>> ? api_key: ---
>>> ##### Amount of logging in the webserver (none, normal, detailed)
>>> ? loglevel: normal
>>> ? password: ---
>>> ? port: 8082
>>> ? webserver: true
>>> 
>>> ######### SECTION dnssec #########
>>> dnssec:
>>> ? log_bogus: false
>>> ? max_dnskeys: 2
>>> ? validation: process
>>> 
>>> ######### SECTION ecs #########
>>> ecs:
>>> ##### List of client netmasks for which EDNS Client Subnet will be added
>>> ? add_for:
>>> ? - 0.0.0.0/0
>>> ? - ::/0
>>> ///
>>> 
>>> The related proxy-map.lua:
>>> 
>>> ///
>>> -- protobufServer("10.10.11.50:514" , "maxQueuedEntries=100",
>>> "logQueries=true", "logResponses=true", "logMappedFrom=false")
>>> protobufServer("10.10.11.50:514")
>>> 
>>> -- AE
>>> addProxyMapping("10.10.10.0/24", "41.55.55.33")
>>> ///
>>> 
>>> I can't provide less sanitised information in the pcap and logs as that
>>> would expose sensitive information (which I think is reasonably sanitised).
>>> But let me know on this point in any case.
>>> 
>>> If you are however referring to something else, then I would appreciate you
>>> specifying the additional information that you would require to assist me in
>>> collecting that info.
>>> 
>>> Appreciate your time
>>> 
>>> Robby
>>> 
>>>>> Design:
>>>>> 
>>>>> client ---> firewall --- ipsec vpn --- firewall ---> recursor ---> internet
>>>>> 
>>>>> Troubleshooting:
>>>>> 
>>>>> - check for blocks due to allow_from (nothing listed for these clients)
>>>>> - check local firewall rules (nothing special or different for specific
>>>>> clients)
>>>>> - tcpdump on the recursor hosts show queries hitting those hosts
>>>>> - pcaps on both firewalls show good traffic
>>>>> - the start of the logs show the ACL for allow_from is correct
>>>>> 
>>>>> PDNS-rec Config:
>>>>> ------------------------
>>>>> 
>>>>> //
>>>>> /######### SECTION incoming #########
>>>>> incoming:
>>>>>  ? listen:
>>>>>  ? - 0.0.0.0
>>>>>  ? - '::'
>>>>>  ? allow_from:
>>>>>  ? - x.x.x.x/y
>>>>>  ? - etc.
>>>>> 
>>>>>  ? port: 53
>>>>>  ? proxy_protocol_from: [a.a.a.a/b]
>>>>>  ? use_incoming_edns_subnet: true
>>>>>  ? max_tcp_clients: 128/
>>>>> //
>>>>> 
>>>>> 
>>>>> PDNS-rec docker config:
>>>>> ---------------------------------
>>>>> 
>>>>> //
>>>>> /---
>>>>> version: '2.0'
>>>>> services:
>>>>>  ? recursor:
>>>>>  ??? image: powerdns/pdns-recursor-51:latest
>>>>>  ??? restart: always
>>>>>  ??? ports:
>>>>>  ????? - "53:53"
>>>>>  ????? - "53:53/udp"
>>>>>  ????? - "8082:8082"
>>>>>  ??? logging:
>>>>>  ????? driver: "syslog"
>>>>>  ??? volumes:
>>>>>  ????? - ./recursor.yml:/etc/powerdns/recursor.yml
>>>>>  ????? - ./named.root.txt:/etc/named.root.txt
>>>>>  ????? - ./proxy-map.lua:/etc/proxy-map.lua/
>>>>> //
>>>>> 
>>>>> PDNS-rec logs:
>>>>> ---------------------
>>>>> 
>>>>> recursor_1? | Apr 29 13:53:49 PowerDNS Recursor 5.1.4 (C) PowerDNS.COM BV
>>>>> recursor_1? | Apr 29 13:53:49 Using 64-bits mode. Built using gcc 10.2.1
>>>>> 20210110 on Apr? 8 2025 10:17:24 by root at localhost.
>>>>> recursor_1? | Apr 29 13:53:49 PowerDNS comes with ABSOLUTELY NO WARRANTY.
>>>>> This is free software, and you are welcome to redistribute it according to
>>>>> the terms of the GPL version 2.
>>>>> recursor_1? | Apr 29 13:53:49 msg="Processing main YAML settings"
>>>>> subsystem="config" level="0" prio="Notice" tid="0" ts="1745934829.121"
>>>>> path="/etc/powerdns/recursor.yml"
>>>>> recursor_1? | Apr 29 13:53:49 msg="YAML config found and processed"
>>>>> subsystem="config" level="0" prio="Notice" tid="0" ts="1745934829.121"
>>>>> configname="/etc/powerdns/recursor.yml"
>>>>> recursor_1? | Apr 29 13:53:49 msg="Enabling IPv4 transport for outgoing
>>>>> queries" subsystem="config" level="0" prio="Notice" tid="0"
>>>>> ts="1745934829.123"
>>>>> recursor_1? | Apr 29 13:53:49 msg="Setting access control"
>>>>> subsystem="config" level="0" prio="Info" tid="0" ts="1745934829.125"
>>>>> acl="allow-from" addresses="x.x.x.x/y a.a.a.a/b etc."
>>>>> recursor_1? | Apr 29 13:53:49 msg="Will not send queries to"
>>>>> subsystem="config" level="0" prio="Notice" tid="0" ts="1745934829.132"
>>>>> addresses="127.0.0.0/8 10.0.0.0/8 100.64.0.0/10 169.254.0.0/16
>>>>> 192.168.0.0/16 172.16.0.0/12 ::1/128 fc00::/7 fe80::/10 0.0.0.0/8
>>>>> 192.0.0.0/24 192.0.2.0/24 198.51.100.0/24 203.0.113.0/24 240.0.0.0/4 ::/96
>>>>> ::ffff:0:0/96 100::/64 2001:db8::/32 0.0.0.0 ::"
>>>>> 
>>>>> PDNS-rec host pcap:
>>>>> ------------------------------
>>>>> 
>>>>> tcpdump -i any -v 'host <client-ip>'
>>>>> tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size
>>>>> 262144 bytes
>>>>> 14:01:49.419703 IP (tos 0x0, ttl 124, id 45946, offset 0, flags [none],
>>>>> proto UDP (17), length 83)
>>>>>  ??? <client-hostname>.65424 > <recursor-hostname>.domain: 16579+ [1au] A?
>>>>> canary.officeapps.live.com. (55)
>>>>> 14:01:49.419758 IP (tos 0x0, ttl 123, id 45946, offset 0, flags [none],
>>>>> proto UDP (17), length 83)
>>>>>  ??? <client-hostname>.65424 > 172.24.0.2.domain: 16579+ [1au] A?
>>>>> canary.officeapps.live.com. (55)
>>>>> 14:01:49.419766 IP (tos 0x0, ttl 123, id 45946, offset 0, flags [none],
>>>>> proto UDP (17), length 83)
>>>>>  ??? <client-hostname>.65424 > 172.24.0.2.domain: 16579+ [1au] A?
>>>>> canary.officeapps.live.com. (55)
>>>>> 
>>>>> Any ideas on what could be wrong or what I'm missing here is appreciated.
>>>>> 
>>>>> Regards
>>>>> 
>>>>> Robby
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> Pdns-users mailing list
>>>>> Pdns-users at mailman.powerdns.com
>>>>> https://mailman.powerdns.com/mailman/listinfo/pdns-users
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://mailman.powerdns.com/pipermail/pdns-users/attachments/20250512/4080c0e8/attachment.htm>
> 
> ------------------------------
> 
> Subject: Digest Footer
> 
> _______________________________________________
> Pdns-users mailing list
> Pdns-users at mailman.powerdns.com
> https://mailman.powerdns.com/mailman/listinfo/pdns-users
> 
> 
> ------------------------------
> 
> End of Pdns-users Digest, Vol 268, Issue 6
> ******************************************

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.powerdns.com/pipermail/pdns-users/attachments/20250512/6142ed77/attachment-0001.htm>