[dnsdist] dnsdist - Bad file descriptor

[Remi Gacogne]

Hi Aleš,

On 05/22/2018 01:14 PM, Aleš Rygl wrote:
> I am facing a bad file descriptor message when preparing our new
> configuration where dnsdist is listening on multiple interfaces. The
> situation is following. dnsdist is running on a sever with three bond
> interfaces. Two of them have two IPv4 and IPv6 address on  them and are
> used for DNS traffic. There is one for the physical interface and the
> second one is a VRRP floating IP, shared between two identical servers.
> 
> Sometimes it happens that dnsdist is not responding to queries on IPv6
> when the server is rebooted. They time out. I have noticed this error:
> 
> May 22 09:34:33 rzt-lvs2 dnsdist[966]: Marking downstream murena4-ipv6
> ([2001:1ae8:2:4010::1:5]:53) as 'up'
> May 22 09:34:33 rzt-lvs2 dnsdist[966]: Error connecting to new server
> with address [2001:1ae8:2:4010::1:5]:53: connecting socket to
> [2001:1ae8:2:4010::1:5]:53: Bad file descriptor
> May 22 09:34:34 rzt-lvs2 dnsdist[966]: Marking downstream murena4-ipv6
> ([2001:1ae8:2:4010::1:5]:53) as 'up'
> May 22 09:34:34 rzt-lvs2 dnsdist[966]: Marking downstream murena4-ipv6
> ([2001:1ae8:2:4010::1:5]:53) as 'up'
> May 22 09:34:34 rzt-lvs2 dnsdist[966]: Error connecting to new server
> with address [2001:1ae8:2:4010::1:5]:53: connecting socket to
> [2001:1ae8:2:4010::1:5]:53: Bad file descriptor
> May 22 09:34:34 rzt-lvs2 dnsdist[966]: Error connecting to new server
> with address [2001:1ae8:2:4010::1:5]:53: connecting socket to
> [2001:1ae8:2:4010::1:5]:53: Bad file descriptor
> May 22 09:34:35 rzt-lvs2 dnsdist[966]: Marking downstream murena4-ipv6
> ([2001:1ae8:2:4010::1:5]:53) as 'up'
> May 22 09:34:35 rzt-lvs2 dnsdist[966]: Marking downstream murena4-ipv6
> ([2001:1ae8:2:4010::1:5]:53) as 'up'
> May 22 09:34:35 rzt-lvs2 dnsdist[966]: Error connecting to new server
> with address [2001:1ae8:2:4010::1:5]:53: connecting socket to
> [2001:1ae8:2:4010::1:5]:53: Bad file descriptor
> May 22 09:34:35 rzt-lvs2 dnsdist[966]: Error connecting to new server
> with address [2001:1ae8:2:4010::1:5]:53: connecting socket to
> [2001:1ae8:2:4010::1:5]:53: Bad file descriptor
> May 22 09:34:36 rzt-lvs2 dnsdist[966]: Marking downstream murena4-ipv6
> ([2001:1ae8:2:4010::1:5]:53) as 'up'
> 
> What is wrong here? The resolver above is reachable via one of the bond
> interfaces and is responding (it is a backup from another site). When
> dnsdist is restarted it is ok. dnsdist is 1.3.0

It's clearly a bug, perhaps introduced in [1] although it looks like
that code was pretty much the same before. At some point a socket error
while trying to forward a query led us to close the socket and try to
re-open a new one, which failed too. We then left the socket as -1, and
the health check thread is trying to reconnect the socket without
creating a new file descriptor, hence the error. I'll fix it.

Would you mind sharing your configuration? I'm wondering why kind of
error caused the disconnection, although it might be a transient network
issue.

[1]: https://github.com/PowerDNS/pdns/pull/6317

-- 
Remi Gacogne
PowerDNS.COM BV - https://www.powerdns.com/