[dnsdist] dnsdist using loopback address as source address for queries

Fri Nov 12 03:05:19 UTC 2021

On 29 Oct 2021, at 15:09, Remi Gacogne via dnsdist <dnsdist at mailman.powerdns.com> wrote:
> The field holding that source address is not updated at runtime, and I would be surprised to see a memory corruption touching only that field. You might actually be able to check that via gdb, by attaching to the process, selecting the right thread with 'thread 7' (it needs to be one of these in responderThread() for the right backend, which is 7 or 9 in your current process based on the gdb output) then 'up' three times to reach the responderThread frame, and 'print *dss' to see the content of the structure.

'print *dss' didn't work, but dss looked like it contained a smart pointer, so I tried 'print *dss._M_ptr - the output of that is at the end of this message. The field seems intact though.

> It also happens on a health-check failure when 'reconnectOnFailure=true' is set on a backend, but that's not the case in your configuration. Actually, I wonder if that would make a difference, do you think you could try setting that option on one of the IPv6 backends?

It's tricky to tell because the issue only happens infrequently, but my impression is after letting the systems run for several days is that this has had an effect. I can say definitively though that the back end going away is the proximal cause - I bounced one of the backends up and down and I was able to trigger failures that coincided with the backend going down. There must be an additional timing or duration factor though, as I've not yet been able to trigger the issue on demand.

Adam

---

$2 = {id = {data = "\371\026\333?\347jHě\345=\274\f\365\031\b"}, hashes = {<std::_Vector_base<unsigned int, std::allocator<unsigned int> >> = {_M_impl = {<std::allocator<unsigned int>> = {<__gnu_cxx::new_allocator<unsigned int>> = {<No data fields>}, <No data fields>}, _M_start = 0x0, _M_finish = 0x0, _M_end_of_storage = 0x0}}, <No data fields>}, d_lock = {d_lock = {_M_impl = {_M_rwlock = {__data = {__readers = 0, __writers = 0, __wrphase_futex = 0, __writers_futex = 0, __pad3 = 0, __pad4 = 0, __cur_writer = 0, __shared = 0, __rwelision = 0 '\000', __pad1 = "\000\000\000\000\000\000", __pad2 = 0, __flags = 0}, __size = '\000' <repeats 55 times>, __align = 0}}}}, sockets = {<std::_Vector_base<int, std::allocator<int> >> = {_M_impl = {<std::allocator<int>> = {<__gnu_cxx::new_allocator<int>> = {<No data fields>}, <No data fields>}, _M_start = 0x5632ed7517f0, _M_finish = 0x5632ed7517f4, _M_end_of_storage = 0x5632ed7517f4}}, <No data fields>}, sourceItfName = {static npos = 18446744073709551615, _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0x5632ed9d4688 ""}, _M_string_length = 0, {_M_local_buf = '\000' <repeats 15 times>, _M_allocated_capacity = 0}}, socketsLock = {<std::__mutex_base> = {_M_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}}, <No data fields>}, connectLock = {<std::__mutex_base> = {_M_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}}, <No data fields>}, mplexer = {_M_t = {_M_t = {<std::_Tuple_impl<0, FDMultiplexer*, std::default_delete<FDMultiplexer> >> = {<std::_Tuple_impl<1, std::default_delete<FDMultiplexer> >> = {<std::_Head_base<1, std::default_delete<FDMultiplexer>, true>> = {<std::default_delete<FDMultiplexer>> = {<No data fields>}, <No data fields>}, <No data fields>}, <std::_Head_base<0, FDMultiplexer*, false>> = {_M_head_impl = 0x5632ed9d4ef0}, <No data fields>}, <No data fields>}}}, d_tlsCtx = {<std::__shared_ptr<TLSCtx, (__gnu_cxx::_Lock_policy)2>> = {<std::__shared_ptr_access<TLSCtx, (__gnu_cxx::_Lock_policy)2, false, false>> = {<No data fields>}, _M_ptr = 0x0, _M_refcount = {_M_pi = 0x0}}, <No data fields>}, tid = {_M_id = {_M_thread = 139732329490176}}, remote = {sin4 = {sin_family = 10, sin_port = 13568, sin_addr = {s_addr = 0}, sin_zero = " \001\006\060\000\001\001`"}, sin6 = {sin6_family = 10, sin6_port = 13568, sin6_flowinfo = 0, sin6_addr = {__in6_u = {__u6_addr8 = " \001\006\060\000\001\001`\000\000\000\000\000\000\001\225", __u6_addr16 = {288, 12294, 256, 24577, 0, 0, 0, 38145}, __u6_addr32 = {805699872, 1610678528, 0, 2499870720}}}, sin6_scope_id = 0}}, qps = {<BasicQPSLimiter> = {_vptr.BasicQPSLimiter = 0x5632ec120428 <vtable for QPSLimiter+16>, d_prev = {d_start = {tv_sec = 0, tv_nsec = 0}, d_needRealTime = false}, d_tokens = 0}, d_rate = 0, d_burst = 0, d_passthrough = true}, idStates = {<std::_Vector_base<IDState, std::allocator<IDState> >> = {_M_impl = {<std::allocator<IDState>> = {<__gnu_cxx::new_allocator<IDState>> = {<No data fields>}, <No data fields>}, _M_start = 0x7f15fa5d4010, _M_finish = 0x7f15fbbd3eb0, _M_end_of_storage = 0x7f15fbbd3eb0}}, <No data fields>}, sourceAddr = {sin4 = {sin_family = 2, sin_port = 0, sin_addr = {s_addr = 0}, sin_zero = "\001\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port = 0, sin6_flowinfo = 0, sin6_addr = {__in6_u = {__u6_addr8 = "\001\000\000\000\000\000\000\000\370C\345\001\375\177\000", __u6_addr16 = {1, 0, 0, 0, 17400, 485, 32765, 0}, __u6_addr32 = {1, 0, 31802360, 32765}}}, sin6_scope_id = 0}}, checkFunction = {<std::_Maybe_unary_or_binary_function<std::tuple<DNSName, unsigned short, unsigned short>, DNSName const&, unsigned short, unsigned short, dnsheader*>> = {<No data fields>}, <std::_Function_base> = {static _M_max_size = 16, static _M_max_align = 8, _M_functor = {_M_unused = {_M_object = 0x0, _M_const_object = 0x0, _M_function_pointer = 0x0, _M_member_pointer = NULL}, _M_pod_data = '\000' <repeats 15 times>}, _M_manager = 0x0}, _M_invoker = 0x0}, checkName = {d_storage = {<boost::container::container_detail::basic_string_base<boost::container::new_allocator<char> >> = {static MinInternalBufferChars = 8, static AlignmentOfValueType = 1, static ShortDataOffset = 1, static ZeroCostInternalBufferChars = 23, static UnalignedFinalInternalBufferChars = 23, members_ = {<boost::container::new_allocator<char>> = {<No data fields>}, m_repr = {r = {dummy = {dummy = "%\004lbdn\004virt\002ja\003net\000\000\375\177\000"}}, s = {h = {is_short = 1 '\001', length = 18 '\022'}, data = "\004lbdn\004virt\002ja\003net\000\000\375\177\000"}}}, static InternalBufferChars = 23, static MinAllocation = <optimized out>}, static InternalBufferChars = 23, static npos = <optimized out>}}, checkType = {static names = {_M_t = {_M_impl = {<std::allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, unsigned short> > >> = {<__gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, unsigned short> > >> = {<No data fields>}, <No data fields>}, <std::_Rb_tree_key_compare<std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const> >> = {_M_key_compare = {<std::binary_function<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, bool>> = {<No data fields>}, <No data fields>}}, <std::_Rb_tree_header> = {_M_header = {_M_color = std::_S_red, _M_parent = 0x5632ed762440, _M_left = 0x5632ed764ef0, _M_right = 0x5632ed7ab9c0}, _M_node_count = 63}, <No data fields>}}}, static numbers = {_M_t = {_M_impl = {<std::allocator<std::_Rb_tree_node<std::pair<unsigned short const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const> > >> = {<__gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<unsigned short const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const> > >> = {<No data fields>}, <No data fields>}, <std::_Rb_tree_key_compare<std::less<unsigned short> >> = {_M_key_compare = {<std::binary_function<unsigned short, unsigned short, bool>> = {<No data fields>}, <No data fields>}}, <std::_Rb_tree_header> = {_M_header = {_M_color = std::_S_red, _M_parent = 0x5632ed7abec0, _M_left = 0x5632ed7abba0, _M_right = 0x5632ed7ac5a0}, _M_node_count = 63}, <No data fields>}}}, code = 6}, checkClass = 1, idOffset = {<std::__atomic_base<unsigned long>> = {static _S_alignment = 8, _M_i = 146830}, static is_always_lock_free = true}, sendErrors = {counter = {__data = "\000\000\000\000\000\000\000", __align = {<No data fields>}}}, outstanding = {counter = {__data = "(\000\000\000\000\000\000", __align = {<No data fields>}}}, reuseds = {counter = {__data = "f=\002\000\000\000\000", __align = {<No data fields>}}}, queries = {counter = {__data = "\243=\002\000\000\000\000", __align = {<No data fields>}}}, responses = {counter = {__data = "\025\000\000\000\000\000\000", __align = {<No data fields>}}}, prev = {sendErrors = {counter = {__data = "\000\000\000\000\000\000\000", __align = {<No data fields>}}}, reuseds = {counter = {__data = "f=\002\000\000\000\000", __align = {<No data fields>}}}, queries = {counter = {__data = "\213=\002\000\000\000\000", __align = {<No data fields>}}}}, tcpDiedSendingQuery = {counter = {__data = "\000\000\000\000\000\000\000", __align = {<No data fields>}}}, tcpDiedReadingResponse = {counter = {__data = "\000\000\000\000\000\000\000", __align = {<No data fields>}}}, tcpGaveUp = {counter = {__data = "\000\000\000\000\000\000\000", __align = {<No data fields>}}}, tcpReadTimeouts = {counter = {__data = "\000\000\000\000\000\000\000", __align = {<No data fields>}}}, tcpWriteTimeouts = {counter = {__data = "\000\000\000\000\000\000\000", __align = {<No data fields>}}}, tcpConnectTimeouts = {counter = {__data = "\000\000\000\000\000\000\000", __align = {<No data fields>}}}, tcpCurrentConnections = {counter = {__data = "\000\000\000\000\000\000\000", __align = {<No data fields>}}}, tcpMaxConcurrentConnections = {counter = {__data = "\002\000\000\000\000\000\000", __align = {<No data fields>}}}, tcpReusedConnections = {counter = {__data = "\000\000\000\000\000\000\000", __align = {<No data fields>}}}, tcpNewConnections = {counter = {__data = "\025\000\000\000\000\000\000", __align = {<No data fields>}}}, tcpAvgQueriesPerConnection = {counter = {__data = "\022:\252S\326Z\310?", __align = {<No data fields>}}}, tcpAvgConnectionDuration = {counter = {__data = "\034\302$41\035\306@", __align = {<No data fields>}}}, socketsOffset = 146830, d_maxInFlightQueriesPerConn = 1, d_tcpConcurrentConnectionsLimit = 0, queryLoad = {counter = {__data = "L5\376m\n\017\201?", __align = {<No data fields>}}}, dropRate = {counter = {__data = "L5\376m\n\017\221?", __align = {<No data fields>}}}, latencyUsec = 0, order = 1, weight = 1, tcpConnectTimeout = 5, tcpRecvTimeout = 30, tcpSendTimeout = 30, checkInterval = 120, lastCheck = 87, sourceItf = 0, retries = 5, xpfRRCode = 0, checkTimeout = 1000, currentCheckFailures = 0 '\000', consecutiveSuccessfulChecks = 0 '\000', maxCheckFailures = 1 '\001', minRiseSuccesses = 1 '\001', sw = {d_start = {tv_sec = 2552988, tv_nsec = 966643874}, d_needRealTime = false}, pools = {_M_t = {_M_impl = {<std::allocator<std::_Rb_tree_node<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >> = {<__gnu_cxx::new_allocator<std::_Rb_tree_node<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >> = {<No data fields>}, <No data fields>}, <std::_Rb_tree_key_compare<std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >> = {_M_key_compare = {<std::binary_function<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool>> = {<No data fields>}, <No data fields>}}, <std::_Rb_tree_header> = {_M_header = {_M_color = std::_S_red, _M_parent = 0x0, _M_left = 0x5632ed9d4e18, _M_right = 0x5632ed9d4e18}, _M_node_count = 0}, <No data fields>}}}, availability = DownstreamState::Availability::Auto, mustResolve = true, upStatus = true, useECS = true, useProxyProtocol = false, setCD = false, disableZeroScope = false, connected = {_M_base = {static _S_alignment = 1, _M_i = true}, static is_always_lock_free = true}, threadStarted = {<std::__atomic_flag_base> = {_M_i = true}, <No data fields>}, tcpFastOpen = false, ipBindAddrNoPort = true, reconnectOnUp = false, name = {static npos = 18446744073709551615, _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0x5632ed9ca950 "[2001:630:1:160::195]:53"}, _M_string_length = 24, {_M_local_buf = "&", '\000' <repeats 14 times>, _M_allocated_capacity = 38}}, nameWithAddr = {static npos = 18446744073709551615, _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0x5632ed9ca9f0 "[2001:630:1:160::195]:53"}, _M_string_length = 24, {_M_local_buf = "&", '\000' <repeats 14 times>, _M_allocated_capacity = 38}}, d_stopped = false}

Jisc is a registered charity (number 1149740) and a company limited by guarantee which is registered in England under company number. 05747339, VAT number GB 197 0632 86. Jisc’s registered office is: 4 Portwall Lane, Bristol, BS1 6NB. T 0203 697 5800.

Jisc Services Limited is a wholly owned Jisc subsidiary and a company limited by guarantee which is registered in England under company number 02881024, VAT number GB 197 0632 86. The registered office is: 4 Portwall Lane, Bristol, BS1 6NB. T 0203 697 5800.

Jisc Commercial Limited is a wholly owned Jisc subsidiary and a company limited by shares which is registered in England under company number 09316933, VAT number GB 197 0632 86. The registered office is: 4 Portwall Lane, Bristol, BS1 6NB. T 0203 697 5800.

For more details on how Jisc handles your data see our privacy notice here: https://www.jisc.ac.uk/website/privacy-notice