[Pdns-users] dnsdist and udp-in-errors

[janasowieso Nicht]

Hi all,

we are using dnsdist and experiencing many udp-in-errors (which didn't
appear while our testing),
we followed the "performance tuning" guide (
https://dnsdist.org/advanced/tuning.html),
currently we have around 1500 queries per second and the udp inerrors are
around 5 times higher than we had on a different/legacy setup with much
more traffic (~20k qps)!

Udp: InDatagrams NoPorts InErrors OutDatagrams RcvbufErrors SndbufErrors
InCsumErrors IgnoredMulti MemErrors
Udp: 2489359211 6885 3878512 2457164053 0 0 3878508 4 0

... on the new setup, it seems we have ~0.156% error rate:
(3878512 * 100) / 2489359211  ~ 0.156
... on our old server (no dnsdist) we have only 0.030% error rate (and this
server gets around 20k qps):
(8214162 * 100) / 27112333635 ~ 0.030

we already:
tuned the udp memory buffers (tested up to 67108864 for rmem and wmem max),
raise the maxopenfiles/LimitNOFILE to 30000,
added multiple addLocal's (up to 12 listeners for ipv4),

... but still, the inerrors are appearing,
via sockstat (ss -numpile), we can also see drops, e.g.:
UNCONN     0          0                              0.0.0.0:53
    0.0.0.0:*        users:(("dnsdist",pid=45388,fd=53)) uid:989
ino:51046314 sk:50b2 <->
         skmem:(r0,rb134217728,t0,tb67108864,f4096,w0,o0,bl0,d4422)

current setup:
dnsdist server with 8 cores and 16GB ram, added 12 addlocals for ipv4,
2 pdns-recursor backends with 8 cores and 16GB ram each.
also we have an anycast setup on the frontends (bird daemon) and the
clients are querying these anycast addresses (where dnsdist is also
listening of course)
the server has enough resources available (cpu load, memory, etc.), so we
couldnt find any bottlenecks yet.

do you maybe have any idea how we can figure out where these errors are
coming from exactly?
or if they really indicate a problem?
or maybe someone has experienced something similar?

would be great if someone has a hint or an idea for us :-)

many thanks and BR
Georg
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.powerdns.com/pipermail/pdns-users/attachments/20241107/22bee7ca/attachment.htm>