[Pdns-users] remote backend

[Alexis Fidalgo]

Finally got it fixed. 
Did this, isolated the backend, made a pcap of the production site of the traffic flowing from pdns -> backend and filtered that to get the json’s and use those as input/real queries was ~350 queries.

A new python script, simple one with requests module using those jsons as inputs and keeping how many queries has responses, and the time for the response. 

With that, i made a min,max,avg calculations on the response times and started the tests with a single thread on the python script and slowly increasing the threads.

I found that ~15 threads started to push the ‘max’ response time from the backend, with 20 threads i had a LOT of queries far beyond 2 seconds max time (the pdns timeout on the backend that i have configured)

Long story short, i have no control over the network or even the mongodb which the backend was using, so i’ve installed a mongodb in a new vm right on the same hypervisor where the pdns+backends run, all problems gone, have no way if it was network problems or mongodb problems (no way to find it) but everything runs perfect now.

Thanks for your help guys, and for pointing me lmdb, not using it for this now (i will as soon as i can change the code) but started to test and using it in a few SIP proxy/routers and it’s great

regards
 



> On 29 Nov 2023, at 11:53, Brian Candler <b.candler at pobox.com> wrote:
> 
> On 29/11/2023 14:04, Alexis Fidalgo wrote:
>> So, by now, i dont know what is making for a query to be answered and another not (timeout) and in a retry is answered ok. (this is why i thought on speed and considered the unix socket but now i know it’s not that)
> 
> Put logging in your remote backend and show what queries it receives and how long it takes to respond to each one. Use these logs to check that the queries generated by PowerDNS are what you expect (it may make multiple requests for a single received DNS query).
> 
> You can also take PowerDNS entirely out of the problem by making a set of suitable test HTTP calls directly to your backend, for the same set queries that PowerDNS would generate. If you can prove that your backend is taking too long to answer them (on the first attempt at least), then you know where to investigate.
> 
> For example, it might be that MongoDB is doing a lot of slow disk seeks (is it spinning rust or SSD?) but once it has the answer, everything it needs is cached in RAM so it's much quicker on the second attempt. Or maybe it's not indexed properly. You really need to drill down further to prove or disprove that idea.
> 
> If you find that MongoDB is the bottleneck and can't be tuned, then there are other options. For example, if this database doesn't change very often, then you could write it out to a CDB file:
> 
> https://cr.yp.to/cdb.html
> https://en.wikipedia.org/wiki/Cdb_(software)
> 
> This is optimised for very fast lookup with minimum seeking, and can be indexed in a single pass - but it can't be modified, so you'd have to regenerate the whole file periodically.  Also it has a 4GB size limit which is probably an issue here (limiting you to avg 14 bytes per key/value pair) so you may need to split into multiple files.
> 
> A suitably-indexed Postgres table with 300 million entries is big but not impossible, and PowerDNS could query it directly.
>