<div dir="ltr">

<div dir="ltr" style="text-align:start;text-indent:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">(resending because image to large for list)</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px"><br class="gmail-Apple-interchange-newline">I'll try to answer the questions and add more info.</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">the ip over DNS queries are like this (answer)</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">    172.25.241.34.domain > dnsdist1.33499: [udp sum ok] 2837 q: NULL?<span> </span><a href="http://cf.uwdnhzkqc8eq3ncrj7uoph15twv4qfm9hbhejiz0rmvushszvow2ukg5jxw.ufbupkfs5rko5zhpitbxqbxkxtclg18z8nyqc1rsflo6w6nhp5tbgppvs8khtwx.rz9nnyx9bjlggmbwwkfebmmdtmir6prilloekh3fcxfg12g3h5jkctrj2jl.s23.2yf.de/" target="_blank" style="color:rgb(17,85,204)">cF.<wbr>UwDnhZkQc8EQ3Ncrj7UOpH15TWv4qF<wbr>m9hBhejIZ0rMVUSHszvOW2ukG5JXW.<wbr>UFbuPkfS5Rko5zhpItBXQBXKXTcLG1<wbr>8Z8Nyqc1RsFLo6W6nhp5TBgpPvS8KH<wbr>tWx.<wbr>Rz9NNyX9BJlGGMBWwKfebMmDtMIR6P<wbr>riLLOEkH3fCxfG12G3h5jkcTRJ2Jl.<wbr>s23.2yf.de</a>. 1/1/1<span> </span><a href="http://cf.uwdnhzkqc8eq3ncrj7uoph15twv4qfm9hbhejiz0rmvushszvow2ukg5jxw.ufbupkfs5rko5zhpitbxqbxkxtclg18z8nyqc1rsflo6w6nhp5tbgppvs8khtwx.rz9nnyx9bjlggmbwwkfebmmdtmir6prilloekh3fcxfg12g3h5jkctrj2jl.s23.2yf.de/" target="_blank" style="color:rgb(17,85,204)">cF.<wbr>UwDnhZkQc8EQ3Ncrj7UOpH15TWv4qF<wbr>m9hBhejIZ0rMVUSHszvOW2ukG5JXW.<wbr>UFbuPkfS5Rko5zhpItBXQBXKXTcLG1<wbr>8Z8Nyqc1RsFLo6W6nhp5TBgpPvS8KH<wbr>tWx.<wbr>Rz9NNyX9BJlGGMBWwKfebMmDtMIR6P<wbr>riLLOEkH3fCxfG12G3h5jkcTRJ2Jl.<wbr>s23.2yf.de</a>. [1m] NULL ns:<span> </span><a href="http://s23.2yf.de/" target="_blank" style="color:rgb(17,85,204)">s23.2yf.de</a>. [22h51m7s] NS<span> </span><a href="http://ems23.2yf.de/" target="_blank" style="color:rgb(17,85,204)">ems23.2yf.de</a>. ar: . OPT UDPsize=4096 (287)</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">As mentioned, the answer TTL is short (1m)</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">The chrome queries are for random, one label domains and are a "feature" as chromium code </div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">explains:</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">>Because this function can be called during startup, when kicking off a URL fetch can eat up 20 ms of time, we delay seven seconds, which is hopefully long enough to be after startup, but still get results back quickly.</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">>This component sends requests to three randomly generated, and thus likely nonexistent, hostnames. If at least two redirect to the same hostname, this suggests the ISP is hijacking NXDOMAIN, and the omnibox should treat similar redirected navigations as 'failed' when deciding whether to prompt the user with a 'did you mean to navigate' infobar for certain search inputs.</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">>trigger: "On startup and when IP address of the computer changes."</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">>We generate a random hostname with between 7 and 15 characters. </div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">obviously this queries are all NXDOMAIN, so there are suposed to be negatively cached</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">Today, to collect some data we disabled the 2 mentiones rules, both and each one by itself.</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">The dns tunnel did not have any visible impact.</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">But when we disabled the skip cache for 1 label queries we noticed again</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">the same behaviour.</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">remember that our cache is configures as:</div><span class="gmail-im" style="color:rgb(80,0,80);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px"><div>cache = newPacketCache(1000000, 86400, 0, 60, 60)</div><div> getPool("dnsdist1"):setCache(<wbr>cache)</div><div> setCacheCleaningDelay(30)</div><div> setCacheCleaningPercentage(<wbr>20)</div><div><br></div></span><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">and it usually stays at 80% whith a 98% hit rate</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">after disabling the skip the cache periodically (every 50 minutes, more or less)</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">filled up to 100%, the hit rate dropped to 92% and our backed queries rate jumped </div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">from 1.6 kqps to almost 6kqps </div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">This stays for almost half and hour and then recovers...</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">regarding:</div><span class="gmail-im" style="color:rgb(80,0,80);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px"><div>> Your cache is limited at 1 million, you could try a bit more. You could also</div><div>> explore the settings of newPacketCache in terms of TTL limits.</div><div><br></div></span><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">the cache size seems usually ok, because our hit rate stays at 98/99%</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">and we don't want to mess a lot with a production environment.</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">Also I don't know how the TTL limits affects negative caching, which seems </div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">to be the driver of this situation.</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">Some graphs links:</div><div><span style="font-size:12.8px">

<span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><a href="https://pasteboard.co/Hk5kkm6.png">https://pasteboard.co/Hk5kkm6.png</a>  -> general and net view</span> </span></div><div><span style="font-size:12.8px"><a href="https://pasteboard.co/Hk5hgU5.png">https://pasteboard.co/Hk5hgU5.png</a>  ->  cache behaviour</span><br></div><div><span style="font-size:12.8px"><a href="https://pasteboard.co/Hk5hRcd.png">https://pasteboard.co/Hk5hRcd.png</a> -> cache hit rate</span><br></div><div><br></div><div><span style="font-size:12.8px"><br></span></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">The graph show cache strange behavior without chrome rule.  until 14:00 when the rule is placed again</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">and cache normalize again.</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">As Daniel Stirnimann mentioned, I also think the issue is about negative caching TTL.</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">We will move to 1.3 in a couple of weeks and will update about this </div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">when info will be available.</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px">Any questions are welcome!</div></div><div class="gmail-yj6qo gmail-ajU" style="outline:none;padding:10px 0px;width:22px;margin:2px 0px 0px;color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><div id="gmail-:316" class="gmail-ajR" tabindex="0" style="background-color:rgb(241,241,241);border:1px solid rgb(221,221,221);clear:both;line-height:6px;outline:none;width:20px"><img class="gmail-ajT" src="https://ssl.gstatic.com/ui/v1/icons/mail/images/cleardot.gif" style="background: url("//ssl.gstatic.com/ui/v1/icons/mail/ellipsis.png") no-repeat; height: 8px; opacity: 0.3; width: 20px;"></div></div>

<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Sun, May 6, 2018 at 3:36 PM, Daniel Stirnimann <span dir="ltr"><<a href="mailto:daniel.stirnimann@switch.ch" target="_blank">daniel.stirnimann@switch.ch</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On 05.05.18 12:40, Ask Bjørn Hansen wrote:<br>
> <br>
>> On May 3, 2018, at 17:25, Nico <<a href="mailto:nicomail@gmail.com">nicomail@gmail.com</a>> wrote:<br>
>><br>
>> After some tcpdumping and testing we found that chrome and dns tunneling were filing the cache,<br>
>> even if the percent of this queries was very low in the total.<br>
> <br>
> What do those queries look like?<br>
<br>
</span>For the chrome part, I guess he is talking about queries like these from<br>
Android mobile devices using Google Chrome:<br>
<br>
xmbltwvfgzoj AAAA<br>
oputhfmeqha AAAA<br>
fpxfkjurisphngo AAAA<br>
oputhfmeqha A<br>
fpxfkjurisphngo A<br>
xmbltwvfgzoj A<br>
<br>
I noticed this too a few weeks ago when playing with an Android<br>
Emulator. I did not look into this more and cannot tell at what interval<br>
they appear exactly. They seem to appear at least every time I started<br>
Google Chrome. The queries are random. Next time they are completely<br>
different but of the same length and same query character set.<br>
<br>
The response is of course NXDOMAIN. Negative caching TTL for the root<br>
zone is 1 day.<br>
<br>
I guess most DNS resolver software limit the negative caching TTL to<br>
something a fair bit lower. I just looked it up for PowerDNS recursor<br>
and it is set to max 3600 sec:<br>
<a href="https://doc.powerdns.com/md/recursor/settings/#max-negative-ttl" rel="noreferrer" target="_blank">https://doc.powerdns.com/md/<wbr>recursor/settings/#max-<wbr>negative-ttl</a><br>
<br>
Maybe the problem is that dnsdist has no max negative ttl limit?<br>
<a href="https://dnsdist.org/guides/cache.html" rel="noreferrer" target="_blank">https://dnsdist.org/guides/<wbr>cache.html</a><br>
<br>
Daniel<br>
______________________________<wbr>_________________<br>
dnsdist mailing list<br>
<a href="mailto:dnsdist@mailman.powerdns.com">dnsdist@mailman.powerdns.com</a><br>
<a href="https://mailman.powerdns.com/mailman/listinfo/dnsdist" rel="noreferrer" target="_blank">https://mailman.powerdns.com/<wbr>mailman/listinfo/dnsdist</a><br>
</blockquote></div><br></div>