[Pdns-dev] PDNS Recursor functionality request re:SERVFAIL outages of today

[John Todd]

On 21 Oct 2016, at 18:48, Greg Owen wrote:

> On 2016-10-21 19:53, John Todd wrote:
>> I’d like to propose an extension to PowerDNS Recursor for 
>> mitigating
>> (partially) events like we had today where major authoritative
>> nameservers were put out of commission. This might be a particularly
>> foolish or error-prone method - it only took me a few minutes to 
>> think
>> up. But I’d at least like to hear a discussion as to why this 
>> isn’t a
>> good idea. The comment of “But this might end up giving out the 
>> wrong
>> answer!” is true, but I view a wrong answer as better than no 
>> answer.
> ...
>> If that query fails due to a SERVFAIL, then the TTL timer on this
>> “old” record is set back to zero and the “old” record is 
>> provided as
>> a response. If an authoritative server is marked as “down” due to
>> repeated SERVFAIL responses (see packetcache-servfail-ttl) then the
>> “old” record is handed back immediately without a new query 
>> attempt,
>> and the TTL timer is set back to zero to keep the answer in a state
>> of perpetual validity as long as....
>
> There are security concerns to doing this.  Most simply, a wrong 
> answer is worse than no answer if the "wrong" answer is a maliciously 
> sourced record.
>
> Consider the two following cases:
>
> 1) The attacker poisons the records for a zone - either indirectly, or 
> via compromise of the actual authoritative servers - and then takes 
> the actual servers down hard, causing SERVFAIL until the owners either 
> fix the servers, weather the DDoS, or redirect the root NS records.
>
> 2) The attacker poisons the records directly via compromise of the 
> actual authoritative servers, and the owner takes the servers down 
> until they can be replaced with clean, secured versions.
>
> In these two cases, the measure you're proposing would persist the 
> malicious entries past their expiration and for the duration of the 
> attack's effectiveness on the authoritative servers.
>
> Even if your measure is triggered manually - in today's event, for 
> example, one says "Gosh, I know records are offline because of a Dyn 
> DDoS, so I know I can compensate by saving records, throw the switch!" 
> - let's say that someone DDoSed *ALL* of Dyn after poisoning records 
> for a single zone.  You'd have no way of knowing - until the incident 
> is over and forensic analysis has hopefully caught that nuance - that 
> you were doing the attacker's work for them.
>
> These attack vectors are not without precedent.  So-called "Dark DDoS" 
> attacks have been used to distract and mislead defenders, providing a 
> smoke screen for other more direct attacks:
>
> http://www.infosecurity-magazine.com/opinions/dark-ddos-growing-cyber-security/
>
> So, in short, your proposal has the caveat that it may extend the 
> damage from an attack in more pernicious ways than simple denial of 
> service.  (I'd rather not get to my bank than get to an impostor 
> posing as my bank!)
>
> ...
>
> I agree it's worth putting some thought into how to increase 
> redundancy and flexibility to compensate for these infrastructure 
> attacks.  For example, perhaps taking your idea but only applying it 
> to signed DNSSEC records which have slightly higher data integrity?  
> It's definitely worth exploring, but let's be careful of known and 
> reasonable ways attackers could take advantage of this compensation.
>
> Thx,
> gowen
>
> -- 
>     gowen - Greg Owen - gowen at swynwyr.com
>     CISSP, GCIA, GCFA, GWAPT


I agree with your caveats to a degree, but I can only imagine the 
results being “worse” in a few edge cases of not-as-clever attacks.

In both the first and second case you describe above, would it not be 
the case that a sophisticated attacker would give an unusually large TTL 
to the poisoned record in order to avoid repair attempts? A TTL can be 
(if I’m reading the RFC correctly) 68 years. I would expect poisoned 
entries to be at least 3600 seconds (which is also the default value in 
packetcache-ttl) if not significantly more, but I can’t say I’ve 
ever paid attention to that number when looking at forensic data online, 
and perhaps I overestimate the baseline sophistication of attackers - 
but I don’t think so.

Of course, we’re trying via a number of other methods to eliminate 
cache poisoning, so that’s a first step on your case #1. DNSSEC is the 
best method I can see at the moment for this, so at a minimum it does 
seem that this extended TTL timer would work with reasonable expected 
“good” results on those records as you suggest, but I don’t think 
it should be limited to just DNSSEC-secured records. In case #2, I 
can’t imagine that a domain operator would have their servers offline 
intentionally for longer than the TTL of the poisoned record - are there 
instances where nameservers are down for several hours intentionally 
after a breach?  In that case, there’s at least an hour of “bad” 
data infecting various recursive servers, and I imagine whatever damage 
that is to happen is significantly done after an hour, and one would 
hope that alternate methods (SSL, or DANE!) would provide an additional 
layer of security.  I am not suggesting that no additional damage will 
be done during the TTL extension period, but that the cases where that 
occurs are few and the benefit of operational continuity during 
authoritative server outages outweighs the risk of longer-duration 
failure modes.


My assertion is: given an attacker with even the most modestly 
intelligent attack method, I would expect the long indefinite extension 
of TTL in the case of SERVFAIL will probably result in conditions not 
significantly worse for end users than if the SERVFAIL TTL extension 
method were not used, even in conditions where a poisoned record is 
inserted into the cache.  No _new_ failure modes or results are being 
introduced by this method.

Implementing the timer of course, could also be an optional method with 
a default of “0”, giving the recursor operator the flexibility for 
enabling/disabling given the requirements of their user community.  I 
can also imagine an additional counter option on this method which 
limits the maximum number of times a TTL may be overridden on a record, 
or an ultimate maximum TTL. There may be other more complex ways of 
allowing a domain operator to signal behavior in SERVFAIL conditions for 
a particular zone or record, such as TXT tags or possibly SRV record 
types, but they imply lookup and caching before a SERVFAIL condition 
which has a slew of unsatisfactory traffic and conditional state-keeping 
issues and possible self-referential loop failures, and chances of 
widespread adoption in a reasonable timeframe are fairly low though 
I’m sure it would make for an interesting IETF sub-track.  A TTL on 
TTL? Ugh. Keeping this simple seems to be the best way forward.

I believe putting this timer override method in the resolver is the 
fastest way to give local resiliency to resolvers faced with 
authoritative server outages.

JT
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.powerdns.com/pipermail/pdns-dev/attachments/20161021/0c1e2437/attachment-0001.html>