[Pdns-users] An important update on new PowerDNS Products

[Ciro Iriarte]

Interesting!, thanks a lot Bert.

Regards,
CI.-

2016-05-14 9:10 GMT-04:00 bert hubert <bert.hubert at powerdns.com>:

> On Fri, May 13, 2016 at 07:59:57PM -0400, Ciro Iriarte wrote:
> > Out of curiosity, how does this part of the platform work?:
>
> Hi Ciro,
>
> In general, I don't want to spam the pdns-users people with information
> about the PowerDNS Platform, as outlined on
> https://www.powerdns.com/platform.html since it is not part of the open
> source stuff, so most of the community won't have any use for the
> information.
>
> (the story of our non-open source work is on
>
> https://blog.powerdns.com/2016/02/23/an-important-update-on-new-powerdns-products/
> )
>
> But let me explain how this hangs together and what is part of the open
> source version. The PowerDNS Recursor 4.0.0 and dnsdist 1.0.0 have the
> ability to emit a stream of protobufs over TCP/IP. In case of the Recursor,
> this also has the 'policy reason' why a request was intercepted by the RPZ
> module.
>
> To receive that stream, use something like xinetd to listen on a TCP/IP
> port
> and store the data to a file. It can then be processed by any tool that can
> understand Protobuf. The schema is here:
> https://github.com/PowerDNS/pdns/blob/master/pdns/dnsmessage.proto
>
> In the very near future this will also be able to emit standard dnstap
> messages.
>
> > Long term full query logging & rapid searching
> >
> >    - Dimensioned at a trillion queries/day (1000 billion) on commodity
> >    hardware with long term retention
> >    - For security research, lawful intercept/data retention requirements,
> >    customer intelligence, quality assurance/diagnostics
>
> This describes our protobuf receiver 'dstore' which through some clever
> programming techniques can store trillions of DNS messages and serve them
> up
> again reasonably quickly. It is not a generic database, but it is really
> fast and nearly maintenance free and has no further dependencies (so you
> don't need to be a "big data engineer" to benefit from it).
>
> This can be very useful to investigate customer complaints of DNS slowness,
> or that a domain was down etc. It is also extremely powerful for finding
> infected users. A commandline like:
>
> $ dgrep t=week pr=spamhaus-dbl | jq ".items[].origRequestor"  | sort |
> uniq -c \
> | sort -rn | head -10
>
> .. will find in a few seconds the top-10 IP addresses that over the past
> week
> had the most queries  blocked by the 'spamhaus-dbl' RPZ. The output of
> dgrep is JSON,
> easily queried and selected by jq.
>
> But again - I don't want to promote our commercial Platform offering here
> too much.  For the open source world, you should be able to bake up a
> solution based on elastic search, kibana etc that ingests our protobufs.
>
>
>         Bert
>
>


-- 
Ciro Iriarte
http://iriarte.it
--
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.powerdns.com/pipermail/pdns-users/attachments/20160518/39c7bc31/attachment.html>