[Pdns-users] An important update on new PowerDNS Products

[bert hubert]

On Fri, May 13, 2016 at 07:59:57PM -0400, Ciro Iriarte wrote:
> Out of curiosity, how does this part of the platform work?:

Hi Ciro,

In general, I don't want to spam the pdns-users people with information
about the PowerDNS Platform, as outlined on
https://www.powerdns.com/platform.html since it is not part of the open
source stuff, so most of the community won't have any use for the
information.

(the story of our non-open source work is on
https://blog.powerdns.com/2016/02/23/an-important-update-on-new-powerdns-products/
)

But let me explain how this hangs together and what is part of the open
source version. The PowerDNS Recursor 4.0.0 and dnsdist 1.0.0 have the
ability to emit a stream of protobufs over TCP/IP. In case of the Recursor,
this also has the 'policy reason' why a request was intercepted by the RPZ
module.

To receive that stream, use something like xinetd to listen on a TCP/IP port
and store the data to a file. It can then be processed by any tool that can
understand Protobuf. The schema is here:
https://github.com/PowerDNS/pdns/blob/master/pdns/dnsmessage.proto

In the very near future this will also be able to emit standard dnstap
messages.

> Long term full query logging & rapid searching
> 
>    - Dimensioned at a trillion queries/day (1000 billion) on commodity
>    hardware with long term retention
>    - For security research, lawful intercept/data retention requirements,
>    customer intelligence, quality assurance/diagnostics

This describes our protobuf receiver 'dstore' which through some clever
programming techniques can store trillions of DNS messages and serve them up
again reasonably quickly. It is not a generic database, but it is really
fast and nearly maintenance free and has no further dependencies (so you
don't need to be a "big data engineer" to benefit from it).

This can be very useful to investigate customer complaints of DNS slowness,
or that a domain was down etc. It is also extremely powerful for finding
infected users. A commandline like:

$ dgrep t=week pr=spamhaus-dbl | jq ".items[].origRequestor"  | sort | uniq -c \
| sort -rn | head -10

.. will find in a few seconds the top-10 IP addresses that over the past week 
had the most queries  blocked by the 'spamhaus-dbl' RPZ. The output of dgrep is JSON, 
easily queried and selected by jq.

But again - I don't want to promote our commercial Platform offering here
too much.  For the open source world, you should be able to bake up a
solution based on elastic search, kibana etc that ingests our protobufs. 


	Bert