[Pdns-users] An important update on new PowerDNS Products
bert hubert
bert.hubert at powerdns.com
Sat May 14 13:10:22 UTC 2016
On Fri, May 13, 2016 at 07:59:57PM -0400, Ciro Iriarte wrote:
> Out of curiosity, how does this part of the platform work?:
Hi Ciro,
In general, I don't want to spam the pdns-users people with information
about the PowerDNS Platform, as outlined on
https://www.powerdns.com/platform.html since it is not part of the open
source stuff, so most of the community won't have any use for the
information.
(the story of our non-open source work is on
https://blog.powerdns.com/2016/02/23/an-important-update-on-new-powerdns-products/
)
But let me explain how this hangs together and what is part of the open
source version. The PowerDNS Recursor 4.0.0 and dnsdist 1.0.0 have the
ability to emit a stream of protobufs over TCP/IP. In case of the Recursor,
this also has the 'policy reason' why a request was intercepted by the RPZ
module.
To receive that stream, use something like xinetd to listen on a TCP/IP port
and store the data to a file. It can then be processed by any tool that can
understand Protobuf. The schema is here:
https://github.com/PowerDNS/pdns/blob/master/pdns/dnsmessage.proto
In the very near future this will also be able to emit standard dnstap
messages.
> Long term full query logging & rapid searching
>
> - Dimensioned at a trillion queries/day (1000 billion) on commodity
> hardware with long term retention
> - For security research, lawful intercept/data retention requirements,
> customer intelligence, quality assurance/diagnostics
This describes our protobuf receiver 'dstore' which through some clever
programming techniques can store trillions of DNS messages and serve them up
again reasonably quickly. It is not a generic database, but it is really
fast and nearly maintenance free and has no further dependencies (so you
don't need to be a "big data engineer" to benefit from it).
This can be very useful to investigate customer complaints of DNS slowness,
or that a domain was down etc. It is also extremely powerful for finding
infected users. A commandline like:
$ dgrep t=week pr=spamhaus-dbl | jq ".items[].origRequestor" | sort | uniq -c \
| sort -rn | head -10
.. will find in a few seconds the top-10 IP addresses that over the past week
had the most queries blocked by the 'spamhaus-dbl' RPZ. The output of dgrep is JSON,
easily queried and selected by jq.
But again - I don't want to promote our commercial Platform offering here
too much. For the open source world, you should be able to bake up a
solution based on elastic search, kibana etc that ingests our protobufs.
Bert
More information about the Pdns-users
mailing list