[Pdns-users] Dnstap and kafka

[Michael Chisina]

Thanks for the info

# is there any need for middleware software if I use dtap (dnstap favoured)
following from github

https://github.com/mimuret/dtap#kafka

And also  configure the following:
Kafka

Make flatting DNSTAP message,And it forawrd to kafka host.

[[OutputKafks]]
Hosts = ["kafka.example.jp:9092"]
Topic  = "dnstap_message"



# the dataflow will be as follows:
 DNS message --->dnstap--->middleware--->Kafka producer connector --->
Kafka streaming---> Kafka consumer connector---> debezium connector
--->timescaledb(postgresql favoured)


Regards
Michael Chisina

On Wed, Jan 13, 2021, 5:27 PM Brian Candler <b.candler at pobox.com> wrote:

> On 13/01/2021 14:58, Michael Chisina via Pdns-users wrote:
>
> I want to frame stream powerdns  recursor DNS query and response using
> dnstap to an apache kafka remote server (202.20.20.1).
> # what are the configurations needed on recursor?
>
> dnstap doesn't talk to kafka.
>
> You'll need to run some middleware software which accepts dnstap
> <https://dnstap.info/>-formatted messages, and writes them to kafka -
> which could be in raw dnstap form, or decoded into some other form (e.g.
> JSON).  Google "dnstap kafka" for some options, or write your own.
>
> Then you configure pdns-recursor to send dnstap messages to this
> middleware server.
>
> If you run the middleware on the same server as pdns-recursor, then they
> can communicate over a unix domain socket.  If you want to communicate to a
> remote server over TCP, then you will need a sufficiently new version of
> libfstrm which supports this. In practice this means that if you are using
> Ubuntu then you need 20.04 not 18.04. See this thread:
>
> https://mailman.powerdns.com/pipermail/pdns-users/2020-June/026724.html
>
>
> # what is dns message schema(s) format for the database creation?
>
> kafka isn't a database.  If you want to write these messages to a
> database, then that's an additional step.  You'll need some more software
> which reads messages from kafka, decodes them, and writes to a database in
> some schema that you define.  Beware that a busy recursor can generate a
> very large volume of messages, so you might want to aggregate them first.
>
> HTH,
>
> Brian.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.powerdns.com/pipermail/pdns-users/attachments/20210114/b495c0d2/attachment.htm>