flowdumper

bwNetFlow Example: Consumer Dumper

This bwNetFlow Kafka Consumer reads flows from a Kafka Topic, applies filters and prints the flows as JSON. The topic is either the belWue general topic or more likely a customer specific topic for one customer ID only.

Connecting

-kafka.brokers string
    Kafka brokers separated by commas (default "127.0.0.1:9092,[::1]:9092")
-kafka.consumer_group string
    Kafka Consumer Group (default "dashboard")
-kafka.pass string
    Kafka password to authenticate with
-kafka.topic string
    Kafka topic to consume from (default "flow-messages-enriched")
-kafka.user string
    Kafka username to authenticate with
-kafka.auth bool
    Disable/Enable authentication to kafka (default true/enabled)
-kafka.tls bool
    Disable/Enable tls encryption to kafka (default true/enabled)

Filters

-filter.IPsv4 string
    If defined, only flows to/from this IP V4 subnet are considered. Leave empty to disable filter. Provide comma separated list to filter for multiple IP subnets.
-filter.IPsv6 string
    If defined, only flows to/from this IP V6 subnet are considered. Leave empty to disable filter. Provide comma separated list to filter for multiple IP subnets.
-filter.customerid string
    If defined, only flows for this customer are considered. Leave empty to disable filter. Provide comma separated list to filter for multiple customers.
-filter.peers string
    If defined, only flows to/from this peer are considered. Leave empty to disable filter. Provide comma separated list to filter for multiple peers.

Example: dumper [... connection options ...] --filter.customerid 10109 --filter.IPsv4 134.60.XY.0/24,134.ZA.BC.128/26 --filter.peers ECIX

Output

{"ipVersion":4,"srcIP":"172.217.21.195","dstIP":"134.60.XY.236","srcPort":443,"dstPort":39132,"proto":6,"peer":"ECIX","bytes":1472,"packets":32}
{"ipVersion":4,"srcIP":"134.60.30.XX","dstIP":"172.217.22.195","srcPort":54045,"dstPort":443,"proto":6,"peer":"ECIX","bytes":5952,"packets":64}

Usage with Docker Image

Use with docker run

docker run \
    -e KAFKA_BROKERS="BELWUE_KAFKA_CLUSTER" \
    -e KAFKA_AUTH="true" \
    -e KAFKA_TLS="true" \
    -e KAFKA_TOPIC="flow-messages-enriched-YOURCID" \
    -e KAFKA_CONSUMER_GROUP="YOURCID-DUMPER" \
    -e KAFKA_USER="YOUR_USERNAME" \
    -e KAFKA_PASS="" \
    -e FILTER_CUSTOMERIDS="" \
    -e FILTER_IPSV4="134.60.0.0/16" \
    -e FILTER_IPSV6="" \
    -e FILTER_PEERS="DFN Stuttgart,DFN Karlsruhe" \
    omi-registry.e-technik.uni-ulm.de/bwnetflow/kafka/consumer_dumper:latest

via docker-compose

version: '2'
services:
  kafka:
    image: omi-registry.e-technik.uni-ulm.de/bwnetflow/kafka/consumer_dumper:latest
    environment:
        KAFKA_BROKERS: ...
        KAFKA_AUTH: false
        KAFKA_TLS: false
        KAFKA_TOPIC: enriched_goflow_topic
        KAFKA_CONSUMER_GROUP: myconsumer
        FILTER_CUSTOMERIDS: 
        FILTER_IPSV4: 
        FILTER_IPSV6: 
        FILTER_PEERS: