segments

package
v1.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 13, 2023 License: GPL-3.0 Imports: 4 Imported by: 0

Documentation

Overview

This package is home to all pipeline segment implementations. Generally, every segment lives in its own package, implements the Segment interface, embeds the BaseSegment to take care of the I/O side of things, and has an additional init() function to register itself using RegisterSegment.

Index

Constants

This section is empty.

Variables

View Source
var (
	ContainerVolumePrefix = ""
)

Functions

func RegisterSegment

func RegisterSegment(name string, s Segment)

Used by Segments to register themselves in their init() functions. Errors and exits immediately on conflicts.

func TestSegment

func TestSegment(name string, config map[string]string, msg *pb.EnrichedFlow) *pb.EnrichedFlow

Used by the tests to run single flow messages through a segment.

Types

type BaseFilterSegment

type BaseFilterSegment struct {
	BaseSegment
	Drops chan<- *pb.EnrichedFlow
}

An extended basis for Segment implementations in the filter group. It contains the necessities to process filtered (dropped) flows.

func (*BaseFilterSegment) SubscribeDrops

func (segment *BaseFilterSegment) SubscribeDrops(drops chan<- *pb.EnrichedFlow)

Set a return channel for dropped flow messages. Segments need to be wary of this channel closing when producing messages to this channel. This method is only called by the flowpipeline tool from the controlflow/branch segment to implement the then/else branches, otherwise this functionality is unused.

type BaseSegment

type BaseSegment struct {
	In  <-chan *pb.EnrichedFlow
	Out chan<- *pb.EnrichedFlow
}

Serves as a basis for any Segment implementations. Segments embedding this type only need the New and the Run methods to be compliant to the Segment interface.

func (*BaseSegment) Rewire

func (segment *BaseSegment) Rewire(in chan *pb.EnrichedFlow, out chan *pb.EnrichedFlow)

This function rewires this Segment with the provided channels. This is typically called only by pipeline.New() and present in any Segment implementation embedding the BaseSegment. The peculiar implementation of passing the full channel list and providing indexes is due to the fact that controlflow segments may want to skip segments and thus need to have all later references available as well.

func (*BaseSegment) ShutdownParentPipeline

func (segment *BaseSegment) ShutdownParentPipeline()

This functions shutdown Parent Pipeline segments on the given syscall. It is used for intended termination within pipeline function, e.g. end pipeline on read from file.

type Segment

type Segment interface {
	New(config map[string]string) Segment                       // for reading the provided config
	Run(wg *sync.WaitGroup)                                     // goroutine, must close(segment.Out) when segment.In is closed
	Rewire(in chan *pb.EnrichedFlow, out chan *pb.EnrichedFlow) // embed this using BaseSegment
	ShutdownParentPipeline()                                    // shut down Parent Pipeline gracefully
}

This interface is central to an Pipeline object, as it operates on a list of them. In general, Segments should embed the BaseSegment to provide the Rewire function and the associated vars.

func LookupSegment

func LookupSegment(name string) Segment

Used by the pipeline package to convert segment names in configuration to actual Segment objects.

Directories

Path Synopsis
alert
http
This segment is used to alert on flows using webhooks - WIP, but basically usable.
This segment is used to alert on flows using webhooks - WIP, but basically usable.
analysis
controlflow
export
clickhouse
Dumps all incoming flow messages to a clickhouse database.
Dumps all incoming flow messages to a clickhouse database.
influx
Collects and exports all flows to influxdb for long term storage.
Collects and exports all flows to influxdb for long term storage.
prometheus
Collects and serves statistics about flows.
Collects and serves statistics about flows.
filter
elephant
Filters out the bulky average of flows.
Filters out the bulky average of flows.
flowfilter
Runs flows through a filter and forwards only matching flows.
Runs flows through a filter and forwards only matching flows.
input
bpf
goflow
Captures Netflow v9 and feeds flows to the following segments.
Captures Netflow v9 and feeds flows to the following segments.
kafkaconsumer
Consumes flows from a Kafka instance and passes them to the following segments.
Consumes flows from a Kafka instance and passes them to the following segments.
stdin
Receives flows from stdin in JSON format, as exported by the json segment.
Receives flows from stdin in JSON format, as exported by the json segment.
modify
addcid
Enriches any passing flow message with a customer id field based on a CIDR match.
Enriches any passing flow message with a customer id field based on a CIDR match.
anonymize
Anonymize uses the CryptoPan prefix-preserving IP address sanitization as specified by J. Fan, J. Xu, M. Ammar, and S. Moon.
Anonymize uses the CryptoPan prefix-preserving IP address sanitization as specified by J. Fan, J. Xu, M. Ammar, and S. Moon.
bgp
Enriches flows with infos from BGP.
Enriches flows with infos from BGP.
dropfields
Drops fields from any passing flow.
Drops fields from any passing flow.
geolocation
Enriches flows with a geolocation.
Enriches flows with a geolocation.
normalize
Rewrites passing flows with all sampling rate affected fields normalized.
Rewrites passing flows with all sampling rate affected fields normalized.
protomap
Enriches any passing flow message with the human readable protocol.
Enriches any passing flow message with the human readable protocol.
remoteaddress
Determines the remote address of flows based on different criteria.
Determines the remote address of flows based on different criteria.
reversedns
Rewrites the Note field of passing flows to the remote addresses reverse DNS entry.
Rewrites the Note field of passing flows to the remote addresses reverse DNS entry.
snmp
Enriches passing flows with human-readable versions of interface ids, sourced from SNMP.
Enriches passing flows with human-readable versions of interface ids, sourced from SNMP.
output
csv
Package csv processes all flows from it's In channel and converts them into CSV format.
Package csv processes all flows from it's In channel and converts them into CSV format.
json
Prints all flows to stdout or a given file in json format, for consumption by the stdin segment or for debugging.
Prints all flows to stdout or a given file in json format, for consumption by the stdin segment or for debugging.
kafkaproducer
Produces all received flows to Kafka instance.
Produces all received flows to Kafka instance.
lumberjack
Send passing flows to one or more lumberjack (Elastic Beats) servers.
Send passing flows to one or more lumberjack (Elastic Beats) servers.
sqlite
Dumps all incoming flow messages to a local sqlite database.
Dumps all incoming flow messages to a local sqlite database.
Serves as a template for new segments and forwards flows, otherwise does nothing.
Serves as a template for new segments and forwards flows, otherwise does nothing.
print
count
Counts the number of passing flows and prints the result on termination.
Counts the number of passing flows and prints the result on termination.
printdots
Prints a dot every n flows.
Prints a dot every n flows.
printflowdump
Prints all incoming flows in a specific flowdump format.
Prints all incoming flows in a specific flowdump format.
testing

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL