filter

type Any ¶ added in v0.4.5

type Any struct {
	core.SimpleFilter `gollumdoc:"embed_type"`
	// contains filtered or unexported fields
}

Any filter plugin

This plugin takes a list of filters and applies each of them to incoming messages until a an accepting filter is found. If any of the listed filters accept the message, it is passed through, otherwise, the message is dropper.

Parameters ¶

- AnyFilters: Defines a list of filters that should be checked before filtering a message. Filters are checked in order, and if the message passes then no further filters are checked.

Examples ¶

This example will accept valid json or messages from "exceptionStream":

ExampleConsumer:
  Type: consumer.Console
  Streams: "*"
  Modulators:
    - filter.Any:
      AnyFilters:
        - filter.JSON
        - filter.Stream:
          Only: exceptionStream

func (filter *Any) ApplyFilter(msg *core.Message) (core.FilterResult, error)

func (filter *Any) Configure(conf core.PluginConfigReader)

type JSON ¶

type JSON struct {
	core.SimpleFilter `gollumdoc:"embed_type"`
	// contains filtered or unexported fields
}

JSON filter

This filter inspects fields in JSON encoded datasets and accepts or rejects messages based on their contents.

Parameters ¶

- Reject: Defines a list of field names and regular expressions. Messages are rejected if the specified field's contents matches the given regular expression. Reject is checked before Accept. Field paths can be defined in the format accepted by tgo.MarshalMap.Path. By default this parameter is set to an empty list.

- Accept: Defines a list of field names and regular expressions. Messages are accepted if the specified field's contents matches the given regular expression. Accept is checked after Reject. Field paths can be defined in the format accepted by tgo.MarshalMap.Path. By default this parameter is set to an empty list.

- ApplyTo: Defines which part of the message the filter is applied to. When set to "", this filter is applied to the message's payload. All other values denotes a metadata key. By default this parameter is set to "".

Examples

ExampleConsumer:
  Type: consumer.Console
  Streams: console
  Modulators:
    - filter.JSON:
      Reject:
        type: ^log\.
      Accept:
        source: ^www\d+\.
        data/active: true

func (filter *JSON) ApplyFilter(msg *core.Message) (core.FilterResult, error)

func (filter *JSON) Configure(conf core.PluginConfigReader)

type None ¶

type None struct {
	core.SimpleFilter `gollumdoc:"embed_type"`
}

None filter plugin

This filter blocks all messages.

Examples ¶

This example starts a Console consumer and blocks all incoming messages:

exampleConsumer:
  Type: consumer.Console
  Streams: console
  Modulators:
    - filter.None

func (filter *None) ApplyFilter(msg *core.Message) (core.FilterResult, error)

func (filter *None) Configure(conf core.PluginConfigReader)

type Rate ¶ added in v0.4.3

type Rate struct {
	core.SimpleFilter `gollumdoc:"embed_type"`
	// contains filtered or unexported fields
}

Rate filter plugin

This plugin blocks messages after a certain number of messages per second has been reached.

Parameters ¶

- MessagesPerSec: This value defines the maximum number of messages per second allowed to pass through this filter. By default this parameter is set to "100".

- Ignore: Defines a list of streams that should not be affected by rate limiting. This is useful for e.g. producers listeing to "*". By default this parameter is set to "empty".

Examples ¶

This example accept ~10 messages in a second except the "noLimit" stream:

ExampleConsumer:
  Type: consumer.Console
  Streams: "*"
  Modulators:
    - filter.Rate:
      MessagesPerSec: 10
      Ignore:
        - noLimit

func (filter *Rate) ApplyFilter(msg *core.Message) (core.FilterResult, error)

func (filter *Rate) Configure(conf core.PluginConfigReader)

type RegExp ¶

type RegExp struct {
	core.SimpleFilter `gollumdoc:"embed_type"`
	// contains filtered or unexported fields
}

RegExp filter

This filter rejects or accepts messages based on regular expressions.

Parameters ¶

- Expression: Messages matching this expression are passed on. This parameter is ignored when set to "". Expression is checked after ExpressionNot. By default this parameter is set to "".

- ExpressionNot: Messages *not* matching this expression are passed on. This parameter is ignored when set to "". ExpressionNot is checked before Expression. By default this parameter is set to "".

- ApplyTo: Defines which part of the message the filter is applied to. When set to "", this filter is applied to the message's payload. All other values denotes a metadata key. By default this parameter is set to "".

Examples ¶

This example accepts only accesslog entries with a return status of 2xx or 3xx not originated from staging systems.

ExampleConsumer:
  Type: consumer.Console
  Streams: console
  Modulators:
    - filter.RegExp:
      ExpressionNot: " stage\\."
      Expression: "HTTP/1\\.1\\\" [23]\\d\\d"

func (filter *RegExp) ApplyFilter(msg *core.Message) (core.FilterResult, error)

func (filter *RegExp) Configure(conf core.PluginConfigReader)

type Sample ¶ added in v0.4.5

type Sample struct {
	core.SimpleFilter
	// contains filtered or unexported fields
}

Sample filter plugin

This plugin can be used to get n out of m messages (downsample). This allows you to reduce the amount of messages; the plugin starts blocking after a certain number of messages has been reached.

Parameters ¶

- SampleRatePerGroup: This value defines how many messages are passed through the filter in each group. By default this parameter is set to "1".

- SampleGroupSize: This value defines how many messages make up a group. Messages over SampleRatePerGroup within a group are filtered. By default this parameter is set to "2".

- SampleRateIgnore: This value defines a list of streams that should not be affected by sampling. This is useful for e.g. producers listening to "*". By default this parameter is set to an empty list.

Examples ¶

This example will block 8 from 10 messages:

exampleConsumer:
  Type: consumer.Console
  Streams: "*"
  Modulators:
    - filter.Sample:
      SampleRatePerGroup: 2
      SampleGroupSize: 10
      SampleIgnore:
        - foo
        - bar

func (filter *Sample) ApplyFilter(msg *core.Message) (core.FilterResult, error)

func (filter *Sample) Configure(conf core.PluginConfigReader)

type Stream ¶ added in v0.4.0

type Stream struct {
	core.SimpleFilter `gollumdoc:"embed_type"`
	// contains filtered or unexported fields
}

Stream filter plugin

The "Stream" filter filters messages by applying black and white lists to the the messages' streams' names.

The blacklist is applied first; messages not rejected by the blacklist are checked against the whitelist. An empty white list matches all streams.

Parameters ¶

- Block: Defines a list of stream names that are blocked. If a message's stream is not in that list, the "Only" list is tested. By default this parameter is empty.

- Only: Defines a list of streams that may pass. Messages from streams that are not in this list are blocked unless the list is empty. By default this parameter is empty.

Examples ¶

This example accepts ALL messages except ones from stream "foo":

ExampleConsumer:
  Type: consumer.Console
  Streams: "*"
  Modulators:
    - filter.Stream:
        Block:
          - foo

This example only accepts messages from stream "foo":

ExampleConsumer:
  Type: consumer.Console
  Streams: "*"
  Modulators:
    - filter.Stream:
        Only:
          - foo

func (filter *Stream) ApplyFilter(msg *core.Message) (core.FilterResult, error)

func (filter *Stream) Configure(conf core.PluginConfigReader)