beats: Index | Files

package beat

import ""


Package Files

beat.go error.go event.go info.go pipeline.go


const FlagField = "log.flags"

FlagField fields used to keep information or errors when events are parsed.


var GracefulExit = errors.New("graceful exit")

GracefulExit is an error that signals to exit with a code of 0.

type Beat Uses

type Beat struct {
    Info      Info     // beat metadata.
    Publisher Pipeline // Publisher pipeline

    InSetupCmd bool // this is set to true when the `setup` command is called

    OverwritePipelinesCallback OverwritePipelinesCallback // ingest pipeline loader callback
    // XXX: remove Config from public interface.
    //      It's currently used by filebeat modules to setup the Ingest Node
    //      pipeline and ML jobs.
    Config *BeatConfig // Common Beat configuration data.

    BeatConfig *common.Config // The beat's own configuration section

    Fields []byte // Data from fields.yml

    ConfigManager management.ConfigManager // config manager

Beat contains the basic beat data and the publisher client used to publish events.

type BeatConfig Uses

type BeatConfig struct {
    // output/publishing related configurations
    Output common.ConfigNamespace `config:"output"`

BeatConfig struct contains the basic configuration of every beat

type Beater Uses

type Beater interface {
    // The main event loop. This method should block until signalled to stop by an
    // invocation of the Stop() method.
    Run(b *Beat) error

    // Stop is invoked to signal that the Run method should finish its execution.
    // It will be invoked at most once.

Beater is the interface that must be implemented by every Beat. A Beater provides the main Run-loop and a Stop method to break the Run-loop. Instantiation and Configuration is normally provided by a Beat-`Creator`.

Once the beat is fully configured, the Run() method is invoked. The Run()-method implements the beat its run-loop. Once the Run()-method returns, the beat shuts down.

The Stop() method is invoked the first time (and only the first time) a shutdown signal is received. The Stop()-method normally will stop the Run()-loop, such that the beat can gracefully shutdown.

type Client Uses

type Client interface {
    Close() error

Client holds a connection to the beats publisher pipeline

type ClientConfig Uses

type ClientConfig struct {
    PublishMode PublishMode

    Processing ProcessingConfig

    CloseRef CloseRef

    // WaitClose sets the maximum duration to wait on ACK, if client still has events
    // active non-acknowledged events in the publisher pipeline.
    // WaitClose is only effective if one of ACKCount, ACKEvents and ACKLastEvents
    // is configured
    WaitClose time.Duration

    // Events configures callbacks for common client callbacks
    Events ClientEventer

    // ACKCount reports the number of published events recently acknowledged
    // by the pipeline.
    ACKCount func(int)

    // ACKEvents reports the events private data of recently acknowledged events.
    // Note: The slice passed must be copied if the events are to be processed
    //       after the handler returns.
    ACKEvents func([]interface{})

    // ACKLastEvent reports the last ACKed event out of a batch of ACKed events only.
    // Only the events 'Private' field will be reported.
    ACKLastEvent func(interface{})

ClientConfig defines common configuration options one can pass to Pipeline.ConnectWith to control the clients behavior and provide ACK support.

type ClientEventer Uses

type ClientEventer interface {
    Closing() // Closing indicates the client is being shutdown next
    Closed()  // Closed indicates the client being fully shutdown

    Published()             // event has been successfully forwarded to the publisher pipeline
    FilteredOut(Event)      // event has been filtered out/dropped by processors
    DroppedOnPublish(Event) // event has been dropped, while waiting for the queue

ClientEventer provides access to internal client events.

type CloseRef Uses

type CloseRef interface {
    Done() <-chan struct{}
    Err() error

CloseRef allows users to close the client asynchronously. A CloseReg implements a subset of function required for context.Context.

type Creator Uses

type Creator func(*Beat, *common.Config) (Beater, error)

Creator initializes and configures a new Beater instance used to execute the beat's run-loop.

type Event Uses

type Event struct {
    Timestamp  time.Time
    Meta       common.MapStr
    Fields     common.MapStr
    Private    interface{} // for beats private use
    TimeSeries bool        // true if the event contains timeseries data

Event is the common event format shared by all beats. Every event must have a timestamp and provide encodable Fields in `Fields`. The `Meta`-fields can be used to pass additional meta-data to the outputs. Output can optionally publish a subset of Meta, or ignore Meta.

func (*Event) Delete Uses

func (e *Event) Delete(key string) error

func (*Event) GetValue Uses

func (e *Event) GetValue(key string) (interface{}, error)

func (*Event) PutValue Uses

func (e *Event) PutValue(key string, v interface{}) (interface{}, error)

func (*Event) SetErrorWithOption Uses

func (e *Event) SetErrorWithOption(jsonErr common.MapStr, addErrKey bool)

SetErrorWithOption sets jsonErr value in the event fields according to addErrKey value.

func (*Event) SetID Uses

func (e *Event) SetID(id string)

SetID overwrites the "id" field in the events metadata. If Meta is nil, a new Meta dictionary is created.

type Info Uses

type Info struct {
    Beat        string    // The actual beat's name
    IndexPrefix string    // The beat's index prefix in Elasticsearch.
    Version     string    // The beat version. Defaults to the libbeat version when an implementation does not set a version
    Name        string    // configured beat name
    Hostname    string    // hostname
    ID          uuid.UUID // ID assigned to beat machine
    EphemeralID uuid.UUID // ID assigned to beat process invocation (PID)

    // Monitoring-related fields
    Monitoring struct {
        DefaultUsername string // The default username to be used to connect to Elasticsearch Monitoring

Info stores a beats instance meta data.

type OverwritePipelinesCallback Uses

type OverwritePipelinesCallback func(*common.Config) error

OverwritePipelinesCallback can be used by the Beat to register Ingest pipeline loader for the enabled modules.

type Pipeline Uses

type Pipeline interface {
    SetACKHandler(PipelineACKHandler) error

type PipelineACKHandler Uses

type PipelineACKHandler struct {
    // ACKCount reports the number of published events recently acknowledged
    // by the pipeline.
    ACKCount func(int)

    // ACKEvents reports the events recently acknowledged by the pipeline.
    // Only the events 'Private' field will be reported.
    ACKEvents func([]interface{})

    // ACKLastEvent reports the last ACKed event per pipeline client.
    // Only the events 'Private' field will be reported.
    ACKLastEvents func([]interface{})

PipelineACKHandler configures some pipeline-wide event ACK handler.

type PipelineConnector Uses

type PipelineConnector interface {
    ConnectWith(ClientConfig) (Client, error)
    Connect() (Client, error)

PipelineConnector creates a publishing Client. This is typically backed by a Pipeline.

type ProcessingConfig Uses

type ProcessingConfig struct {
    // EventMetadata configures additional fields/tags to be added to published events.
    EventMetadata common.EventMetadata

    // Meta provides additional meta data to be added to the Meta field in the beat.Event
    // structure.
    Meta common.MapStr

    // Fields provides additional 'global' fields to be added to every event
    Fields common.MapStr

    // DynamicFields provides additional fields to be added to every event, supporting live updates
    DynamicFields *common.MapStrPointer

    // Processors passes additional processor to the client, to be executed before
    // the pipeline processors.
    Processor ProcessorList

    // KeepNull determines whether published events will keep null values or omit them.
    KeepNull bool

    // Private contains additional information to be passed to the processing
    // pipeline builder.
    Private interface{}

ProcessingConfig provides additional event processing settings a client can pass to the publisher pipeline on Connect.

type Processor Uses

type Processor interface {
    String() string // print full processor description
    Run(in *Event) (event *Event, err error)

Processor defines the minimal required interface for processor, that can be registered with the publisher pipeline.

type ProcessorList Uses

type ProcessorList interface {
    All() []Processor

type PublishMode Uses

type PublishMode uint8

PublishMode enum sets some requirements on the client connection to the beats publisher pipeline

const (
    // DefaultGuarantees are up to the pipeline configuration itself.
    DefaultGuarantees PublishMode = iota

    // OutputChooses mode fully depends on the output and its configuration.
    // Events might be dropped based on the users output configuration.
    // In this mode no events are dropped within the pipeline. Events are only removed
    // after the output has ACKed the events to the pipeline, even if the output
    // did drop the events.

    // GuaranteedSend ensures events are retried until acknowledged by the output.
    // Normally guaranteed sending should be used with some client ACK-handling
    // to update state keeping track of the sending status.

    // DropIfFull drops an event to be send if the pipeline is currently full.
    // This ensures a beats internals can continue processing if the pipeline has
    // filled up. Useful if an event stream must be processed to keep internal
    // state up-to-date.

Package beat imports 6 packages (graph) and is imported by 3967 packages. Updated 2020-01-29. Refresh now. Tools for package owners.