beats: Index | Files

package log

import ""

Package log harvests different inputs for new information. Currently two harvester types exist:

 * log
 * stdin

The log harvester reads a file line by line. In case the end of a file is found
with an incomplete line, the line pointer stays at the beginning of the incomplete
line. As soon as the line is completed, it is read and returned.

The stdin harvesters reads data from stdin.


Package Files

config.go file.go harvester.go input.go log.go stdin.go


const (
    ScanOrderAsc     = "asc"
    ScanOrderDesc    = "desc"
    ScanSortNone     = ""
    ScanSortModtime  = "modtime"
    ScanSortFilename = "filename"

Contains available scan options


var (
    ErrFileTruncate = errors.New("detected file being truncated")
    ErrRenamed      = errors.New("file was renamed")
    ErrRemoved      = errors.New("file was removed")
    ErrInactive     = errors.New("file inactive")
    ErrClosed       = errors.New("reader closed")
var ValidScanOrder = map[string]struct{}{
    ScanOrderAsc:  {},
    ScanOrderDesc: {},

ValidScanOrder of valid scan orders

var ValidScanSort = map[string]struct{}{
    ScanSortNone:     {},
    ScanSortModtime:  {},
    ScanSortFilename: {},

ValidScanOrder of valid scan orders

func NewInput Uses

func NewInput(
    cfg *common.Config,
    outlet channel.Connector,
    context input.Context,
) (input.Input, error)

NewInput instantiates a new Log

type File Uses

type File struct {

func (File) Continuable Uses

func (File) Continuable() bool

func (File) HasState Uses

func (File) HasState() bool

func (File) Removed Uses

func (f File) Removed() bool

type FileSortInfo Uses

type FileSortInfo struct {
    // contains filtered or unexported fields

type Harvester Uses

type Harvester struct {
    // contains filtered or unexported fields

Harvester contains all harvester related data

func NewHarvester Uses

func NewHarvester(
    config *common.Config,
    state file.State,
    states *file.States,
    publishState func(file.State) bool,
    outletFactory OutletFactory,
) (*Harvester, error)

NewHarvester creates a new harvester

func (*Harvester) ID Uses

func (h *Harvester) ID() uuid.UUID

ID returns the unique harvester identifier

func (*Harvester) Run Uses

func (h *Harvester) Run() error

Run start the harvester and reads files line by line and sends events to the defined output

func (*Harvester) SendStateUpdate Uses

func (h *Harvester) SendStateUpdate()

SendStateUpdate send an empty event with the current state to update the registry close_timeout does not apply here to make sure a harvester is closed properly. In case the output is blocked the harvester will stay open to make sure no new harvester is started. As soon as the output becomes available again, the finished state is written and processing can continue.

func (*Harvester) Setup Uses

func (h *Harvester) Setup() error

Setup opens the file handler and creates the reader for the harvester

func (*Harvester) Stop Uses

func (h *Harvester) Stop()

Stop stops harvester and waits for completion

type Input Uses

type Input struct {
    // contains filtered or unexported fields

Input contains the input and its config

func (*Input) Run Uses

func (p *Input) Run()

Run runs the input

func (*Input) Stop Uses

func (p *Input) Stop()

Stop stops all harvesters and then stops the input

func (*Input) Wait Uses

func (p *Input) Wait()

Wait waits for the all harvesters to complete and only then call stop

type Log Uses

type Log struct {
    // contains filtered or unexported fields

Log contains all log related data

func NewLog Uses

func NewLog(
    fs harvester.Source,
    config LogConfig,
) (*Log, error)

NewLog creates a new log instance to read log sources

func (*Log) Close Uses

func (f *Log) Close()

Close closes the done channel but no th the file handler

func (*Log) Read Uses

func (f *Log) Read(buf []byte) (int, error)

Read reads from the reader and updates the offset The total number of bytes read is returned.

type LogConfig Uses

type LogConfig struct {
    Backoff       time.Duration `config:"backoff" validate:"min=0,nonzero"`
    BackoffFactor int           `config:"backoff_factor" validate:"min=1"`
    MaxBackoff    time.Duration `config:"max_backoff" validate:"min=0,nonzero"`
    CloseInactive time.Duration `config:"close_inactive"`
    CloseRemoved  bool          `config:"close_removed"`
    CloseRenamed  bool          `config:"close_renamed"`
    CloseEOF      bool          `config:"close_eof"`
    CloseTimeout  time.Duration `config:"close_timeout" validate:"min=0"`

type OutletFactory Uses

type OutletFactory func() channel.Outleter

OutletFactory provides an outlet for the harvester

type Pipe Uses

type Pipe struct {
    File *os.File

restrict file to minimal interface of FileSource to prevent possible casts to additional interfaces supported by underlying file

func (Pipe) Close Uses

func (p Pipe) Close() error

func (Pipe) Continuable Uses

func (p Pipe) Continuable() bool

func (Pipe) HasState Uses

func (p Pipe) HasState() bool

func (Pipe) Name Uses

func (p Pipe) Name() string

func (Pipe) Read Uses

func (p Pipe) Read(b []byte) (int, error)

func (Pipe) Removed Uses

func (p Pipe) Removed() bool

func (Pipe) Stat Uses

func (p Pipe) Stat() (os.FileInfo, error)

Package log imports 31 packages (graph) and is imported by 100 packages. Updated 2020-02-07. Refresh now. Tools for package owners.