log

package
v0.0.0-...-e951c9a Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 6, 2022 License: AGPL-3.0 Imports: 27 Imported by: 0

Documentation

Index

Constants

View Source
const (
	IPv4Charset = "0123456789."
	IPv6Charset = "0123456789abcdefABCDEF:."
)
View Source
const (
	ConvertBytes    = "bytes"
	ConvertDuration = "duration"
	ConvertFloat    = "float"
)
View Source
const MaxInternedStrings = 1024

Variables

View Source
var (
	ErrIPFilterInvalidPattern   = errors.New("ip: invalid pattern")
	ErrIPFilterInvalidOperation = errors.New("ip: invalid operation")
)
View Source
var EmptyLabelsResult = NewLabelsResult(labels.Labels{}, labels.Labels{}.Hash())
View Source
var (

	// NoopLabelFilter is a label filter that doesn't filter out any values.
	NoopLabelFilter = noopLabelFilter{}
)
View Source
var TrueFilter = trueFilter{}

TrueFilter is a filter that returns and matches all log lines whatever their content.

Functions

func IsNoopPipeline

func IsNoopPipeline(p Pipeline) bool

IsNoopPipeline tells if a pipeline is a Noop.

Types

type AnalyzablePipeline

type AnalyzablePipeline interface {
	Pipeline
	Stages() []Stage
	LabelsBuilder() *BaseLabelsBuilder
}

type BaseLabelsBuilder

type BaseLabelsBuilder struct {
	// contains filtered or unexported fields
}

BaseLabelsBuilder is a label builder used by pipeline and stages. Only one base builder is used and it contains cache for each LabelsBuilders.

func NewBaseLabelsBuilder

func NewBaseLabelsBuilder() *BaseLabelsBuilder

NewLabelsBuilder creates a new base labels builder.

func NewBaseLabelsBuilderWithGrouping

func NewBaseLabelsBuilderWithGrouping(groups []string, parserKeyHints ParserHint, without, noLabels bool) *BaseLabelsBuilder

NewBaseLabelsBuilderWithGrouping creates a new base labels builder with grouping to compute results.

func (*BaseLabelsBuilder) ForLabels

func (b *BaseLabelsBuilder) ForLabels(lbs labels.Labels, hash uint64) *LabelsBuilder

ForLabels creates a labels builder for a given labels set as base. The labels cache is shared across all created LabelsBuilders.

func (BaseLabelsBuilder) Hash

func (h BaseLabelsBuilder) Hash(lbs labels.Labels) uint64

Hash hashes the labels

func (*BaseLabelsBuilder) ParserLabelHints

func (b *BaseLabelsBuilder) ParserLabelHints() ParserHint

ParserLabelHints returns a limited list of expected labels to extract for metric queries. Returns nil when it's impossible to hint labels extractions.

type BinaryLabelFilter

type BinaryLabelFilter struct {
	Left  LabelFilterer
	Right LabelFilterer
	// contains filtered or unexported fields
}

func NewAndLabelFilter

func NewAndLabelFilter(left LabelFilterer, right LabelFilterer) *BinaryLabelFilter

NewAndLabelFilter creates a new LabelFilterer from a and binary operation of two LabelFilterer.

func NewOrLabelFilter

func NewOrLabelFilter(left LabelFilterer, right LabelFilterer) *BinaryLabelFilter

NewOrLabelFilter creates a new LabelFilterer from a or binary operation of two LabelFilterer.

func (*BinaryLabelFilter) Process

func (b *BinaryLabelFilter) Process(ts int64, line []byte, lbs *LabelsBuilder) ([]byte, bool)

func (*BinaryLabelFilter) RequiredLabelNames

func (b *BinaryLabelFilter) RequiredLabelNames() []string

func (*BinaryLabelFilter) String

func (b *BinaryLabelFilter) String() string

type BytesLabelFilter

type BytesLabelFilter struct {
	Name  string
	Value uint64
	Type  LabelFilterType
}

func NewBytesLabelFilter

func NewBytesLabelFilter(t LabelFilterType, name string, b uint64) *BytesLabelFilter

NewBytesLabelFilter creates a new label filterer which parses bytes string representation (1KB) from the value of the named label and compares it with the given b value.

func (*BytesLabelFilter) Process

func (d *BytesLabelFilter) Process(_ int64, line []byte, lbs *LabelsBuilder) ([]byte, bool)

func (*BytesLabelFilter) RequiredLabelNames

func (d *BytesLabelFilter) RequiredLabelNames() []string

func (*BytesLabelFilter) String

func (d *BytesLabelFilter) String() string

type DurationLabelFilter

type DurationLabelFilter struct {
	Name  string
	Value time.Duration
	Type  LabelFilterType
}

func NewDurationLabelFilter

func NewDurationLabelFilter(t LabelFilterType, name string, d time.Duration) *DurationLabelFilter

NewDurationLabelFilter creates a new label filterer which parses duration string representation (5s) from the value of the named label and compares it with the given d value.

func (*DurationLabelFilter) Process

func (d *DurationLabelFilter) Process(_ int64, line []byte, lbs *LabelsBuilder) ([]byte, bool)

func (*DurationLabelFilter) RequiredLabelNames

func (d *DurationLabelFilter) RequiredLabelNames() []string

func (*DurationLabelFilter) String

func (d *DurationLabelFilter) String() string

type Filterer

type Filterer interface {
	Filter(line []byte) bool
	ToStage() Stage
}

Filterer is a interface to filter log lines.

func NewAndFilter

func NewAndFilter(left Filterer, right Filterer) Filterer

NewAndFilter creates a new filter which matches only if left and right matches.

func NewAndFilters

func NewAndFilters(filters []Filterer) Filterer

NewAndFilters creates a new filter which matches only if all filters match

func NewFilter

func NewFilter(match string, mt labels.MatchType) (Filterer, error)

NewFilter creates a new line filter from a match string and type.

type FiltererFunc

type FiltererFunc func(line []byte) bool

LineFilterFunc is a syntax sugar for creating line filter from a function

func (FiltererFunc) Filter

func (f FiltererFunc) Filter(line []byte) bool

type IPLabelFilter

type IPLabelFilter struct {
	// contains filtered or unexported fields
}

func NewIPLabelFilter

func NewIPLabelFilter(pattern string, label string, ty LabelFilterType) *IPLabelFilter

NewIPLabelFilter is used to construct ip filter as label filter for the given `label`.

func (*IPLabelFilter) PatternError

func (f *IPLabelFilter) PatternError() error

PatternError will be used `labelFilter.Stage()` method so that, if the given pattern is wrong it returns proper 400 error to the client of LogQL.

func (*IPLabelFilter) Process

func (f *IPLabelFilter) Process(_ int64, line []byte, lbs *LabelsBuilder) ([]byte, bool)

`Process` implements `Stage` interface

func (*IPLabelFilter) RequiredLabelNames

func (f *IPLabelFilter) RequiredLabelNames() []string

`RequiredLabelNames` implements `Stage` interface

func (*IPLabelFilter) String

func (f *IPLabelFilter) String() string

`String` implements fmt.Stringer inteface, by which also implements `LabelFilterer` inteface.

type IPLineFilter

type IPLineFilter struct {
	// contains filtered or unexported fields
}

func NewIPLineFilter

func NewIPLineFilter(pattern string, ty labels.MatchType) (*IPLineFilter, error)

NewIPLineFilter is used to construct ip filter as a `LineFilter`

func (*IPLineFilter) Filter

func (f *IPLineFilter) Filter(line []byte) bool

Filter implement `Filterer` interface. Used by `LineFilter`

func (*IPLineFilter) Process

func (f *IPLineFilter) Process(_ int64, line []byte, _ *LabelsBuilder) ([]byte, bool)

`Process` implements `Stage` interface

func (*IPLineFilter) RequiredLabelNames

func (f *IPLineFilter) RequiredLabelNames() []string

`RequiredLabelNames` implements `Stage` interface

func (*IPLineFilter) ToStage

func (f *IPLineFilter) ToStage() Stage

ToStage implements `Filterer` interface.

type IPMatchType

type IPMatchType int

type IPMatcher

type IPMatcher interface{}

Should be one of the netaddr.IP, netaddr.IPRange, netadd.IPPrefix.

type JSONExpression

type JSONExpression struct {
	Identifier string
	Expression string
}

func NewJSONExpr

func NewJSONExpr(identifier, expression string) JSONExpression

type JSONExpressionParser

type JSONExpressionParser struct {
	// contains filtered or unexported fields
}

func NewJSONExpressionParser

func NewJSONExpressionParser(expressions []JSONExpression) (*JSONExpressionParser, error)

func (*JSONExpressionParser) Process

func (j *JSONExpressionParser) Process(_ int64, line []byte, lbs *LabelsBuilder) ([]byte, bool)

func (*JSONExpressionParser) RequiredLabelNames

func (j *JSONExpressionParser) RequiredLabelNames() []string

type JSONParser

type JSONParser struct {
	// contains filtered or unexported fields
}

func NewJSONParser

func NewJSONParser() *JSONParser

NewJSONParser creates a log stage that can parse a json log line and add properties as labels.

func (*JSONParser) Process

func (j *JSONParser) Process(_ int64, line []byte, lbs *LabelsBuilder) ([]byte, bool)

func (*JSONParser) RequiredLabelNames

func (j *JSONParser) RequiredLabelNames() []string

type LabelFilterType

type LabelFilterType int

LabelFilterType is an enum for label filtering types.

const (
	LabelFilterEqual LabelFilterType = iota
	LabelFilterNotEqual
	LabelFilterGreaterThan
	LabelFilterGreaterThanOrEqual
	LabelFilterLesserThan
	LabelFilterLesserThanOrEqual
)

Possible LabelFilterType.

func (LabelFilterType) String

func (f LabelFilterType) String() string

type LabelFilterer

type LabelFilterer interface {
	Stage
	fmt.Stringer
}

LabelFilterer can filter extracted labels.

func ReduceAndLabelFilter

func ReduceAndLabelFilter(filters []LabelFilterer) LabelFilterer

ReduceAndLabelFilter Reduces multiple label filterer into one using binary and operation.

type LabelFmt

type LabelFmt struct {
	Name  string
	Value string

	Rename bool
}

LabelFmt is a configuration struct for formatting a label.

func NewRenameLabelFmt

func NewRenameLabelFmt(dst, target string) LabelFmt

NewRenameLabelFmt creates a configuration to rename a label.

func NewTemplateLabelFmt

func NewTemplateLabelFmt(dst, template string) LabelFmt

NewTemplateLabelFmt creates a configuration to format a label using text template.

type LabelsBuilder

type LabelsBuilder struct {
	*BaseLabelsBuilder
	// contains filtered or unexported fields
}

LabelsBuilder is the same as labels.Builder but tailored for this package.

func (*LabelsBuilder) BaseHas

func (b *LabelsBuilder) BaseHas(key string) bool

BaseHas returns the base labels have the given key

func (*LabelsBuilder) Del

func (b *LabelsBuilder) Del(ns ...string) *LabelsBuilder

Del deletes the label of the given name.

func (*LabelsBuilder) Get

func (b *LabelsBuilder) Get(key string) (string, bool)

Get returns the value of a labels key if it exists.

func (*LabelsBuilder) GetErr

func (b *LabelsBuilder) GetErr() string

GetErr return the current error label value.

func (*LabelsBuilder) GetErrorDetails

func (b *LabelsBuilder) GetErrorDetails() string

func (*LabelsBuilder) GroupedLabels

func (b *LabelsBuilder) GroupedLabels() LabelsResult

GroupedLabels returns the LabelsResult from the builder. Groups are applied and the cache is used when possible.

func (*LabelsBuilder) HasErr

func (b *LabelsBuilder) HasErr() bool

HasErr tells if the error label has been set.

func (*LabelsBuilder) HasErrorDetails

func (b *LabelsBuilder) HasErrorDetails() bool

func (LabelsBuilder) Hash

func (h LabelsBuilder) Hash(lbs labels.Labels) uint64

Hash hashes the labels

func (*LabelsBuilder) LabelsResult

func (b *LabelsBuilder) LabelsResult() LabelsResult

LabelsResult returns the LabelsResult from the builder. No grouping is applied and the cache is used when possible.

func (*LabelsBuilder) Map

func (b *LabelsBuilder) Map() map[string]string

func (*LabelsBuilder) Reset

func (b *LabelsBuilder) Reset()

Reset clears all current state for the builder.

func (*LabelsBuilder) Set

func (b *LabelsBuilder) Set(n, v string) *LabelsBuilder

Set the name/value pair as a label.

func (*LabelsBuilder) SetErr

func (b *LabelsBuilder) SetErr(err string) *LabelsBuilder

SetErr sets the error label.

func (*LabelsBuilder) SetErrorDetails

func (b *LabelsBuilder) SetErrorDetails(desc string) *LabelsBuilder

func (*LabelsBuilder) UnsortedLabels

func (b *LabelsBuilder) UnsortedLabels(buf labels.Labels) labels.Labels

type LabelsFormatter

type LabelsFormatter struct {
	// contains filtered or unexported fields
}

func NewLabelsFormatter

func NewLabelsFormatter(fmts []LabelFmt) (*LabelsFormatter, error)

NewLabelsFormatter creates a new formatter that can format multiple labels at once. Either by renaming or using text template. It is not allowed to reformat the same label twice within the same formatter.

func (*LabelsFormatter) Process

func (lf *LabelsFormatter) Process(ts int64, l []byte, lbs *LabelsBuilder) ([]byte, bool)

func (*LabelsFormatter) RequiredLabelNames

func (lf *LabelsFormatter) RequiredLabelNames() []string

type LabelsResult

type LabelsResult interface {
	String() string
	Labels() labels.Labels
	Hash() uint64
}

LabelsResult is a computed labels result that contains the labels set with associated string and hash. The is mainly used for caching and returning labels computations out of pipelines and stages.

func NewLabelsResult

func NewLabelsResult(lbs labels.Labels, hash uint64) LabelsResult

NewLabelsResult creates a new LabelsResult from a labels set and a hash.

type LineExtractor

type LineExtractor func([]byte) float64

LineExtractor extracts a float64 from a log line.

var (
	CountExtractor LineExtractor = func(line []byte) float64 { return 1. }
	BytesExtractor LineExtractor = func(line []byte) float64 { return float64(len(line)) }
)

type LineFormatter

type LineFormatter struct {
	*template.Template
	// contains filtered or unexported fields
}

func NewFormatter

func NewFormatter(tmpl string) (*LineFormatter, error)

NewFormatter creates a new log line formatter from a given text template.

func (*LineFormatter) Process

func (lf *LineFormatter) Process(ts int64, line []byte, lbs *LabelsBuilder) ([]byte, bool)

func (*LineFormatter) RequiredLabelNames

func (lf *LineFormatter) RequiredLabelNames() []string

type LogfmtParser

type LogfmtParser struct {
	// contains filtered or unexported fields
}

func NewLogfmtParser

func NewLogfmtParser() *LogfmtParser

NewLogfmtParser creates a parser that can extract labels from a logfmt log line. Each keyval is extracted into a respective label.

func (*LogfmtParser) Process

func (l *LogfmtParser) Process(_ int64, line []byte, lbs *LabelsBuilder) ([]byte, bool)

func (*LogfmtParser) RequiredLabelNames

func (l *LogfmtParser) RequiredLabelNames() []string

type NumericLabelFilter

type NumericLabelFilter struct {
	Name  string
	Value float64
	Type  LabelFilterType
}

func NewNumericLabelFilter

func NewNumericLabelFilter(t LabelFilterType, name string, v float64) *NumericLabelFilter

NewNumericLabelFilter creates a new label filterer which parses float64 string representation (5.2) from the value of the named label and compares it with the given f value.

func (*NumericLabelFilter) Process

func (n *NumericLabelFilter) Process(_ int64, line []byte, lbs *LabelsBuilder) ([]byte, bool)

func (*NumericLabelFilter) RequiredLabelNames

func (n *NumericLabelFilter) RequiredLabelNames() []string

func (*NumericLabelFilter) String

func (n *NumericLabelFilter) String() string

type ParserHint

type ParserHint interface {
	// Tells if a label with the given key should be extracted.
	ShouldExtract(key string) bool
	// Tells if there's any hint that start with the given prefix.
	// This allows to speed up key searching in nested structured like json.
	ShouldExtractPrefix(prefix string) bool
	// Tells if we should not extract any labels.
	// For example in :
	//		 sum(rate({app="foo"} | json [5m]))
	// We don't need to extract any labels from the log line.
	NoLabels() bool
}

ParserHint are hints given to LogQL parsers. This is specially useful for parser that extract implicitly all possible label keys. This is used only within metric queries since it's rare that you need all label keys. For example in the following expression:

sum by (status_code) (rate({app="foo"} | json [5m]))

All we need to extract is the status_code in the json parser.

type PatternParser

type PatternParser struct {
	// contains filtered or unexported fields
}

func NewPatternParser

func NewPatternParser(pn string) (*PatternParser, error)

func (*PatternParser) Process

func (l *PatternParser) Process(_ int64, line []byte, lbs *LabelsBuilder) ([]byte, bool)

func (*PatternParser) RequiredLabelNames

func (l *PatternParser) RequiredLabelNames() []string

type Pipeline

type Pipeline interface {
	ForStream(labels labels.Labels) StreamPipeline
}

Pipeline can create pipelines for each log stream.

func NewFilteringPipeline

func NewFilteringPipeline(f []PipelineFilter, p Pipeline) Pipeline

NewFilteringPipeline creates a pipeline where entries from the underlying log stream are filtered by pipeline filters before being passed to the pipeline representing the queried data. Filters are always upstream of the pipeline

func NewNoopPipeline

func NewNoopPipeline() Pipeline

NewNoopPipeline creates a pipelines that does not process anything and returns log streams as is.

func NewPipeline

func NewPipeline(stages []Stage) Pipeline

NewPipeline creates a new pipeline for a given set of stages.

type PipelineFilter

type PipelineFilter struct {
	Start    int64
	End      int64
	Matchers []*labels.Matcher
	Pipeline Pipeline
}

PipelineFilter contains a set of matchers and a pipeline that, when matched, causes an entry from a log stream to be skipped. Matching entries must also fall between 'start' and 'end', inclusive

type RegexpParser

type RegexpParser struct {
	// contains filtered or unexported fields
}

func NewRegexpParser

func NewRegexpParser(re string) (*RegexpParser, error)

NewRegexpParser creates a new log stage that can extract labels from a log line using a regex expression. The regex expression must contains at least one named match. If the regex doesn't match the line is not filtered out.

func (*RegexpParser) Process

func (r *RegexpParser) Process(_ int64, line []byte, lbs *LabelsBuilder) ([]byte, bool)

func (*RegexpParser) RequiredLabelNames

func (r *RegexpParser) RequiredLabelNames() []string

type SampleExtractor

type SampleExtractor interface {
	ForStream(labels labels.Labels) StreamSampleExtractor
}

SampleExtractor creates StreamSampleExtractor that can extract samples for a given log stream.

func LabelExtractorWithStages

func LabelExtractorWithStages(
	labelName, conversion string,
	groups []string, without, noLabels bool,
	preStages []Stage,
	postFilter Stage,
) (SampleExtractor, error)

LabelExtractorWithStages creates a SampleExtractor that will extract metrics from a labels. A set of log stage is executed before the conversion. A Filtering stage is executed after the conversion allowing to remove sample containing the __error__ label.

func NewFilteringSampleExtractor

func NewFilteringSampleExtractor(f []PipelineFilter, e SampleExtractor) SampleExtractor

NewFilteringSampleExtractor creates a sample extractor where entries from the underlying log stream are filtered by pipeline filters before being passed to extract samples. Filters are always upstream of the extractor.

func NewLineSampleExtractor

func NewLineSampleExtractor(ex LineExtractor, stages []Stage, groups []string, without, noLabels bool) (SampleExtractor, error)

NewLineSampleExtractor creates a SampleExtractor from a LineExtractor. Multiple log stages are run before converting the log line.

type Stage

type Stage interface {
	Process(ts int64, line []byte, lbs *LabelsBuilder) ([]byte, bool)
	RequiredLabelNames() []string
}

Stage is a single step of a Pipeline. A Stage implementation should never mutate the line passed, but instead either return the line unchanged or allocate a new line.

var NoopStage Stage = &noopStage{}

NoopStage is a stage that doesn't process a log line.

func ReduceStages

func ReduceStages(stages []Stage) Stage

ReduceStages reduces multiple stages into one.

type StageFunc

type StageFunc struct {
	// contains filtered or unexported fields
}

func (StageFunc) Process

func (fn StageFunc) Process(ts int64, line []byte, lbs *LabelsBuilder) ([]byte, bool)

func (StageFunc) RequiredLabelNames

func (fn StageFunc) RequiredLabelNames() []string

type StreamPipeline

type StreamPipeline interface {
	BaseLabels() LabelsResult
	// Process processes a log line and returns the transformed line and the labels.
	// The buffer returned for the log line can be reused on subsequent calls to Process and therefore must be copied.
	Process(ts int64, line []byte) (resultLine []byte, resultLabels LabelsResult, matches bool)
	ProcessString(ts int64, line string) (resultLine string, resultLabels LabelsResult, matches bool)
}

StreamPipeline transform and filter log lines and labels. A StreamPipeline never mutate the received line.

func NewStreamPipeline

func NewStreamPipeline(stages []Stage, labelsBuilder *LabelsBuilder) StreamPipeline

type StreamSampleExtractor

type StreamSampleExtractor interface {
	BaseLabels() LabelsResult
	Process(ts int64, line []byte) (float64, LabelsResult, bool)
	ProcessString(ts int64, line string) (float64, LabelsResult, bool)
}

StreamSampleExtractor extracts sample for a log line. A StreamSampleExtractor never mutate the received line.

type StringLabelFilter

type StringLabelFilter struct {
	*labels.Matcher
}

func NewStringLabelFilter

func NewStringLabelFilter(m *labels.Matcher) *StringLabelFilter

NewStringLabelFilter creates a new label filterer which compares string label. This is the only LabelFilterer that can filter out the __error__ label. Unlike other LabelFilterer which apply conversion, if the label name doesn't exist it is compared with an empty value.

func (*StringLabelFilter) Process

func (s *StringLabelFilter) Process(_ int64, line []byte, lbs *LabelsBuilder) ([]byte, bool)

func (*StringLabelFilter) RequiredLabelNames

func (s *StringLabelFilter) RequiredLabelNames() []string

type UnpackParser

type UnpackParser struct {
	// contains filtered or unexported fields
}

func NewUnpackParser

func NewUnpackParser() *UnpackParser

NewUnpackParser creates a new unpack stage. The unpack stage will parse a json log line as map[string]string where each key will be translated into labels. A special key _entry will also be used to replace the original log line. This is to be used in conjunction with Promtail pack stage. see https://grafana.com/docs/loki/latest/clients/promtail/stages/pack/

func (*UnpackParser) Process

func (u *UnpackParser) Process(_ int64, line []byte, lbs *LabelsBuilder) ([]byte, bool)

func (UnpackParser) RequiredLabelNames

func (UnpackParser) RequiredLabelNames() []string

Directories

Path Synopsis
Adapted from https://github.com/go-logfmt/logfmt/ but []byte as parameter instead Original license is MIT.
Adapted from https://github.com/go-logfmt/logfmt/ but []byte as parameter instead Original license is MIT.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL