luci: go.chromium.org/luci/logdog/server/collector Index | Files | Directories

package collector

import "go.chromium.org/luci/logdog/server/collector"

Package collector implements the LogDog Collector daemon's log parsing and registration logic.

The LogDog Collector is responsible for ingesting logs from the Butler sent via transport, stashing them in intermediate storage, and registering them with the LogDog Coordinator service.

Index

Package Files

collector.go doc.go

Constants

const (
    // DefaultMaxMessageWorkers is the default number of concurrent worker
    // goroutones to employ for a single message.
    DefaultMaxMessageWorkers = 4
)

type Collector Uses

type Collector struct {
    // Coordinator is used to interface with the Coordinator client.
    //
    // On production systems, this should wrapped with a caching client (see
    // the stateCache sub-package) to avoid overwhelming the server.
    Coordinator coordinator.Coordinator

    // Storage is the intermediate storage instance to use.
    Storage storage.Storage

    // StreamStateCacheExpire is the maximum amount of time that a cached stream
    // state entry is valid. If zero, DefaultStreamStateCacheExpire will be used.
    StreamStateCacheExpire time.Duration

    // MaxMessageWorkers is the maximum number of concurrent workers to employ
    // for any given message. If <= 0, DefaultMaxMessageWorkers will be applied.
    MaxMessageWorkers int
}

Collector is a stateful object responsible for ingesting LogDog logs, registering them with a Coordinator, and stowing them in short-term storage for streaming and processing.

A Collector's Close should be called when finished to release any internal resources.

func (*Collector) Close Uses

func (c *Collector) Close()

Close releases any internal resources and blocks pending the completion of any outstanding operations. After Close, no new Process calls may be made.

func (*Collector) Process Uses

func (c *Collector) Process(ctx context.Context, msg []byte) error

Process ingests an encoded ButlerLogBundle message, registering it with the LogDog Coordinator and stowing it in a temporary Storage for streaming retrieval.

If a transient error occurs during ingest, Process will return an error. If no error occurred, or if there was an error with the input data, no error will be returned.

Directories

PathSynopsis
coordinatorPackage coordinator implements a minimal interface to the Coordinator service that is sufficient for Collector usage.

Package collector imports 17 packages (graph). Updated 2018-08-15. Refresh now. Tools for package owners.