Documentation ¶
Index ¶
- Constants
- Variables
- type Batch
- type Batcher
- type Config
- type Counter
- type GzipFormatter
- type HTTPFormatter
- type HTTPOutlet
- type KinesisFormatter
- type KinesisRecord
- type LogLine
- type LogLineReader
- type LogplexBatchFormatter
- type LogplexLineFormatter
- type NewHTTPFormatterFunc
- type Shuttle
- type SubFormatter
Constants ¶
const ( InputFormatRaw = iota InputFormatRFC3164 InputFormatRFC5424 )
Input format constants. TODO: ensure these are really used properly
const ( DefaultMaxLineLength = 10000 // Logplex max is 10000 bytes, so default to that DefaultInputFormat = InputFormatRaw DefaultFrontBuff = 1000 DefaultBackBuff = 50 DefaultTimeout = 5 * time.Second DefaultWaitDuration = 250 * time.Millisecond DefaultMaxAttempts = 3 DefaultStatsInterval = 0 * time.Second DefaultStatsSource = "" DefaultPrintVersion = false DefaultVerbose = false DefaultSkipVerify = false DefaultPriVal = "190" DefaultVersion = "1" DefaultProcID = "shuttle" DefaultAppName = "token" DefaultHostname = "shuttle" DefaultMsgID = "- -" DefaultLogsURL = "" DefaultNumBatchers = 2 DefaultNumOutlets = 4 DefaultBatchSize = 500 DefaultID = "" DefaultDrop = true )
Default option values
const ( // EOFRetrySleep is the amount of time to sleep between retries caused by an io.EOF, in ms. EOFRetrySleep = 100 // OtherRetrySleep is the tIme to sleep between retries for any other error, in ms. OtherRetrySleep = 1000 // DepthHighWatermark is the high watermark, beyond which the outlet looses batches instead of retrying. DepthHighWatermark = 0.6 // RetryFormat is the format string for retries RetryFormat = "at=post retry=%t msgcount=%d inbox.length=%d request_id=%q attempts=%d error=%q\n" // RetryWithTypeFormat if the format string for retries that also have a type RetryWithTypeFormat = "at=post retry=%t msgcount=%d inbox.length=%d request_id=%q attempts=%d error=%q errtype=\"%T\"\n" )
const ( // LogplexBatchTimeFormat is the format of timestamps as expected by Logplex LogplexBatchTimeFormat = "2006-01-02T15:04:05.000000+00:00" // LogplexContentType is the content type logplex expects LogplexContentType = "application/logplex-1" )
Variables ¶
var (
DefaultFormatterFunc = NewLogplexBatchFormatter
)
Defaults that can't be constants
Functions ¶
This section is empty.
Types ¶
type Batch ¶
type Batch struct { UUID *uuid.UUID // contains filtered or unexported fields }
Batch holds incoming log lines and provides some helpers for dealing with the grouping of logLines
type Batcher ¶
type Batcher struct {
// contains filtered or unexported fields
}
Batcher coalesces logs coming via inLogs into batches, which are sent out via outBatches
func NewBatcher ¶
NewBatcher created an empty Batcher for the provided shuttle
type Config ¶
type Config struct { MaxLineLength int BackBuff int FrontBuff int BatchSize int NumBatchers int NumOutlets int InputFormat int MaxAttempts int LogsURL string Prival string Version string Procid string Hostname string Appname string Msgid string StatsSource string SkipVerify bool PrintVersion bool Verbose bool UseGzip bool Drop bool WaitDuration time.Duration Timeout time.Duration StatsInterval time.Duration ID string FormatterFunc NewHTTPFormatterFunc // Loggers Logger *log.Logger ErrLogger *log.Logger // contains filtered or unexported fields }
Config holds the various config options for a shuttle
func NewConfig ¶
func NewConfig() Config
NewConfig returns a newly created Config, filled in with defaults
func (*Config) ComputeHeader ¶
func (c *Config) ComputeHeader()
ComputeHeader computes the syslogFrameHeaderFormat once so we don't have to do that for every formatter itteration Should be called after setting up the rest of the config or if the config changes
type Counter ¶
Counter is used to track 2 values for a given metric. The first item is the "all time" metric counterand the second is the last value since the metric was ReadAndReset. Counters are safe for concurrent use.
func NewCounter ¶
NewCounter returns a new Counter initialized to the initial value
type GzipFormatter ¶
type GzipFormatter struct {
// contains filtered or unexported fields
}
GzipFormatter is an HTTPFormatter that is built with a delegate HTTPFormatter but which compresses the request body
func NewGzipFormatter ¶
func NewGzipFormatter(delegate HTTPFormatter) *GzipFormatter
NewGzipFormatter builds a new GzipFormatter with the supplied delegate
func (*GzipFormatter) MsgCount ¶
func (g *GzipFormatter) MsgCount() int
MsgCount return the number of messages contained in the formatted batch
type HTTPFormatter ¶
type HTTPFormatter interface { Request() (*http.Request, error) // Request() returns a *http.Request ready to be handled by an outlet SubFormatter }
HTTPFormatter is the interface that http outlets use to format a HTTP request.
func NewKinesisFormatter ¶
func NewKinesisFormatter(b Batch, eData []errData, config *Config) HTTPFormatter
NewKinesisFormatter constructs a proper HTTPFormatter for Kinesis http targets
func NewLogplexBatchFormatter ¶
func NewLogplexBatchFormatter(b Batch, eData []errData, config *Config) HTTPFormatter
NewLogplexBatchFormatter returns a new LogplexBatchFormatter wrapping the provided batch
type HTTPOutlet ¶
type HTTPOutlet struct { // User supplied loggers Logger *log.Logger // contains filtered or unexported fields }
HTTPOutlet handles delivery of batches to HTTP endpoints by creating formatters for each request. HTTPOutlets handle retries, response parsing and lost counters
func NewHTTPOutlet ¶
func NewHTTPOutlet(s *Shuttle) *HTTPOutlet
NewHTTPOutlet returns a properly constructed HTTPOutlet for the given shuttle
func (*HTTPOutlet) Outlet ¶
func (h *HTTPOutlet) Outlet()
Outlet receives batches from the inbox and submits them to logplex via HTTP.
type KinesisFormatter ¶
KinesisFormatter formats batches destined for AWS Kinesis HTTP endpoints Kinesis has a very small payload side, so recommend setting config.BatchSize in the 1-3 range so as to not loose logs because we go over the batch size. Kinesis formats the Data using the LogplexLineFormatter, which is additionally base64 encoded.
func (*KinesisFormatter) MsgCount ¶
func (kf *KinesisFormatter) MsgCount() int
MsgCount returns the number of records that the formatter is formatting
func (*KinesisFormatter) Request ¶
func (kf *KinesisFormatter) Request() (*http.Request, error)
Request constructs a request for this formatter See: http://docs.aws.amazon.com/kinesis/latest/APIReference/API_PutRecord.html
type KinesisRecord ¶
type KinesisRecord struct {
// contains filtered or unexported fields
}
KinesisRecord is used to marshal LoglexLineFormatters to Kinesis Records for the PutRecords API Call
func (KinesisRecord) WriteTo ¶
func (r KinesisRecord) WriteTo(w io.Writer) (n int64, err error)
WriteTo writes the LogplexLineFormatter to the provided writer in Kinesis' PutRecordsFormat. Conforms to the WriterTo interface.
Since the data should be relatively small just read the LogplexLineFormatter into ram making a bunch of temporary garbage. AFAICT it's just not worth it to try and string these together with io.Pipes and the like. :-(
type LogLine ¶
type LogLine struct {
// contains filtered or unexported fields
}
LogLine holds the new line terminated log messages and when shuttle received them.
func NewLogLine ¶
NewLogLine returns a newly constructed LogLine containing the provided line and time
type LogLineReader ¶
type LogLineReader struct {
// contains filtered or unexported fields
}
LogLineReader performs the reading of lines from an io.ReadCloser, encapsulating lines into a LogLine and emitting them on outbox
func NewLogLineReader ¶
func NewLogLineReader(o chan<- LogLine, m metrics.Registry) LogLineReader
NewLogLineReader constructs a new reader with it's own Outbox.
func (LogLineReader) Enqueue ¶
func (rdr LogLineReader) Enqueue(ll LogLine)
Enqueue a single log line and increment the line counters
func (LogLineReader) ReadLogLines ¶
func (rdr LogLineReader) ReadLogLines(input io.ReadCloser) error
ReadLogLines reads lines from the ReadCloser
type LogplexBatchFormatter ¶
LogplexBatchFormatter implements on io.Reader that returns Logplex formatted log lines. Wraps log lines in length prefixed rfc5424 formatting, splitting them as necessary to config.MaxLineLength
func (*LogplexBatchFormatter) MsgCount ¶
func (bf *LogplexBatchFormatter) MsgCount() int
MsgCount of the wrapped batch.
type LogplexLineFormatter ¶
type LogplexLineFormatter struct {
// contains filtered or unexported fields
}
LogplexLineFormatter formats individual loglines into length prefixed rfc5424 messages via an io.Reader interface
func NewLogplexErrorFormatter ¶
func NewLogplexErrorFormatter(err errData, config *Config) *LogplexLineFormatter
NewLogplexErrorFormatter returns a LogplexLineFormatter for the error data. These can be used to inject error data into the log stream
func NewLogplexLineFormatter ¶
func NewLogplexLineFormatter(ll LogLine, config *Config) *LogplexLineFormatter
NewLogplexLineFormatter returns a new LogplexLineFormatter wrapping the provided LogLine
func (*LogplexLineFormatter) AppName ¶
func (llf *LogplexLineFormatter) AppName() string
AppName returns the name of app name field based on the inputFormat For use in syslog framing
func (*LogplexLineFormatter) MsgCount ¶
func (llf *LogplexLineFormatter) MsgCount() int
MsgCount is always 1 for a Line
func (*LogplexLineFormatter) Read ¶
func (llf *LogplexLineFormatter) Read(p []byte) (n int, err error)
Implements the io.Reader interface tries to fill p as full as possible before returning
func (*LogplexLineFormatter) Reset ¶
func (llf *LogplexLineFormatter) Reset()
Reset the reader so that the log line can be re-read
type NewHTTPFormatterFunc ¶
type NewHTTPFormatterFunc func(b Batch, eData []errData, config *Config) HTTPFormatter
NewHTTPFormatterFunc defines the function type for defining creating and returning a new Formatter
type Shuttle ¶
type Shuttle struct { LogLineReader LogLines chan LogLine Batches chan Batch MetricsRegistry metrics.Registry Drops, Lost *Counter NewFormatterFunc NewHTTPFormatterFunc Logger *log.Logger ErrLogger *log.Logger // contains filtered or unexported fields }
Shuttle is the main entry point into the library
func NewShuttle ¶
NewShuttle returns a properly constructed Shuttle with a given config
type SubFormatter ¶
type SubFormatter interface { MsgCount() int // MsgCount is the number of messages after formatting io.Reader }
SubFormatter formats a complete batch or a subsection of a batch. It may split lines in the batch as needed by the destination, making the MsgCount() of the formatter different from the MsgCount of the source batch. A formatter may emitt more (likely) or less bytes for a given LogLine than the actual Logline.