loges: github.com/araddon/loges Index | Files | Directories

package loges

import "github.com/araddon/loges"

Index

Package Files

filter_file.go filter_fileflatten.go filter_fluentd.go filter_graphite.go input_http.go input_monit.go input_stdio.go input_tail.go logstash.go sink_elasticsearch.go sink_justtransform.go sink_stdout.go

func FirstNonWhitespace Uses

func FirstNonWhitespace(bs []byte) (r rune, ok bool)

Returns the first non-whitespace rune in the byte slice. Returns false if the input is not valid UTF-8 or the input contains no runes except whitespace. Returns

func FlattenMonit Uses

func FlattenMonit(r *MonitRequest) (url.Values, int64)

takes a monit object struture, and converts to a map[string]interface{} which it serializes to json, to send over a wire

func HttpRun Uses

func HttpRun(httpPort string, msgsOut chan *LineEvent)

We are starting an http -> collector

accepts monit data

func IsJson Uses

func IsJson(by []byte) bool

func IsJsonArray Uses

func IsJsonArray(by []byte) bool

Determines if the bytes is a json array, only looks at prefix

not parsing the entire thing

func IsJsonObject Uses

func IsJsonObject(by []byte) bool

func MakeCustomHandler Uses

func MakeCustomHandler(msgsOut chan *LineEvent) http.HandlerFunc

func MakeFileFlattener Uses

func MakeFileFlattener(filename string, msgChan chan *LineEvent) func(string)

This formatter reads go files and performs:

1.  Squashes multiple lines into one (as needed), Tries to squash panics(go) into one line
2.  Reads out the LineType/Level [DEBUG,INFO,METRIC] into a field

This expects log files in this format

2013-05-25 13:25:32.475 authctx.go:169: [DEBUG] sink       Building sink for kafka from factory method

func MakeMonitHandler Uses

func MakeMonitHandler(msgsOut chan *LineEvent) http.HandlerFunc

func MonitParse Uses

func MonitParse(data []byte) (url.Values, int64)

accepts a smokr message of monit format, and parses and converts to a name/value pair format

func RunTransforms Uses

func RunTransforms(parallel int, msgChan chan *LineEvent)

just run the transforms

func StdinPruducer Uses

func StdinPruducer(msgChan chan *LineEvent)

sends messages from stdin for consumption

func TailFile Uses

func TailFile(filename string, config tail.Config, done chan bool, msgChan chan *LineEvent)

func ToElasticSearch Uses

func ToElasticSearch(msgChan chan *LineEvent, esType, esHost, ttl string,
    exitIfNoMsgs time.Duration, sendMetrics bool)

read the message channel and send to elastic search Uses the background Bulk Indexor, which has the **Possibility** of losing data if app panic/quits (but is much faster than non-bulk)

@exitIfNoMsgs :  Should we panic if we don't see messages after this duration?

func ToStdout Uses

func ToStdout(msgChan chan *LineEvent, colorize bool)

read the message channel and send to stdout

func TransformRegister Uses

func TransformRegister(txform LineTransform)

func UpdateLogstashIndex Uses

func UpdateLogstashIndex()

update the index occasionally

type Event Uses

type Event struct {
    Source      string                 `json:"@source"`
    Type        string                 `json:"@type"`
    Timestamp   time.Time              `json:"@timestamp"`
    Message     string                 `json:"@message"`
    Tags        []string               `json:"@tags,omitempty"`
    IndexFields map[string]interface{} `json:"@idx,omitempty"`
    Fields      map[string]interface{} `json:"@fields,omitempty"`
    Raw         *json.RawMessage       `json:"jsonfields,omitempty"`
    // contains filtered or unexported fields
}

A Logstash formatted event

func NewEvent Uses

func NewEvent(eventType, source, message string) *Event

func NewTsEvent Uses

func NewTsEvent(eventType, source, message string, t time.Time) *Event

New event using existing time stamp

func (*Event) Id Uses

func (e *Event) Id() string

func (*Event) Index Uses

func (e *Event) Index() string

func (*Event) SetId Uses

func (e *Event) SetId(id string)

Set the id instead of using mongo objectid as Id

func (*Event) String Uses

func (e *Event) String() string

type GraphiteRunner Uses

type GraphiteRunner struct {
    Mu sync.Mutex
    // contains filtered or unexported fields
}

func NewGraphiteRunner Uses

func NewGraphiteRunner(addr string) *GraphiteRunner

func (*GraphiteRunner) Close Uses

func (g *GraphiteRunner) Close()

type LineEvent Uses

type LineEvent struct {
    Data      []byte
    Prefix    string
    Ts        time.Time // Date string if found
    LogLevel  string    // [METRIC, INFO,  DEBUG]
    Source    string    // Source = filename if file, else monit, etc
    Offset    uint64
    Item      interface{}
    WriteErrs uint16
}

Representing data about a log line from file we are tailing

func (*LineEvent) IsJson Uses

func (l *LineEvent) IsJson() bool

func (*LineEvent) IsMetric Uses

func (l *LineEvent) IsMetric() bool

type LineTransform Uses

type LineTransform func(*LineEvent) *Event

func FileFormatter Uses

func FileFormatter(logstashType string, tags []string) LineTransform

file format [date source jsonmessage] parser

func FluentdFormatter Uses

func FluentdFormatter(logstashType string, tags []string) LineTransform

Fluentd format [date source jsonmessage] parser

func GraphiteTransform Uses

func GraphiteTransform(logstashType, addr, prefix string, metricsToEs bool) LineTransform

type MonitBlock Uses

type MonitBlock struct {
    Percent float32 `xml:"percent"`
    Usage   string  `xml:"usage"`
    Total   string  `xml:"total"`
}

<block>

<percent>57.5</percent>
<usage>2051.3 MB</usage>
<total>37544.3 MB</total>

</block>

func (*MonitBlock) PercentTotalStr Uses

func (m *MonitBlock) PercentTotalStr() string

type MonitCpu Uses

type MonitCpu struct {
    Percent      float32 `xml:"percent"`
    PercentTotal float32 `xml:"percenttotal"`
}

func (*MonitCpu) PercentTotalStr Uses

func (m *MonitCpu) PercentTotalStr() string

type MonitINode Uses

type MonitINode struct {
    Percent float32 `xml:"percent"`
    Usage   float32 `xml:"usage"`
    Total   float32 `xml:"total"`
}

<inode>

<percent>57.5</percent>
<usage>36786.6</usage>
<total>70127.3</total>

</inode>

func (*MonitINode) PercentTotalStr Uses

func (m *MonitINode) PercentTotalStr() string

type MonitMemory Uses

type MonitMemory struct {
    Percent       float32 `xml:"percent"`
    PercentTotal  float32 `xml:"percenttotal"`
    Kilobyte      int     `xml:"kilobyte"`
    KilobyteTotal int     `xml:"kilobytetotal"`
}

func (*MonitMemory) PercentTotalStr Uses

func (m *MonitMemory) PercentTotalStr() string

type MonitPlatform Uses

type MonitPlatform struct {
    Name    string `xml:"name"`
    Release string `xml:"release"`
    Version string `xml:"version"`
    Machine string `xml:"machine"`
    Cpu     int    `xml:"cpu"`
    Memory  int    `xml:"memory"`
    Swap    int    `xml:"swap"`
}

type MonitRequest Uses

type MonitRequest struct {
    Id            int            `xml:"id,attr"`
    Incarnation   string         `xml:"incarnation,attr"`
    Version       string         `xml:"version,attr"`
    Server        MonitServer    `xml:"server"`
    Platform      MonitPlatform  `xml:"platform"`
    Services      []MonitService `xml:"services>service"`
    ServiceGroups interface{}    `xml:"servicegroups"`
}

type MonitServer Uses

type MonitServer struct {
    Uptime      int    `xml:"uptime"`
    Poll        int    `xml:"poll"`
    StartDelay  int    `xml:"startdelay"`
    Host        string `xml:"localhostname"`
    ControlFile string `xml:"controlfile"`
}

type MonitService Uses

type MonitService struct {
    Type   int         `xml:"type"` // 0 = filesystem, 3 = service, 5= System(os)
    Name   string      `xml:"name,attr"`
    Ts     int         `xml:"collected_sec"`
    Tsu    int         `xml:"collected_usec"`
    Status int         `xml:"status"`
    Memory MonitMemory `xml:"memory"`
    Cpu    MonitCpu    `xml:"cpu"`
    Block  MonitBlock  `xml:"block"`
    INode  MonitINode  `xml:"inode"`
}

type NvMetrics Uses

type NvMetrics struct {
    url.Values
}

func NewNvMetrics Uses

func NewNvMetrics(qs string) (NvMetrics, error)

func (*NvMetrics) MetricTypeVal Uses

func (uv *NvMetrics) MetricTypeVal(name string) (string, string)

func (*NvMetrics) Value Uses

func (uv *NvMetrics) Value(name string) (interface{}, error)

Directories

PathSynopsis
examples/customformat
kafka
loges

Package loges imports 25 packages (graph) and is imported by 9 packages. Updated 2018-03-07. Refresh now. Tools for package owners.