slogproto

package module
v0.0.0-...-456c38a Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 13, 2023 License: MPL-2.0 Imports: 15 Imported by: 0

README

slogproto

Warning This is an experimental module and is subject to change.

Go slog.Handler using Protocol Buffers. This can reduce the size of log messages when saving them to disk or sending them over the network, and can reduce the amount of time spent marshaling and unmarshaling log messages at the cost of human readability. To enable interopability with other tools, the slp CLI can read protobuf encoded slog.Records from STDIN (or a file) and output them as JSON to STDOUT. Logs can be filtered using CEL expressions.

Installation

For use with Go programs:

$ go get -u -v github.com/picatz/slogproto@latest

To use the slp CLI:

$ go install -v github.com/picatz/slogproto/cmd/slp@latest

Usage

package main

import (
	"os"

	"github.com/picatz/slogproto"
	"golang.org/x/exp/slog"
)

func main() {
	logger := slog.New(slogproto.NewHander(os.Stdout))

	logger.Info("example", slog.Int("something", 1))
}

$ go run main.go | slp
{"time":"2023-08-01T03:12:11.272826Z","level":"INFO","msg":"example","something":1}

$ slp output.log
{"time":"..","level":"...","msg":"...", ... }
{"time":"..","level":"...","msg":"...", ... }
{"time":"..","level":"...","msg":"...", ... }
...

Note Input to slp can be from STDIN or a file.

Filtering

The filter flag can be used to filter logs using a given expression. The expression is evaluated against the slog.Record and must return a boolean value. For each log record that the expression evaluates as true will be output to STDOUT as JSON.

  • attrs is a map of all the attributes in the log record, not including the message, level, or time.

    attrs.something == 1

    has(attrs.something) && attrs.something == 1

    Important Invalid access to an attribute will cause the filter to fail at evaluation time. Wrap the expression in a has() function to check if the attribute exists before accessing it.

  • msg is the message in the log record.

  • level is the level in the log record.

  • time is the timestamp in the log record.

$ slp --filter='has(attrs.something)' output.log
{"time":"2023-08-11T00:06:00.474782Z","level":"INFO","msg":"example","something":1}
$  --filter='msg == "this is a test"' test.log
{"time":"2023-08-11T00:06:00.474033Z","level":"INFO","msg":"this is a test","test":{"test2":"1","test3":1,"test1":1}}

File Format

The file format is a series of delimited Protocol Buffer messages. Each message is prefixed with a 32-bit unsigned integer representing the size of the message. The message itself is a protobuf encoded slog.Record.

╭────────────────────────────────────────────────────────────╮
│  Message Size  │  Protocol Buffer Message  │  ...  │  EOF  │
╰────────────────────────────────────────────────────────────╯

Comparisons to Other Formats

Using the following record written 1024 times:

{
	"time": $timestamp,
	"level": "INFO",
	"msg": "hello world",
	"i": $n
}
Format GZIP (KB) Snappy (KB) Zstandard (KB) Uncompressed (KB)
Protobuf 5.48 11.17 3.58 41.88
JSON 5.79 9.59 5.04 86.81
Text 2.93 7.66 1.31 69.92

Documentation

Overview

Package slogproto provides a protocol buffer definition for the slog format (golang.org/x/exp/slog).

It attempts to have minimial dependencies and minimize memory allocations.

Index

Examples

Constants

This section is empty.

Variables

View Source 
var (
	Level_name = map[int32]string{
		0: "Unset",
		1: "Info",
		2: "Warn",
		3: "Error",
		4: "Debug",
	}
	Level_value = map[string]int32{
		"Unset": 0,
		"Info":  1,
		"Warn":  2,
		"Error": 3,
		"Debug": 4,
	}
)

Enum value maps for Level.

View Source 
var File_slog_proto protoreflect.FileDescriptor

Functions

func Read 

func Read(ctx context.Context, r io.Reader, fn func(r *slog.Record) bool) error

Types

type Handler 

type Handler struct {
	// contains filtered or unexported fields
}

Handler implements the slog.Handler interface and writes the log record to the writer as a protocol buffer encoded struct containing the log record, including the levem, message and attributes.

func NewHandler 

func NewHandler(w io.Writer) *Handler

NewHandler returns a new Handler that writes to the writer.

Example 

h := slogproto.NewHandler(os.Stdout)
Example 
package main

import (
	"bytes"

	"github.com/picatz/slogproto"
	"golang.org/x/exp/slog"
)

func main() {
	var logBuffer bytes.Buffer

	logger := slog.New(slogproto.NewHandler(&logBuffer))

	logger.Info("this is a test",
		slog.Group("test",
			slog.Int("test1", 1),
			slog.String("test2", "1"),
			slog.Float64("test3", 1.0),
		),
	)

	logger.Info("example", slog.Int("something", 1))
}

Output:

func (*Handler) Enabled 

func (h *Handler) Enabled(ctx context.Context, level slog.Level) bool

Enabled returns true if the level is enabled for the handler.

func (*Handler) Handle 

func (h *Handler) Handle(ctx context.Context, r slog.Record) error

Handle writes the log record to the writer as a protocol buffer encoded struct containing the log record, including the level, message and attributes.

It will only be called when Enabled returns true. The Context argument is as for Enabled. It is present solely to provide Handlers access to the context's values. Canceling the context should not affect record processing. (Among other things, log messages may be necessary to debug a cancellation-related problem.)

Handle methods that produce output should observe the following rules:

  • If r.Time is the zero time, ignore the time.
  • If r.PC is zero, ignore it.
  • Attr's values should be resolved.
  • If an Attr's key and value are both the zero value, ignore the Attr. This can be tested with attr.Equal(Attr{}).
  • If a group's key is empty, inline the group's Attrs.
  • If a group has no Attrs (even if it has a non-empty key), ignore it.

func (*Handler) WithAttrs 

func (h *Handler) WithAttrs(attrs []slog.Attr) slog.Handler

WithAttrs returns a new Handler whose attributes consist of both the receiver's attributes and the arguments.

The Handler owns the slice: it may retain, modify or discard it.

func (*Handler) WithGroup 

func (h *Handler) WithGroup(name string) slog.Handler

WithGroup returns a new Handler with the given group appended to the receiver's existing groups.

The keys of all subsequent attributes, whether added by With or in a Record, should be qualified by the sequence of group names.

How this qualification happens is up to the Handler, so long as this Handler's attribute keys differ from those of another Handler with a different sequence of group names.

A Handler should treat WithGroup as starting a Group of Attrs that ends at the end of the log event. That is, 

logger.WithGroup("s").LogAttrs(level, msg, slog.Int("a", 1), slog.Int("b", 2))

should behave like 

logger.LogAttrs(level, msg, slog.Group("s", slog.Int("a", 1), slog.Int("b", 2)))

If the name is empty, WithGroup returns the receiver.

type Level 

type Level int32
const (
	Level_Unset Level = 0
	Level_Info  Level = 1
	Level_Warn  Level = 2
	Level_Error Level = 3
	Level_Debug Level = 4
)

func (Level) Descriptor 

func (Level) Descriptor() protoreflect.EnumDescriptor

func (Level) Enum 

func (x Level) Enum() *Level

func (Level) EnumDescriptor deprecated 

func (Level) EnumDescriptor() ([]byte, []int)

Deprecated: Use Level.Descriptor instead.

func (Level) Number 

func (x Level) Number() protoreflect.EnumNumber

func (Level) String 

func (x Level) String() string

func (Level) Type 

func (Level) Type() protoreflect.EnumType

type Record 

type Record struct {
	Time    *timestamppb.Timestamp `protobuf:"bytes,1,opt,name=time,proto3" json:"time,omitempty"`
	Message string                 `protobuf:"bytes,2,opt,name=message,proto3" json:"message,omitempty"`
	Level   Level                  `protobuf:"varint,3,opt,name=level,proto3,enum=slog.Level" json:"level,omitempty"`
	Attrs   map[string]*Value      `` /* 151-byte string literal not displayed */
	// contains filtered or unexported fields
}

func (*Record) Descriptor deprecated 

func (*Record) Descriptor() ([]byte, []int)

Deprecated: Use Record.ProtoReflect.Descriptor instead.

func (*Record) GetAttrs 

func (x *Record) GetAttrs() map[string]*Value

func (*Record) GetLevel 

func (x *Record) GetLevel() Level

func (*Record) GetMessage 

func (x *Record) GetMessage() string

func (*Record) GetTime 

func (x *Record) GetTime() *timestamppb.Timestamp

func (*Record) ProtoMessage 

func (*Record) ProtoMessage()

func (*Record) ProtoReflect 

func (x *Record) ProtoReflect() protoreflect.Message

func (*Record) Reset 

func (x *Record) Reset()

func (*Record) String 

func (x *Record) String() string

type Value 

type Value struct {

	// Types that are assignable to Kind:
	//
	//	*Value_Bool
	//	*Value_Float
	//	*Value_Int
	//	*Value_String_
	//	*Value_Time
	//	*Value_Duration
	//	*Value_Uint
	//	*Value_Group_
	//	*Value_Any
	Kind isValue_Kind `protobuf_oneof:"kind"`
	// contains filtered or unexported fields
}

func (*Value) Descriptor deprecated 

func (*Value) Descriptor() ([]byte, []int)

Deprecated: Use Value.ProtoReflect.Descriptor instead.

func (*Value) GetAny 

func (x *Value) GetAny() *anypb.Any

func (*Value) GetBool 

func (x *Value) GetBool() bool

func (*Value) GetDuration 

func (x *Value) GetDuration() *durationpb.Duration

func (*Value) GetFloat 

func (x *Value) GetFloat() float64

func (*Value) GetGroup 

func (x *Value) GetGroup() *Value_Group

func (*Value) GetInt 

func (x *Value) GetInt() int64

func (*Value) GetKind 

func (m *Value) GetKind() isValue_Kind

func (*Value) GetString_ 

func (x *Value) GetString_() string

func (*Value) GetTime 

func (x *Value) GetTime() *timestamppb.Timestamp

func (*Value) GetUint 

func (x *Value) GetUint() uint64

func (*Value) ProtoMessage 

func (*Value) ProtoMessage()

func (*Value) ProtoReflect 

func (x *Value) ProtoReflect() protoreflect.Message

func (*Value) Reset 

func (x *Value) Reset()

func (*Value) String 

func (x *Value) String() string

type Value_Any 

type Value_Any struct {
	Any *anypb.Any `protobuf:"bytes,9,opt,name=any,proto3,oneof"`
}

type Value_Bool 

type Value_Bool struct {
	Bool bool `protobuf:"varint,1,opt,name=bool,proto3,oneof"`
}

type Value_Duration 

type Value_Duration struct {
	Duration *durationpb.Duration `protobuf:"bytes,6,opt,name=duration,proto3,oneof"`
}

type Value_Float 

type Value_Float struct {
	Float float64 `protobuf:"fixed64,2,opt,name=float,proto3,oneof"`
}

type Value_Group 

type Value_Group struct {
	Attrs map[string]*Value `` /* 151-byte string literal not displayed */
	// contains filtered or unexported fields
}

func (*Value_Group) Descriptor deprecated 

func (*Value_Group) Descriptor() ([]byte, []int)

Deprecated: Use Value_Group.ProtoReflect.Descriptor instead.

func (*Value_Group) GetAttrs 

func (x *Value_Group) GetAttrs() map[string]*Value

func (*Value_Group) ProtoMessage 

func (*Value_Group) ProtoMessage()

func (*Value_Group) ProtoReflect 

func (x *Value_Group) ProtoReflect() protoreflect.Message

func (*Value_Group) Reset 

func (x *Value_Group) Reset()

func (*Value_Group) String 

func (x *Value_Group) String() string

type Value_Group_ 

type Value_Group_ struct {
	Group *Value_Group `protobuf:"bytes,8,opt,name=group,proto3,oneof"`
}

type Value_Int 

type Value_Int struct {
	Int int64 `protobuf:"varint,3,opt,name=int,proto3,oneof"`
}

type Value_String_ 

type Value_String_ struct {
	String_ string `protobuf:"bytes,4,opt,name=string,proto3,oneof"`
}

type Value_Time 

type Value_Time struct {
	Time *timestamppb.Timestamp `protobuf:"bytes,5,opt,name=time,proto3,oneof"`
}

type Value_Uint 

type Value_Uint struct {
	Uint uint64 `protobuf:"varint,7,opt,name=uint,proto3,oneof"`
}

Source Files

View all Source files

Directories

Path Synopsis
cmd
slp

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.