butteredscones

package module
v0.0.0-...-eb37774 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 17, 2015 License: MIT Imports: 18 Imported by: 0

README

butteredscones Build Status

butteredscones watches log files on disk and forwards them to logstash via the lumberjack protocol. It is similar to logstash-forwarder.

I'm a lumberjack and I'm OK

I sleep all night and I work all day.

[...]

On Wednesdays I go shoppin'

And have buttered scones for tea.

Configuration

Like logstash-forwarder, butteredscones is configured via a JSON file.

{
  "state": "/var/lib/butteredscones/state.db",

  "network": {
    "servers": [
      {
        "addr": "192.168.0.1:5043",
        "name": "logstash.internal.example.com",
      }
    ],
    "certificate":  "/etc/butteredscones/forwarder.crt",
    "key":          "/etc/butteredscones/forwarder.key",
    "ca":           "/etc/butteredscones/ca.crt",
    "timeout":      15
  },

  "statistics": {
    "addr": "127.0.0.1:8088"
  },

  "files": [
    {
      "paths":  ["/var/log/messages", "/var/log/*.log"],
      "fields": {"type": "syslog"}
    }
  ]
}

state is where butteredscones keeps information about how far it has read into each file. The directory where it lives must be writable by the user that runs the butteredscones process.

network/servers can include one or more servers. If multiple servers are present, butteredscones will send to all servers concurrently. Specifying an name for a server is optional. If specified, the addr will be used to connect, but the name will be used to verify the certificate. This allows butteredscones to connect properly even if DNS is broken.

The SSL certificate presented by the remote logstash server must be signed by the specified CA, if the "ca" option is specified. Otherwise, butteredscones will not communicate with the remote server.

If given, statistics/addr specifies a socket address where an HTTP server will listen. Statistics about what butteredscones is doing will be written in JSON format. Use these statistics to debug problems or write automated monitoring tools. For example: curl -si http://localhost:8088

files supports glob patterns. butteredscones will periodically check for new files that match the glob pattern and tail them.

Currently, butteredscones does not support log files that are truncated or renamed. This is not a use case the original developers had. However, if it interests you, pull requests are welcomed.

Development & Packaging

To build the static binary, butteredscones:

script/build

To run the tests:

script/test

To package butteredscones into a debian package:

GOOS=linux GOARCH=amd64 VERSION=0.0.1 script/deb

Future Work

  • Support input from standard in
  • Support files which are truncated in place

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type BoltSnapshotter

type BoltSnapshotter struct {
	DB *bolt.DB
}

func (*BoltSnapshotter) HighWaterMark

func (s *BoltSnapshotter) HighWaterMark(filePath string) (*HighWaterMark, error)

func (*BoltSnapshotter) SetHighWaterMarks

func (s *BoltSnapshotter) SetHighWaterMarks(marks []*HighWaterMark) error

type ClientStatistics

type ClientStatistics struct {
	Status string `json:"status"`

	// The number of lines sent successfully to the client
	LinesSent int `json:"lines_sent"`

	// The last time lines were successfully sent to this client
	LastSendTime time.Time `json:"last_send_time"`

	// The number of lines in the last chunk successfully sent to this client
	LastChunkSize int `json:"last_chunk_size"`
}

type Configuration

type Configuration struct {
	State      string                  `json:"state"`
	Network    NetworkConfiguration    `json:"network"`
	Statistics StatisticsConfiguration `json:"statistics"`
	Files      []FileConfiguration     `json:"files"`
	MaxLength  int                     `json:"max_length"`
}

func LoadConfiguration

func LoadConfiguration(configFile string) (*Configuration, error)

func (*Configuration) BuildTLSConfig

func (c *Configuration) BuildTLSConfig() (*tls.Config, error)

type ExponentialBackoff

type ExponentialBackoff struct {
	Minimum time.Duration
	Maximum time.Duration
	// contains filtered or unexported fields
}

func (*ExponentialBackoff) Current

func (b *ExponentialBackoff) Current() time.Duration

func (*ExponentialBackoff) Next

func (b *ExponentialBackoff) Next() time.Duration

func (*ExponentialBackoff) Reset

func (b *ExponentialBackoff) Reset()

type FileConfiguration

type FileConfiguration struct {
	Paths  []string          `json:"paths"`
	Fields map[string]string `json:"fields"`
}

type FileData

type FileData struct {
	client.Data
	*HighWaterMark
}

type FileReader

type FileReader struct {
	C         chan []*FileData
	ChunkSize int
	MaxLength int
	// contains filtered or unexported fields
}

func NewFileReader

func NewFileReader(file *os.File, fields map[string]string, chunkSize, maxLength int) (*FileReader, error)

func (*FileReader) FilePath

func (h *FileReader) FilePath() string

type FileReaderPool

type FileReaderPool struct {
	// contains filtered or unexported fields
}

func NewFileReaderPool

func NewFileReaderPool() *FileReaderPool

func (*FileReaderPool) Add

func (p *FileReaderPool) Add(reader *FileReader)

func (*FileReaderPool) Counts

func (p *FileReaderPool) Counts() (available int, locked int)

func (*FileReaderPool) IsPathInPool

func (p *FileReaderPool) IsPathInPool(filePath string) bool

func (*FileReaderPool) LockNext

func (p *FileReaderPool) LockNext() *FileReader

TODO: Figure out how to make this block, rather than return nil

func (*FileReaderPool) Remove

func (p *FileReaderPool) Remove(reader *FileReader)

func (*FileReaderPool) Unlock

func (p *FileReaderPool) Unlock(reader *FileReader)

func (*FileReaderPool) UnlockAll

func (p *FileReaderPool) UnlockAll(readers []*FileReader)

type FileReaderPoolStatistics

type FileReaderPoolStatistics struct {
	// The number of files in the pool that are available to be read
	Available int `json:"available"`

	// The number of files in the pool that are locked, ready to be sent, but
	// haven't been yet.
	Locked int `json:"locked"`
}

type FileStatistics

type FileStatistics struct {
	// The current size of the file.
	Size int64 `json:"size"`

	// The current position (in bytes) that has been read into the file. This
	// might be greater than SnapshotPosition if there are lines buffered into
	// memory that haven't been acknowledged by the server
	Position int64 `json:"position"`

	// The last time the file was read from into the in-memory buffer.
	LastRead time.Time `json:"last_read"`

	// The current position (in bytes) that has been successfully sent and
	// acknowledged by the remote server.
	SnapshotPosition int64 `json:"snapshot_position"`

	// The last time a line from this file was successfully sent and acknowledged
	// by the remote server.
	LastSnapshot time.Time `json:"last_snapshot"`
}

type HighWaterMark

type HighWaterMark struct {
	FilePath string

	// Position is the index in the file after a given line. Seeking to it would
	// read the next line.
	Position int64
}

type MemorySnapshotter

type MemorySnapshotter struct {
	// contains filtered or unexported fields
}

func (*MemorySnapshotter) HighWaterMark

func (s *MemorySnapshotter) HighWaterMark(filePath string) (*HighWaterMark, error)

func (*MemorySnapshotter) SetHighWaterMarks

func (s *MemorySnapshotter) SetHighWaterMarks(marks []*HighWaterMark) error

type NetworkConfiguration

type NetworkConfiguration struct {
	Servers     []ServerConfiguration `json:"servers"`
	Certificate string                `json:"certificate"`
	Key         string                `json:"key"`
	CA          string                `json:"ca"`
	Timeout     int                   `json:"timeout"`
	SpoolSize   int                   `json:"spool_size"`
}

type ServerConfiguration

type ServerConfiguration struct {
	Addr string `json:"addr"`
	Name string `json:"name"`
}

type Snapshotter

type Snapshotter interface {
	HighWaterMark(filePath string) (*HighWaterMark, error)
	SetHighWaterMarks(marks []*HighWaterMark) error
}

type Spooler

type Spooler struct {
	In  chan *FileData
	Out chan []*FileData
	// contains filtered or unexported fields
}

Spooler accepts items on the In channel and chunks them into items on the Out channel.

func NewSpooler

func NewSpooler(size int, timeout time.Duration) *Spooler

func (*Spooler) Spool

func (s *Spooler) Spool()

Spool accepts items from the In channel and spools them into the Out channel. To stop the spooling, close the In channel.

type Statistics

type Statistics struct {
	// contains filtered or unexported fields
}

Statistics keeps stats about the current operation of the program. It is meant to keep snapshot-in-time stats, as opposed to counters or timers that statsd offers.

Statistics may be exposed by APIs that allow human- or machine-readable monitoring.

var GlobalStatistics *Statistics = NewStatistics()

func NewStatistics

func NewStatistics() *Statistics

func (*Statistics) DeleteFileStatistics

func (s *Statistics) DeleteFileStatistics(filePath string)

func (*Statistics) IncrementClientLinesSent

func (s *Statistics) IncrementClientLinesSent(clientName string, linesSent int)

func (*Statistics) MarshalJSON

func (s *Statistics) MarshalJSON() ([]byte, error)

func (*Statistics) SetClientStatus

func (s *Statistics) SetClientStatus(clientName string, status string)

func (*Statistics) SetFilePosition

func (s *Statistics) SetFilePosition(filePath string, position int64)

func (*Statistics) SetFileSnapshotPosition

func (s *Statistics) SetFileSnapshotPosition(filePath string, snapshotPosition int64)

func (*Statistics) UpdateFileReaderPoolStatistics

func (s *Statistics) UpdateFileReaderPoolStatistics(available int, locked int)

func (*Statistics) UpdateFileSizeStatistics

func (s *Statistics) UpdateFileSizeStatistics()

UpdateFileSizeStatistics updates the Size attribute of each file, so it's easier to compare how much progress butteredscones has made through a file.

UpdateFileSizeStatistics should be called before displaying statistics to an end user.

type StatisticsConfiguration

type StatisticsConfiguration struct {
	Addr string `json:"addr"`
}

type StatisticsServer

type StatisticsServer struct {
	Statistics *Statistics
	Addr       string
}

StatisticsServer constructs an HTTP server that returns JSON formatted statistics. These statistics can be used for debugging or automated monitoring.

func (*StatisticsServer) ListenAndServe

func (s *StatisticsServer) ListenAndServe() error

type Supervisor

type Supervisor struct {

	// Optional settings
	SpoolSize int
	MaxLength int

	// How frequently to glob for new files that may have appeared
	GlobRefresh time.Duration
	// contains filtered or unexported fields
}

func NewSupervisor

func NewSupervisor(files []FileConfiguration, clients []client.Client, snapshotter Snapshotter, maxLength int) *Supervisor

func (*Supervisor) Start

func (s *Supervisor) Start()

Start pulls things together and plays match-maker.

func (*Supervisor) Stop

func (s *Supervisor) Stop()

Stop stops the supervisor cleanly, making sure all progress has been snapshotted before exiting.

Directories

Path Synopsis
cmd

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL