logd-go

module

v0.0.0-...-65871c1 Latest Latest Go to latest Published: May 9, 2018 License: MIT

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

github.com/ernestrc/logd-go

Links

Open Source Insights

README ¶

Logd

Logd is a log processor daemon that exposes a lua API to run arbitrary logic on structured logs.

Usage

-- This can suplied to the logd executable: tail -n0 -f /var/log/mylog.log | logd -R my_script.lua

local logd = require("logd")
local os = require("os")
local tick = 100

function logd.on_tick ()
	tick = tick * 2
	logd.config_set("tick", tick)
	logd.debug({ next_tick = tick, msg = "triggered!" })
end

-- define entry point
function logd.on_log(logptr)
	logd.debug(string.format("processed log: %s", logd.log_string(logptr)))
end

-- define signal handler
function logd.on_signal(signal)
	logd.debug({ msg = string.format("My Lua script received signal: %s", signal) })

	if signal == "SIGUSR1" then
		logd.debug({ msg = "realoading after this function returns.." })
	else
		os.exit(1)
	end
end

-- example usage of "config_set" builtin
logd.config_set("tick", tick)

Lua API

Builtin	Description
`function logd.config_set (key, value)`	Set a configuration key/value pair. See Config table for more information.
`function logd.http_get (url, [headers]) body, err`	Perform a blocking HTTP GET request to `url` and return the `body` of the response and/or non-nil `err` if there was an error
`function logd.http_post (url, payload, contentType [, affinity])`	Perform an HTTP POST request to the given URL with the given `payload` and `Content-Type` header set to `contentType`. Call is non-blocking unless HTTP client is applying back-pressure. `affinity` defines an HTTP queue affinity to synchronize HTTP requests.
`function logd.kafka_offset (key, value)`	Create a new named kafka offset.
`function logd.kafka_message (key, value, topic, partition [, offsetptr]) msgptr`	Create a new kafka message and return a pointer to it. `partition` can be set to -1 to use any partition. `offsetptr` can be null.
`function logd.kafka_produce (msgptr)`	Produce a single message. This is an asynchronous call that enqueues the message on the internal transmit queue, thus returning immediately unless Producer is applying back-pressure. The delivery report will be supplied via `on_kafka_report` callback if specified.
`function logd.log_get (logptr, key) value`	Get a property from the structured log.
`function logd.log_set (logptr, key, value)`	Set a property to the structured log.
`function logd.log_remove (logptr, key)`	Remove a property from the structured log.
`function logd.log_reset (logptr)`	Reset all log properties.
`function logd.log_string (logptr) str`	Serialize a structured log into a string (with the same format used by the parser).
`function logd.log_json (logptr) str`	Serialize the structured log into a JSON string.
`function logd.debug (string\|table)`	Write arbitrary data to the process' debug log.

Hook	Description
`function logd.on_log (logptr)`	Logs are parsed and supplied to this handler. Use `logd.log_*` set of functions to manipulate them.
`function logd.on_error (logptr, error)`	When `protected` configuration is set to true, runtime errors are supplied to this handler.
`function logd.on_signal (signal)`	Define an OS signal handler. Note that the collector handles SIGUSR1 by default to reload script but behavior can be overwritten by this handler.
`function logd.on_tick ()`	Define interval handler. Interval duration can be configued via `tick` configuration.
`function logd.on_http_error (url, method, error)`	Define a `logd.http_post` asynchronous error handler.
`function logd.on_kafka_report (msgptr, kerr)`	The delivery report callback is used by librdkafka to signal the status of a message posting, it will be called once for each message to report the status of message delivery.

Config	Description
`protected`	Run Lua code in protected mode. Runtime errors will be supplied to `logd.on_error` hook.
`http.concurrency`	Number of `logd.http_post` queues to instantiate.
`http.timeout`	`logd.http_post` timeout response timeout.
`http.channel_buffer`	Number of pending requests per queue before the HTTP client applies backpressure to `logd.http_post`.
`kafka.*`	Property passed directly to librdkafka to configure the Kafka producer. Please check https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md for more information.
`tick`	Interval in milliseconds to call `on_tick`.

Lua libraries included:

package
table
io
os
string
bit32
math
debug

Parser

The parser expects logs to be in the following format:

YYYY-MM-dd hh:mm:ss	LEVEL	[Thread]	Class	key: value...

Build

If you do not have librdkafka (v0.11.1) installed on your system and if you have docker installed, you can use make static to compile and statically link a logd executable without needing to install any dependencies.

TODO

Add more input parsers. i.e.:
- https://en.wikipedia.org/wiki/Common_Log_Format

Directories ¶

Path	Synopsis
cmd
logd
http
logging
lua

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL