logging

logging: A slightly more advanced logging apparatus for Go

go/logging provides basic compatibility with the standard library log package insofar as its public API, with missing methods provided via a wrapper. However, go/logging also provides most "standard" (depending on whose standard you consult) logging levels available in logging utilities oft-used in other environments.

go/logging goes a step further than the stdlib log package--and a handful of other Golang logging libraries--in that it allows greater control over the log format by exposing an API to override output, per level, as it is logged (see AddFormatOveride). go/logging also provides a few other conveniences such as: Individual backends per logging level, hierarchical logging, and chained loggers. Some of these features are shared with competing libraries.

Why go/logging and not ${personal_preference}?

Because the plethora of other logging choices were not available when I first started writing go/logging and the ones that were out there I didn't like. Seriously.

In fact, at that time, the choices were essentially large logging libraries that did too much, libraries that attempted to foist a substantial paradigm shift outside what most users actually want (simpler is better), and the stdlib takes that latter bit too far (you can be too simple).

Thus I wrote go/logging. Consequently, I also still maintain it. If you don't like the idea of this library, you are absolutely welcome to not use it.

Changes in v2.x+

go/logging v2.x and greater introduce changes incompatible with prio versions and thereby require adapting existing code to the new API. In particular:

• Exported APIs have changed and new interfaces have been added. In particular, ImmutableLogger and PkgLogger now join StdLogger. ImmutableLogger is used in go/logging where logger options should not be changed by client code.
• go/logging always has a root logger present. To obtain it, simply call logging.Logger().
• Optionally, you may create your own root loggers as needed. There are two ways to do this: First, with Get(), which will register all loggers created via this method with go/logging; subsequent calls to Get() will return previously registered loggers. Second, using New(), an unregistered logger may be obtained.
• A new function Register() has been introduced to register top-level loggers which will then be accessible via Get(). There are some use cases where it may be desirable to register a logger sometime after creation; perhaps you wish to configure the logger first, or modify it in some form, but don't wish for the actual logger to be ready for use until this happens. Calling Register() will overwrite any previously existing logger with the specified name. (Note: Register() operates on the Log.Name() value rather than a user-supplied logger name.)
• Load() has also been added as a top-level function to return previously Register()'d root loggers or root loggers that were created via Get(). Load() returns a tuple containing the named logger and a boolean value indicating whether or not the logger exists.
• Must* constructor functions have been removed.
• The default formatter has been changed from the formatter returned via NewDefaultFormatter and replaced with the one assembled by NewFastFormatter. On average, this has improved log processing latency between 30 to 40% and up to 60% for some workloads (e.g. log chaining). This formatter has been backported to the v1.x branch.
• v2.x now requires Go v1.16 or greater.

Notes about v1

go/logging v1.x will likely no longer see further updates with the introduction of the v2 branch. However, there are some changes or additions that are far too important to allow v1.x to languish.

As an example, with the introduction of v2, a new format handler was introduced early in its development cycle that substantially reduced processing time for generated log messages. This change was deemed far too important to skip the v1.x branch and thus was backported. Both branches now enjoy similar speed ups.

If you're comfortable enough using v1.x, I would strongly suggest that you continue using it. If you're authoring new code, I would suggest using v2.x instead. Both branches will be maintained at least into the near term; however, if a v3 branch ever surfaces, v1.x will likely be deprecated.

Quick Start

This README is only intended as a quick start guide. For more detailed usage documentation, please refer to the docs/ directory in this repository (or in your local copy!).

This version of the quick start guide covers the v2.x branch. For documentation referencing v1.x, please see the appropriate branch.

Prerequisites

Obtain go/logging (v2 API):

go get git.destrealm.org/go/logging/v2

Or grab the v1 API (see the appropriate documentation elsewhere):

go get git.destrealm.org/go/logging

Getting Started

To create and use a logger:

import "logging"

func main() {
    // Returns the default root logger.
    logger := logging.Logger()

    // Do something.

    logger.Error("an error occurred!")
}

To change the default logging level (logging.Warning):

logger := logging.Logger()

// Set the log level to info.
logger.SetLevel(logging.Info)

Defined Levels

go/logging defines the following levels: Fatal, Error, Warning, Notice, Info, and Debug. For each of these levels there is an exported log function with the suffix f (e.g. Error and Errorf) for logging C-style formatted strings and their arguments.

Additionally, go/logging also aliases common abbreviations; e.g., Warning() and Warn() are identical as are Information() and Info(). For compatibility with the standard library log, go/logging also defines Panic and Fatal functions.

Output Control

Backends

go/logging supports any backend that exposes an io.Writer interface. By default, go/logging will output entries to io.Stdout. To change this, call SetBackend() on any already-created Log instance.

Note: SetBackend() always wraps io.Writer instances with a synchronous wrapper to reduce the likelihood that output from multiple sources will be interleaved. If this is not desirable for whatever reason, SetRawBackend() will apply the specified backend without wrapping it in synchronization primitives.

It is also possible to configure Log instances with different backends depending on level. Here, we discuss examples of common use cases.

Change Backend

To change the backend to a file:

fp, _ := os.OpenFile("filename", os.O_WRONLY|os.O_TRUNC, os.FileMode(0644))
logger.SetBackend(fp)

To change the backend for certain levels:

fp, _ := os.OpenFile("filename", os.O_WRONLY|os.O_TRUNC, os.FileMode(0644))
logger.SetBackendForLevel(logging.Error)
logger.SetBackendForLevel(logging.Warning)

Note: This will only change the backend for the specified level.

Clear Backend

Sometimes it may be necessary to clear the backend associated with a specific level. This will force the logger for that level to revert to the default backend:

logger.ClearBackend(logging.Error)

Wrapper Support

As with SetBackend(), SetBackendForLevel() also includes an option to bypass the io.Writer synchronization by calling SetRawBackendForLevel().

All wrapped writers are of type *logging.SyncWriter and implement the Wrapper interface. If SetBackend() receives a backend that already implements logging.Wrapper the writer will not be re-wrapped.

Chaining

go/logging provides an API for chaining multiple loggers together. This is mostly useful if you intend to direct output to different io.Writers depending on the configured logging level. For example, it's possible to output Info logs to STDOUT, Error logs to a specific error file, and everything to its own file for later analysis (although the recommended way to accomplish this is to use SetBackendForLevel()).

Chained loggers must start with the most specific log and complete the chain with the least restrictive. In our example, above, we might construct the logger chain as:

applog := logging.Logger()
applog.SetBackendForLevel(logging.Info, os.Stdout)
applog.SetBackendForLevel(logging.Error, errorLog) // errorLog is a file pointer

everything := logging.Logger()
everything.SetBackend(everythingLog) // everythignLog is a file pointer

appLog.Chain(everything)

Chaining is intended primarily for use cases where a logger cascade is needed such that all loggers in the chain receive the same input. For v2 go/logging, chaining is a means of tying together otherwise unconnected root loggers.

Inheritance

go/logging supports creating a logger hierarchy where a child logger inherits all attributes (including backend configuration) from its parent. Inheritance is a different concept from chaining; whereas chain logs can be completely unrelated and comprise different root loggers, inheritance affects direct descendants of a particular log. For example, if a parent logger has its level set to logging.WARN, all descendants of that log will inherit the parent level until the child logger's log level is changed.

Creating an inheritance chain can be done in one of two ways. The first, by a method on the logger instead itself:

logger := logging.Logger()
webLogger := logger.Inherit("web")
requestLogger := webLogger.Inherit("request")

And the second via a naive retrieval method:

logger := logging.Logger()
requestLogger := logger.GetLogger("web.request")

// ...or...

requestLogger := logger.Inherit("web.request")

As such, inherited loggers can more easily be obtained via dot notation or via the individual loggers themselves:

logger := logging.Logger()

// As above.
requestLogger := logger.GetLogger("web.request")

// Or...
webLogger := logger.GetLogger("web")
requstLogger := webLogger.GetLogger("request")

Intermediate loggers are always created as needed. This is a notable departure from the v1 API!

Iteration and Inheritance

It's possible to iterate through a list of loggers using inheritance and iteration. Each logger has Iterate() and IterateAll() methods that accept a function consuming a single *Log parameter that will apply this function to each child logger.

Iterate() and IterateAll() differ in their intent: In the former, Iterate(), all direct descendants of a *Log will be traversed and is a fast way to apply changes to multiple children of a given logger; in the latter, IterateAll(), the entire inheritance tree (children, children of children, ad nauseum) is traversed.

For example:

// Iterate() usage.
logger := logging.Logger()
logger.Iterate(func(lg *logging.Log) {
    // Discard all output for child loggers.
    lg.SetBackend(logging.Discard)
})

If iteration over all descendants is required, use IterateAll() instead as Iterate() only applies itself to the immediate descendants of the current logger:

// IterateAll() usage:
logger := logging.Logger()
logger.IterateAll(func(lg *logging.Log) {
    // Discard all output for child loggers.
    lg.SetBackend(logging.Discard)
})

Advanced Topics

This section covers the less common use cases of go/logging that may be handy for specific edge cases or for implementers who require more control over the loggers' behavior.

Formatters

go/logging exposes some primitives for crafting custom formatters. These formatters are simple functions accepting an io.Writer, a logging.ImmutableLog, and a *logging.Message. Formatters can be used to transform log messages before they are written to their final location or, in a particularly creative use case, could be used to intercept log messages to manipulate them before writing (the passed-in io.Writer need not be used). Currently, three formatters are provided: The "default" formatter, which provides simple text template replacement; the template formatter, which uses Golang's text/template to format output; and the "fast" formatter which uses a function stack derived from an output template to cache intermediate logging output and is, on average, 30-50% faster than the default formatter. Future versions will likely remove the current default formatter entirely.

To create a formatter, it is necessary to register a function that matches the logging.FormatterFunc signature with a logger's SetFormatter() function. Formatters then receive the arguments: An io.Writer instance, a reference to the current logger instance of type logging.Log, and a *logging.Message pointer.

logging.Message contains all of the metadata associated with a log entry: The timestamp, the error level, and its message.

When constructing a new formatter, it's important to keep in mind a few things:

• The io.Writer instance is the output channel that interfaces directly with the backend; this may be a file, STDOUT, or anything else implementing the io.Writer interface. This writer may or may not be buffered, so you will need to buffer your output internally if you plan on making several small writes per log entry. Doing so will improve throughput in most circumstances.
• The io.Writer formatter functions receive will be encapsulated, by default, within a logging.SyncWriter which satisfies the logging.Wrapper interface except in cases where the client code has configured the backend via SetRawBackend() or SetRawBackendForLevel(). If you are uncertain if the io.Writer you have received is wrapped, you will need to manually check the writer's interface.
• If you need to do initialization of an object, do so within a factory function that creates an instance of your formatter and return the logging.FormatterFunc from that. Be aware that your formatter will be invoked for every call to each logging method.
• You can use the formatter if you need to translate for a specific backend, such as when targeting syslog or systemd-journald. Though, bear in mind that you may be better off using chaining combined with a custom backend. Future versions of go/logging will contain a sub-repository for custom backends.
• logging.Log contains an internal options map (*Log.Options) that isn't used for anything go/logging-specific. Instead, it is intended as a metadata store for custom formatters. If you need to persist something per-logger, you should add it to the Options map and then access this from within your formatter. Be aware that this map is not protected by any synchronization primitives.

Metadata Logging

go/logging supports logging additional information besides simply writing strings to a particular backend. As of v2.1+, go/logging now includes support for printing metadata and file/line number information alongside the log message. This feature is opt-in per logger (it's slow; do not use it in high traffic code paths), returns a logging.Log copy, and will print the contents of a given string-keyed map after the log message. To use this feature, call either WithMeta or WithMetaOpts (the latter reserved, presently, for toggling line numbers):

logger := logging.Get("example")
meta := map[string]interface{}{
  "path": "/tmp/failed.txt",
  "code": 42,
}

mlog := logger.WithMeta(meta)
mlog.Error("failed to open file")

// Prints:
//
// [ERRO] - 2021-11-30 02:33:42 MST - failed to open file - [<path=/tmp/failed.txt> <code=42>]

Or substitute WithMeta with WithMetaOpts:

mlog := logger.WithMetaOpts(meta, logging.MetaOptions{LineNo: true})
mlog.Error("failed to open file")
// Prints:
//
// [ERRO] - 2021-11-30 02:33:42 MST - failed to open file - [<path=/tmp/failed.txt> <code=42>] - main.go:483

At present, the logging.MetaOptions struct allows for toggling callers' line numbers on or off and for shifting line number calculation to the previous caller (by setting MetaOptions.Caller to logging.PreviousCaller). At present there is no means of controlling the template used to generate the formatted output for processing metadata. Moreover, metadata cannot be changed without additional calls to WithMeta* or in the event the original map is changed.

Template Loggers

It may be necessary, on occasion, to configure a default state for all subsequently created root loggers. As of v2.3.0 this is now possible via the logging.SetTemplate() global function which will configure the specified logger as a "template" logger from which all other loggers derive their default settings.

To set the package template to the default root logger:

logging.SetTemplate(logging.Logger())

To set the package template to a custom logger:

logger := logging.Get("application")
logger.SetLevel(logging.Info)
logging.SetTemplate(logger)

// All loggers will now have a level of logging.Info:

lg := logging.Get("another-application")

To clear the template logger, simply call logging.ClearTemplate():

// Set the template logger to the default root logger.
logging.SetTemplate(logging.Logger())

// Do some things and then reset the template to nil:

logging.ClearTemplate()

Once ClearTemplate is called, the template logger is reset and all loggers created will use the package defaults.

A word of warning: Because all loggers will inherit their defaults from the global template, care should be taken to ensure that only those attributes that make sense to be inherited are changed. This could provoke unexpected behaviors; use wisely!

Panic() and Exit()

With the introduction of v2.0, the package mechanisms that allowed for overriding the behavior of Panic() and Exit() when calling any of the Panic* and Fatal* logger methods were inadvertently removed and never re-added. As of v2.3+, these have now made a reaappearance and can be overridden at the package level:

// Set Panic() to a no-op to disable panics when logged:
logging.Panic = func(string){}

// Set Exit() to panic instead:
logging.Exit = func(s string){
  panic(s)
}

Similarly, Panic* and Fatal* can be overridden per-logger rather than at the package level (this is preferred):

logger := logging.Get("panic-and-exit-example")

// Set Panic* to a noop.
logger.SetOnPanic(func(string){})

// Set Fatal* to exit instead.
logger.SetOnExit(func(s string){ panic(s) })

LICENSE

go/logging is licensed under the fairly liberal and highly permissive NCSA license. We prefer this license as it combines the best of BSD and MIT licenses while providing coverage for associated documentation and other works that are not strictly considered original source code. Consequently, all go/logging documentation is likewise covered under the same license as the codebase itself.

As with BSD-like licenses attribution is required.

Copyright (c) 2015-2021.