config

config - Go configuration values using structs

• config - Go configuration values using structs
  • Quick Start
  • Configuration Value Names
    • Environment Variables
  • Required VS Optional Settings
  • Nested Structs
    • Recursive Definitions
  • Generating Configuration Samples
  • Extending The Project
    • Custom Loaders
      • Hot Reloads
    • Custom Sample Generators
  • Developing
    • Make targets
    • Diving Deeper
  • License

Quick Start

package main

import (
    "context"
    "fmt"

    "github.com/stackopsd/config"
)

// Define a struct that includes all the configuration values you need.
type Settings struct {
    DBUser string
    DBPassword string
    UseTLS bool
}

func main() {

    ctx, cancel := context.WithCancel(context.Background())
    defer cancel()

    // Create an instance of that configuration struct.
    var settings Settings

    // Create a Loader from a file. Values in the file will have any ENV vars
    // interpolated such as `key: ${VARIABLE}`.
    loader := config.NewFileLoader("/path/to/some/config.json_or_yaml")

    // Use the Loader to populate the configuration struct.
    if err := config.Load(ctx, loader, &settings); err != nil {
        panic(err)
    }

    // Print some loaded values.
    fmt.Println(settings.DBUser, settings.DBPassword, settings.UseTLS)
    // Print a YAML configuration sample from the configuration struct.
    fmt.Println(config.RenderYAML(settings))
}

# An example YAML file that would work with the above.
db_user: "myUser"
db_password: "${DB_PWD}" # ENV interpolation is done automatically
use_tls: true

Configuration Value Names

The expectation is that all structs being loaded will conform to the Effective Go suggestion of using MixedCase for field names. The system will automatically convert field names into a human readable snake_case format. The system attempts to chunk names in an intuitive way. Here are some examples conversions:

-   Value => value
-   SomeValue => some_value
-   DNSResolver => dns_resolver
-   HTTPServerAddress => http_server_address
-   HTTP2Enabled => http2_enabled
-   HTTPV1Enabled => httpv1_enabled
-   Http2ServerAddress => http2_server_address

Generally, the chosen name should be easily guessable. The actual algorithm is based on several aspects of a string. Word detection is driven by the use of upper, lower, and digit characters. The following table defines the word selection:

• Given l=lower case, U=upper case, 0=digit
• UU => continue word
• lU => complete word ending in l and start word beginning with U
• Ul =>
  • if last letter then
    • continue word
  • else if next letter is U then
    • continue word
  • else if word has more than one letter then
    • complete word ending before U and start new word beginning with Ul
  • else
    • continue word
• ll => continue word
• l0 => continue word
• U0 => continue word
• 00 => continue word
• 0l => continue word
• 0U => complete word ending in 0 and start word beginning with U

Environment Variables

Using the LoaderENV puts some additional restrictions on how the variables are named. Structured formats like YAML and JSON have built-in mechanisms for describing structure that don't exist in environment variables. Notably, environment variables:

• Do not have list or array notation
• Do not have object or map notation
• Are only able to store strings
• Have names that are restricted to ASCII characters [a-zA-Z0-9_]

In order to make loading complex data from the environment possible we've had to be a little creative with the variable naming scheme:

• The same conversion rules listed above are used to convert struct fields into variable names.
• The hierarchy of values is determined by use of double underscore (__)
• Slice values are denoted with integer sections.
• All variable names are converted to upper case.

To illustrate, consider the following configuration struct:

type SubConfig struct {
    MapValue map[string]int
}
type Config struct {
    StringValue string
    IntValue int
    SliceValue []SubConfig
}

This struct manages several types, a slice of nested structs, and each nested struct contains a map. This would be somewhat easy to express in a syntax like YAML:

string_value: "a"
int_value: 1
slice_value:
    - map_value:
        b: 1
        c: 2
    - map_value:
        d: 3
        e: 4

To express the same configuration in environment variables looks like:

APP__STRING_VALUE="a"
APP__INT_VALUE=1
APP__SLICE_VALUE__0__MAP_VALUE__B=1
APP__SLICE_VALUE__0__MAP_VALUE__C=2
APP__SLICE_VALUE__1__MAP_VALUE__D=3
APP__SLICE_VALUE__1__MAP_VALUE__E=4

The key notes are how the double underscore (__) is used as a separator for both keys and array offsets and how offset values are numeric only. Expressing complex configuration with environment variables will quickly become verbose but it is possible to do if required. You are recommended to use the RenderEnv helper to print an initial set of values and then modify them as needed rather that writing every variable by hand.

Unlike with the YAML and JSON documents, you should avoid variable names that are not within the ASCII range and limited to the character set [a-zA-Z0-9_]. If you happen to have a shell that supports full unicode for environment variables then you may try to use non-ASCII characters but this is not a supported feature. If you need unicode support then please use the YAML or JSON loaders instead.

Required VS Optional Settings

Everything is required by default. You must use pointer types anywhere a value is optional. For example:

type Settings struct {
    Required string
    Optional *string
}

Using a pointer type causes the system to attempt to look up the value before setting it. If no value is found for the setting then the pointer is left as nil or whatever existing value was set for the field. If there is a value provided from the loader then the pointer value will be set to a non-zero value.

Nested Structs

Structs may be nested either by embedding or by having a named field in the configuration struct:

type DatabaseSettings struct {
    User string
    Password string
}

type RuntimeSettings struct {
    Address string
    Port int
}

type Settings struct {
    RuntimeSettings // Embedded. All fields are "hoisted" to the top level.
    DB DatabaseSettings // Nested. All fields are namespaced under db.
}

Embedding works as you would in Go and exposes all of the attributes contained in the embedded struct through the parent. This means that address and port would not need prefixes to load. In contrast, defining struct fields will cause any loader to search for a subtree. For example, here is a YAML file that satisfies the above configuration:

address: localhost
port: 8080
db:
    user: "myUser"
    password: "${DB_PWD}"

Recursive Definitions

This library supports recursive configuration definitions by using a pointer type. For example:

type Settings struct {
    Value string
    More *Settings
}

This follows the exact same semantics as other optional fields. Using a non-pointer type for recursion will result in infinite recursion.

Generating Configuration Samples

Given any configuration struct, it is possible to generate sample configuration files. The library currently has support for generating sample YAML and ENV text. Sample JSON is a planned feature. The RenderYAML method consumes any configuration struct and emits a valid YAML document that can be used for either documentation or as a base configuration to modify. Additionally, the description field tag is recognized for help output in the YAML document:

type DatabaseSettings struct {
    User string `description:"The DB username."`
    Password string `description:"The DB password."`
}

type RuntimeSettings struct {
    Address string `description:"The address on which to listen."`
    Port int `description:"The port on which to listen."`
}

type Settings struct {
    RuntimeSettings
    DB DatabaseSettings `description:"Database related settings."`
}

func main() {
    conf := &Settings{
        RuntimeSettings: RuntimeSettings{
            User: "root", // The default username
            Password: "root", // The default password
        },
        DB: DatabaseSettings{
            Address: "0.0.0.0", // The default address on which to listen.
            Port: 8080, // The default port on which to listen.
        },
    }
    fmt.Println(config.RenderYAML(context.Background(), conf))
}

The above will print something like:

user: "root" # (string): The DB username.
password: "root" # (string): The DB password.
db: # (DatabaseSettings): Database related settings.
    address: "0.0.0.0" # (string): The address on which to listen.
    port: 8080 # (int): The port on which to listen.

The RenderENV method works similarly but also takes in a constant prefix value that is applied to all output. This matches the LoaderENV which has the same behavior.

All default values are printed as they are set when the struct is passed to the render method. Each field is annotated with the description tag if it is present and the expected type of the field. Using the same code block from the YAML example we could call:

fmt.Println(config.RenderENV(context.Background(), "APP", conf))

and get something like:

APP__USER="root" # (string): The DB username.
APP__PASSWORD="root" # (string): The DB password.
APP__DB__ADDRESS="0.0.0.0" # (string): The address on which to listen.
APP__DB__PORT=8080 # (int): The port on which to listen.

Extending The Project

Custom Loaders

A Loader is any source of configuration data. Each loader implements the following interface:

// PathSegment is a lookup element that is used as part of a full
// sequence to extract a value from a Loader. String path segments are
// considered key-based lookups and numeric segments are considered positional
// lookups. Positional lookups in this case are currently limited to offset
// positions of array/list/slice/etc. data structures. All other lookups should
// leverage the key-based lookups.
type PathSegment interface {
	// Key returns the internal value if it is a string. If the internal value
	// is not a string then the boolean value returns as false.
	Key() (string, bool)
	// Offset returns the internal value if it is an integer. If the internal
	// value is not an integer then then boolean value returns false.
	Offset() (int, bool)
}

// Path is an ordered sequence of PathSegment values that are used to look up
// values in a Loader.
type Path []PathSegment

// Loader is any data source from which configuration values might be drawn.
type Loader interface {
	// Load accepts a hierarchical path to a configuration value and returns the
	// raw form. The output must be interpreted by the caller in some way to
	// make it meaningful. The boolean value indicates whether or not the value
	// was found in the loader or if some default/nil value is returned.
	// Additionally, the error value can be non-nil if there was an exception
	// fetching data.
	//
	// Non-leaf values must be returned as either a map type or a slice type.
	Load(ctx context.Context, path Path) (interface{}, bool, error)
}

For convenience, there is an included LoaderMap that can wrap any map[string]interface{}. If your data source can be converted into a map[string]interface{} then you get your integration for "free" by calling NewLoaderMap(myMap).

Dynamic and remote loaders are also possible so long as they implement the Loader interface. One major point of note is that a call to Load that selects either an array or a sub-tree must return those as a slice or map type respectively. This behavior is required by the library in order to correctly load complex structures such as slices of objects or maps of slice of objects.

Hot Reloads

With respect to dynamic loaders, it is also important to call out that this library does not provide any kind of built-in "hot reload" feature. It is possible to implement this but it will require a few extra steps as the primary intent of the library relates to static configurations. Here is some code for inspiration if you are looking to implement a hot reload:

package main

import (
    "fmt"

    "github.com/stackopsd/config"
)

type Config struct {
    Example string `description:"An example field"`
}

func main() {
    conf := &Config{}

    // We're using a variable map here to simulate a dynamic configuration
    // source. We will be able to alter the values and see the effects.
    src := map[string]interface{}{
        "example": "one",
    }
    loader := config.NewLoaderMap(m)

    if err := config.Load(context.Background(), loader, conf); err != nil {
        panic(err)
    }
    fmt.Println(conf.Example) // "one"

    src["example"] = "two"
    if err := config.Load(context.Background(), loader, conf); err != nil {
        panic(err)
    }
    fmt.Println(conf.Example) // "two"
}

The key point is that loading values operates on a pointer to the configuration struct. This makes it possible to re-load new values into the existing instance as often as needed. It is important to note that this introduces a data race if used in a system where the values are being loaded and read concurrently. You will need to implement your own concurrency controls to prevent issues.

Custom Sample Generators

The recommended way to create a new sample configuration text generator is to implement the following interface:

// Visit is a data container that includes all relevant data known about a
// value given to a visitor.
type Visit struct {
	// Name of the field that the visit represents.
	Name PathSegment

	// Tags includes any struct field annotations.
	Tags reflect.StructTag

	// Type is the reflect.Type of the resolved value. This value will never
	// be a pointer kind because pointers will be unrolled to the actual value
	// type.
	Type reflect.Type

	// Setter is a reflect.Value that represent the actual value of the field.
	// This type will match whatever type is recorded by Field. This is the
	// value that should be modified if any Value.Set() calls are made.
	Setter reflect.Value

	// Value is a reflect.Value that represents the resolved value of the field.
	// The type of this value will match the Type field. It may not be
	// addressable or settable under certain circumstances and can be the value
	// of `nil`.
	Value reflect.Value
}

// Visitor implementations are used in conjunction with Walk to process a
// configuration struct. The fields of the struct are processed in reflection
// order which is usually the order in which the fields are defined within the
// struct. Embedded types are unrolled depth-first. Any other struct types
// encountered are passed to VisitStruct. No recursive processing of struct
// types other than embedded types is performed unless the visitor makes an
// explicit call to Walk again. That is, visitors are responsible for engaging
// in recursive processing if they need to visit fields on a nested struct.
type Visitor interface {
	VisitTime(ctx context.Context, v Visit) error
	VisitSlice(ctx context.Context, v Visit) error
	VisitMap(ctx context.Context, v Visit) error
	VisitStruct(ctx context.Context, v Visit) error
	VisitPrimitive(ctx context.Context, v Visit) error
}

An implementation of the Visitor interface can be used in conjunction with the Walk method to process all fields of the configuration struct. See the included VisitorRenderYAML implementation for an example of how to both handle special data types, like time.Duration, and how to correctly handle recursively calling Walk for nested structs.

Developing

Make targets

This project includes a Makefile that makes it easier to develop on the project. The following are the included targets:

• fmt

  Format all Go code using goimports.

• generate

  Regenerate any generated code. See gen.go for code generation commands.

• lint

  Run golangci-lint using the included linter configurations.

• test

  Run all unit tests for the project and generate a coverage report.

• integration

  Run all integration tests for the project and generate a coverage report.

• coverage

  Generate a combined coverage report from the test and integration target outputs.

• update

  Update all project dependencies to their latest versions.

• tools

  Generate a local bin/ directory that contains all of the tools used to lint, test, and format, etc.

• updatetools

  Update all project ools to their latest versions.

• vendor

  Generate a vendor directory.

• clean/cleancoverage/cleantools/cleanvendor

  Remove files created by the Makefile. This does not affect any code changes you may have made.

Diving Deeper

This library makes heavy use of reflection to perform its functions and has to handle a number of reflection related edge cases. If you are interested in reflection, or digging into the internals of this library, then the Walk method and the VisitorLoader struct are the best places to start.

Walk is a heavily annotated function that iterates over struct fields and calls a Visitor to actually process the fields. This method highlights many of the complexities of heavily reflected code and documents the thought process that went into each decision for how things are handled. The VisitorLoader handles the complexity of loading values from a Loader and actually setting the field values on the configuration struct. This code handles a smaller set of "gotchas" related to reflection and correctly setting different kinds of values.

License

Copyright 2019 Kevin Conway

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.