igconfig

igconfig

This package can be used to load configuration values from a configuration file, environment variables, Consul, Vault and/or command-line parameters.

Install

go get github.com/worldline-go/igconfig

Example

cfg tag value are case insensitive and weakly dash/underscore so Network-Name, network_name, NeTWoK-NaMe and NeTWoKNaMe are same in both tag and configs.

NOTE if secret tag not found it will check cfg tag after that it will check variable's name.

type Config struct {
	NetworkName string `cfg:"networkName" env:"NETWORK_NAME"`
	// application specific vault
	DBSchema     string `cfg:"dbSchema"     env:"SCHEMA"       default:"transaction"`
	DBDataSource string `cfg:"dbDataSource" env:"DBDATASOURCE" log:"false"`
	DBType       string `cfg:"dbType"       env:"DBTYPE"       default:"pgx"`

	CustomConfig map[string]interface{} `cfg:"customConfig"`
}

// ---

var cfg Config

if err := igconfig.LoadConfig("myappname", &cfg); err != nil {
    log.Fatal().Err(err).Msg("unable to load configuration settings.")
}

Also check example:
Examples section
Example usage _example/readFromAll/main.go

Description

There is only a single exported function:

func LoadConfig(appName string, config interface{}) error

or if specific loaders needed:

func LoadWithLoaders(appName string, configStruct interface{}, loaders ...loader.Loader) error

There are also context accepted functions LoadConfigWithContext, LoadWithLoadersWithContext.

• appName is name of application. It is used in Consul and Vault to find proper path for variables also file name for file loader.
• config must be a pointer to struct. Otherwise, the function will fail with an error.
• loaders is list of Loaders to use.

Config struct

All exported fields of this structure will be checked and filled based on their tags or field names.

The field type is taken into consideration when processing parameters.

If the given value cannot be converted to the field's type, an error will be returned. Failing to provide proper value for a type is human-made error, which means that something is not right.

Fields can be given a tag with identifier "default", where the value of the tag will then be used as the default value for that field. If the default value cannot be converted to the field's type, an error will be returned.

Config struct can have inner structs as a fields, but not all Loaders might support them. For example Consul and Vault support inner structs, while Env and Flags don't.

Tags

Config structs can have special tags for fine-grained field name configuration.

There are no required tags, but setting them can improve readability and understanding.

cfg

cfg tag is fallback tag when no Loader-specific tag can be found. As such defining only this tag can be enough for most situations.

env

env tag specifies a name of environmental variable to get value from.

cmd

cmd tag is used to set flag names for fields.

secret

secret tag specifies name of field in Vault that should be used to fill the field.
If not exist it use as struct's field name.

default

default is special tag.

Unlike other tags it does not point to a place from which value should be taken, but instead it itself holds value.

default:"data" will mean that value of string field that has this tag will be data.

This tag is optional

Loaders

Loaders are actual specification on how fields should be filled.

igconfig provides a simple interface for creating new loaders.

Below is a sorted list of currently provided loaders that are included by default(if not stated otherwise)

Change order of loaders and configurations:

// this is the default loaders; change configurations and order or eliminate some loaders
loaders := []loader.Loader{
	&loader.Default{},
	&loader.Consul{},
	&loader.Vault{},
	&loader.File{},
	&loader.Env{},
	&loader.Flags{},
}

// read configurations with custom loaders
if err := igconfig.LoadWithLoaders("test", &conf, loaders...); err != nil {
    log.Fatal().Err(err).Msg("unable to load configuration settings.")
}

Default

This loader uses default tag to get value for fields.

Consul

Loads configuration from Consul and uses map decoder with cfg tag to decode data from Consul to a struct.

If not give CONSUL_HTTP_ADDR as environment variable, this config will skip!

For connection to Consul server you need to set some of environment variables.

While it is possible to change decoder from YAML to JSON for example it is not recommended if there are no objective reasons to do so. YAML is superior to JSON in terms of readability while providing as much ability to write configurations.

For better configurability configuration struct might include cfg tag for fields to specify a proper name to bind from Consul, if this tag is skipper - lowercase field name will be used to bind.

For example:

type Config struct {
    Field1 int `cfg:"field"`
    Str struct {
        Inner string
    }
}

will match this YAML

field: 50
str:
  inner: "test string"

Vault

Loads configuration from Vault and uses MapDecoder to decode data from Vault to a struct.

First Vault loads in finops/data/generic path and after that process application's configuration in finops/data/<appname> path.

generic path can have inner path, vault loader combine them.

To use additional path to load with appname set AdditionalPaths value in the loader.
Default value is [{Map: "", Name: "generic"}], Map is a wrapper for read value in key-value format, generic doesn't have map value so it will apply what read and append directly in our config.

To read more than one path, just append your path to the loader.VaultSecretAdditionalPaths slice.

Use Map value to wrap readed data with a key and Name is a path of configuration.

loader.VaultSecretAdditionalPaths = append(
    loader.VaultSecretAdditionalPaths,
    loader.AdditionalPath{Map: "loadtest", Name: "loadtest"},
)

If not given any of VAULT_ADDR, VAULT_AGENT_ADDR or CONSUL_HTTP_ADDR as environment variable, this config will skip!

If CONSUL_HTTP_ADDR exists, it uses Consul to get vault address.

For authentication, you should set VAULT_ROLE_ID and VAULT_ROLE_SECRET environment variables.

File

TOML, YAML and JSON files supported, and file path should be located on CONFIG_FILE env variable.
If that environment variable not found, file loader check working directory and /etc path with this formation <appName>.[toml|yml|yaml|json] (if there is more than appName with different suffixes, order is toml > yml > yaml > json).
The appName used as the file name is not the full name, only the part after the last slash. So if your app name is transactions/consumers/internal/apm/, the loader will try to load a file with the name apm.

The key is checked against the exported field names of the config struct and the field tag identified by cfg.
If a key is matched, the corresponding field in the struct will be filled with the value from the configuration file.

NOTE: if cfg tag not exists, it is still read values in file and match struct's field name!
Don't want to read a value just delete it in your config file or add cfg:"-".

FileLoader editable, you can add your own decoder or new file format or order of file suffixes.

Environment variables

For all exported fields from the config struct the name and the field tag identified by "env" will be checked if a corresponding environment variable is present. The tag may contain a list of names separated by comma's. The comparison is upper-case, even if tag specifies lower- or mixed-case. Lower-case environment variables are ignored.

Once a match is found the value from the corresponding environment variable is placed in the struct field, and no further comparisons will be done for that field.

Set value in inner struct:

type Config struct {
    Inner Inner
}

type Inner struct {
	GetENV       string `env:"TEST_ENV"` // env:"test_EnV" same as TEST_ENV
	// GetENV       string `cfg:"TEST_ENV"` // cfg tag also usable
}

To set GetENV value use INNER_TEST_ENV environment value. Or you can change INNER name with env tag.

type Config struct {
    Inner Inner `env:"IN"`
}

Now value use IN_TEST_ENV

NOTE: if env tag not exists, it will check cfg tag and if both not exists, it will check struct's field name as uppercase.

Flags (command-line parameters)

For all exported fields from the config struct the tag of the field identified by "cmd" will be checked if a corresponding command-line parameter is present. The tag may contain a list of names separated by comma's. The comparison is always done in a case-sensitive manner. Once a match is found the value from the corresponding command-line parameter is placed in the struct field, and no further comparisons are done for that field. For boolean struct fields the command-line parameter is checked as a flag. For all other field types the command-line parameter should have a compatible value. Parameters can be supplied on the command-line as described in the standard Go package "flag".

Log with context

Set a new zerolog logger and attach to the context, igconfig will use that context's logger.

// set new schemas for log
logConfig := log.With().Str("component", "config").Logger()
// replace context.Background() with own context
logCtx := logConfig.WithContext(context.Background())

// call igconfig with context
if err := igconfig.LoadConfigWithContext(logCtx, "test", &conf); err != nil {
    log.Ctx(logCtx).Fatal().Err(err).Msg("unable to load configuration settings.")
}

Print configuration

secret tag is disabled to print but if you want to print it add aditional option to secret called loggable or log.

Info string     `cfg:"info" secret:"info,log"` // Set secret's log option

Or you can set general log to manage it all tags in releated field.

User string     `cfg:"user" secret:"user" log:"true"` // Set general log

conf := config.AppConfig{} // set up config value somehow
// log is zerolog/log package
log.Info().
    EmbedObject(Printer{Value: conf}).
    Msg("loaded config")

Examples

(
export CONFIG_FILE=_example/readFromAll/dataFile/test.yml
go run _example/readFromAll/main.go
)

docker run -it --rm --cap-add=IPC_LOCK --name=dev-vault -p 8200:8200 vault

# export address for http
export VAULT_ADDR="http://127.0.0.1:8200"
# login with root token (appears in docker output)
vault login <token>
# unseal it
vault operator unseal <unsealkey>
# create kv secret engine
vault secrets enable -path=finops -version=2 kv
# create policy to read
{
cat <<EOF
path "finops/*" {
  capabilities = ["read", "list"]
}
path "finops/data/generic/super-secret" {
  capabilities = ["deny"]
}
EOF
} | vault policy write finops-read -

# create a approle with policy and enable connection without secret_id
vault auth enable approle
vault write auth/approle/role/my-role bind_secret_id=false secret_id_bound_cidrs="127.0.0.0/8,172.17.0.0/16" policies="default","finops-read"
# learn role-id
ROLE_ID=$(vault read -field=role_id auth/approle/role/my-role/role-id)

# fill some data
vault kv put finops/generic/keycloack @_example/readFromAll/dataVault/generic_keycloack.json
vault kv put finops/generic/super-secret @_example/readFromAll/dataVault/generic_supersecret.json
vault kv put finops/test @_example/readFromAll/dataVault/test.json
vault kv put finops/loadtest @_example/readFromAll/dataVault/loadtest.json

(
export VAULT_ADDR="http://localhost:8200"
export VAULT_ROLE_ID=${ROLE_ID}
# export CONSUL_HTTP_ADDR="consul:8500"
export MIGRATIONS_TEST_ENV="testing_testing_1234"
go run _example/readFromAll/main.go
)

docker run -it --rm --name=dev-consul --net=host consul:1.10.4

export CONSUL_HTTP_ADDR="localhost:8500"
go run _example/readFromAll/main.go

export CONSUL_HTTP_ADDR="localhost:8500"
go run _example/dynamicConsul/main.go

Development

go test --race -cover ./...

mkdir _out
go test -cover -coverprofile cover.out -outputdir ./_out/ ./...
# Auto open html result
go tool cover -html=./_out/cover.out
# Export HTML
# go tool cover -html=./_out/cover.out -o ./_out/coverage.html

# yaml to json
cat _example/readFromAll/dataFile/test.yml | yq

# json to yaml
cat _example/readFromAll/dataVault/generic_keycloack.json | yq -y