xload

package module
v0.7.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 17, 2024 License: MIT Imports: 11 Imported by: 4

README

xload - Load Anything

xload is a struct first data loader that simplifies the process of loading data from various sources into Go structs. It is designed to handle diverse use cases, from configuration loading to internationalization labels and even experiments.

Why xload?

We often encounter the need to load data from various sources, such as environment variables, command-line arguments, files, or remote configurations. This data may include configuration settings, experiment parameters, internationalization labels, design assets and more. Implementing these data loading methods can lead to boilerplate code, making the codebase hard to maintain and less readable.

xload is inspired by the popular go-envconfig library but extends it with custom data loaders and allows customizing struct tags, making it flexible for different use cases.

Installation

go get -u github.com/gojekfarm/xtools/xload

How xload Works

At a high level, xload takes a Go struct annotated with tags and populates it with data from any source. The sources can be anything that implements the Loader interface. By using xload, you can separate the data loading process from how and where the data is used.

Loading from Environment Variables

Let's take a look at a simple example of how to use xload to load data from environment variables into a struct:

type AppConfig struct {
    // your application config goes here
}

func DefaultAppConfig() AppConfig {
    return AppConfig{
        // set default values here
    }
}

func main() {
    ctx := context.Background()
    cfg := DefaultAppConfig()

    err := xload.Load(ctx, &cfg)
    if err != nil {
        panic(err)
    }

    // use cfg
}
Custom Types

With xload, you can also use custom types, such as time.Duration, to make the configuration more descriptive and easier to understand:

For example, timeouts need not be ambiguous int values, they can be typed time.Duration values:

type AppConfig struct {
    Timeout time.Duration `env:"TIMEOUT"` // TIMEOUT=5s will be parsed as 5 seconds
}

This helps you do away with naming conventions like *_TIMEOUT_MS or *_TIMEOUT_SECS.

Nested Structs

xload supports nested structs, slices, maps, and custom types making it easy to group, reuse and maintain complex configurations.

type HTTPConfig struct {
    Port int `env:"PORT"`
    Host string `env:"HOST"`
}

type AppConfig struct {
    Service1 HTTPConfig `env:",prefix=SERVICE1_"`
    Service2 HTTPConfig `env:",prefix=SERVICE2_"`
}

Inbuilt Loaders

xload comes with a few inbuilt loaders:

  • OSLoader - Loads data from environment variables
  • PrefixLoader - Prefixes keys before loading from another loader
  • SerialLoader - Loads data from multiple loaders, with last non-empty value winning

Custom Loaders

You can also write your own custom loaders by implementing the Loader interface:

type Loader interface {
    Load(ctx context.Context, key string) (string, error)
}

Documentation

Overview

Package xload is a struct first data loading library. xload provides a simple Loader interface to implement custom loaders and compose them in different ways.

Index

Examples

Constants

This section is empty.

Variables

View Source
var (
	// ErrNotPointer is returned when the given config is not a pointer.
	ErrNotPointer = errors.New("xload: config must be a pointer")
	// ErrNotStruct is returned when the given config is not a struct.
	ErrNotStruct = errors.New("xload: config must be a struct")
	// ErrMissingKey is returned when the key is missing from the tag.
	ErrMissingKey = errors.New("xload: missing key on required field")
)

Functions

func FlattenMap

func FlattenMap(m map[string]interface{}, sep string) map[string]string

FlattenMap flattens a map[string]interface{} into a map[string]string. Nested maps are flattened using given separator.

func Load

func Load(ctx context.Context, v any, opts ...Option) error

Load loads values into the given struct using the given options. "env" is used as the default tag name. xload.OSLoader is used as the default loader.

Example (ArrayDelimiter)
package main

import (
	"context"

	"github.com/gojekfarm/xtools/xload"
)

func main() {
	type AppConf struct {
		// value will be split by |, instead of ,
		// e.g. HOSTS=host1|host2|host3
		Hosts []string `env:"HOSTS,delimiter=|"`
	}

	var conf AppConf

	err := xload.Load(context.Background(), &conf)
	if err != nil {
		panic(err)
	}
}
Output:

Example (ConcurrentLoading)
package main

import (
	"context"
	"time"

	"github.com/gojekfarm/xtools/xload"
)

func main() {
	type AppConf struct {
		Host    string        `env:"HOST"`
		Debug   bool          `env:"DEBUG"`
		Timeout time.Duration `env:"TIMEOUT"`
	}

	var conf AppConf

	loader := xload.LoaderFunc(func(ctx context.Context, key string) (string, error) {
		// lookup value from a remote service

		// NOTE: this function is called for each key concurrently
		// so make sure it is thread-safe.
		// Using a pooled client is recommended.
		return "", nil
	})

	err := xload.Load(
		context.Background(),
		&conf,
		xload.Concurrency(3), // load 3 keys concurrently
		xload.FieldTagName("env"),
		loader,
	)
	if err != nil {
		panic(err)
	}
}
Output:

Example (CustomDecoder)
package main

import (
	"context"
	"time"

	"github.com/gojekfarm/xtools/xload"
)

type Host string

func (h *Host) Decode(val string) error {

	return nil
}

func main() {
	// Custom decoder can be used for any type that
	// implements the Decoder interface.

	type AppConf struct {
		Host    Host          `env:"HOST"`
		Debug   bool          `env:"DEBUG"`
		Timeout time.Duration `env:"TIMEOUT"`
	}

	var conf AppConf

	err := xload.Load(context.Background(), &conf)
	if err != nil {
		panic(err)
	}
}
Output:

Example (CustomLoader)
package main

import (
	"context"
	"time"

	"github.com/gojekfarm/xtools/xload"
)

func main() {
	type AppConf struct {
		Host    string        `env:"HOST"`
		Debug   bool          `env:"DEBUG"`
		Timeout time.Duration `env:"TIMEOUT"`
	}

	var conf AppConf

	loader := xload.LoaderFunc(func(ctx context.Context, key string) (string, error) {
		// lookup value from somewhere
		return "", nil
	})

	err := xload.Load(
		context.Background(),
		&conf,
		xload.FieldTagName("env"),
		loader,
	)
	if err != nil {
		panic(err)
	}
}
Output:

Example (CustomTagNames)
package main

import (
	"context"
	"time"

	"github.com/gojekfarm/xtools/xload"
)

func main() {
	type AppConf struct {
		Host    string        `custom:"HOST"`
		Debug   bool          `custom:"DEBUG"`
		Timeout time.Duration `custom:"TIMEOUT"`
	}

	var conf AppConf

	err := xload.Load(context.Background(), &conf, xload.FieldTagName("custom"))
	if err != nil {
		panic(err)
	}
}
Output:

Example (DecodingJSONValue)
package main

import (
	"context"
	"encoding/json"

	"github.com/gojekfarm/xtools/xload"
)

type ServiceAccount struct {
	ProjectID string `json:"project_id"`
	ClientID  string `json:"client_id"`
}

func (sa *ServiceAccount) UnmarshalJSON(data []byte) error {
	type Alias ServiceAccount

	var alias Alias

	err := json.Unmarshal(data, &alias)
	if err != nil {
		return err
	}

	*sa = ServiceAccount(alias)

	return nil
}

func main() {
	// Decoding JSON value can be done by implementing
	// the json.Unmarshaler interface.
	//
	// If using json.Unmarshaler, use type alias to avoid
	// infinite recursion.

	type AppConf struct {
		ServiceAccount ServiceAccount `env:"SERVICE_ACCOUNT"`
	}

	var conf AppConf

	err := xload.Load(context.Background(), &conf)
	if err != nil {
		panic(err)
	}
}
Output:

Example (Default)
package main

import (
	"context"
	"time"

	"github.com/gojekfarm/xtools/xload"
)

func main() {
	type AppConf struct {
		Host    string        `env:"HOST"`
		Debug   bool          `env:"DEBUG"`
		Timeout time.Duration `env:"TIMEOUT"`
	}

	var conf AppConf

	err := xload.Load(context.Background(), &conf)
	if err != nil {
		panic(err)
	}
}
Output:

Example (ExtendingStructs)
package main

import (
	"context"
	"net/url"
	"time"

	"github.com/gojekfarm/xtools/xload"
)

func main() {
	type Host struct {
		URL       url.URL `env:"URL"`
		Telemetry bool    `env:"TELEMETRY"`
	}

	type DB struct {
		Host
		Username string `env:"USERNAME"`
		Password string `env:"PASSWORD"`
	}

	type HTTP struct {
		Host
		Timeout time.Duration `env:"TIMEOUT"`
	}

	type AppConf struct {
		DB   DB   `env:",prefix=DB_"`
		HTTP HTTP `env:",prefix=HTTP_"`
	}

	var conf AppConf

	err := xload.Load(context.Background(), &conf)
	if err != nil {
		panic(err)
	}
}
Output:

Example (MapSeparator)
package main

import (
	"context"

	"github.com/gojekfarm/xtools/xload"
)

func main() {
	type AppConf struct {
		// key value pair will be split by :, instead of =
		// e.g. HOSTS=db:localhost,cache:localhost
		Hosts map[string]string `env:"HOSTS,separator=:"`
	}

	var conf AppConf

	err := xload.Load(context.Background(), &conf)
	if err != nil {
		panic(err)
	}
}
Output:

Example (PrefixLoader)
package main

import (
	"context"
	"time"

	"github.com/gojekfarm/xtools/xload"
)

func main() {
	type AppConf struct {
		Host    string        `env:"HOST"`
		Debug   bool          `env:"DEBUG"`
		Timeout time.Duration `env:"TIMEOUT"`
	}

	var conf AppConf

	err := xload.Load(
		context.Background(),
		&conf,
		xload.PrefixLoader("MYAPP_", xload.OSLoader()),
	)
	if err != nil {
		panic(err)
	}
}
Output:

Example (Required)
package main

import (
	"context"
	"time"

	"github.com/gojekfarm/xtools/xload"
)

func main() {
	type AppConf struct {
		Host    string        `env:"HOST,required"`
		Debug   bool          `env:"DEBUG"`
		Timeout time.Duration `env:"TIMEOUT"`
	}

	var conf AppConf

	// if HOST is not set, Load will return ErrRequired
	err := xload.Load(context.Background(), &conf)
	if err != nil {
		panic(err)
	}
}
Output:

Example (Structs)
package main

import (
	"context"

	"github.com/gojekfarm/xtools/xload"
)

func main() {
	type DBConf struct {
		Host string `env:"HOST"` // will be loaded from DB_HOST
		Port int    `env:"PORT"` // will be loaded from DB_PORT
	}

	type HTTPConf struct {
		Host string `env:"HTTP_HOST"` // will be loaded from HTTP_HOST
		Port int    `env:"HTTP_PORT"` // will be loaded from HTTP_PORT
	}

	type AppConf struct {
		DB   DBConf   `env:",prefix=DB_"` // example of prefix for nested struct
		HTTP HTTPConf // example of embedded struct
	}

	var conf AppConf

	err := xload.Load(context.Background(), &conf)
	if err != nil {
		panic(err)
	}
}
Output:

Example (TransformFieldName)
package main

import (
	"context"
	"strings"
	"time"

	"github.com/gojekfarm/xtools/xload"
)

func main() {
	type AppConf struct {
		Host    string        `env:"MYAPP_HOST"`
		Debug   bool          `env:"MYAPP_DEBUG"`
		Timeout time.Duration `env:"MYAPP_TIMEOUT"`
	}

	var conf AppConf

	// transform converts key from MYAPP_HOST to myapp.host
	transform := func(next xload.Loader) xload.LoaderFunc {
		return func(ctx context.Context, key string) (string, error) {
			newKey := strings.ReplaceAll(key, "_", ".")
			newKey = strings.ToLower(newKey)

			return next.Load(ctx, newKey)
		}
	}

	err := xload.Load(
		context.Background(),
		&conf,
		transform(xload.OSLoader()),
	)
	if err != nil {
		panic(err)
	}
}
Output:

Types

type Concurrency added in v0.4.1

type Concurrency int

Concurrency allows customising the number of goroutines to use. Default is 1.

type Decoder

type Decoder interface {
	Decode(string) error
}

Decoder is the interface for custom decoders.

type ErrInvalidMapValue

type ErrInvalidMapValue struct {
	// contains filtered or unexported fields
}

ErrInvalidMapValue is returned when the map value is invalid.

func (ErrInvalidMapValue) Error added in v0.6.0

func (e ErrInvalidMapValue) Error() string

type ErrInvalidPrefix

type ErrInvalidPrefix struct {
	// contains filtered or unexported fields
}

ErrInvalidPrefix is returned when the prefix option is used on a non-struct key.

func (ErrInvalidPrefix) Error added in v0.6.0

func (e ErrInvalidPrefix) Error() string

type ErrInvalidPrefixAndKey added in v0.4.1

type ErrInvalidPrefixAndKey struct {
	// contains filtered or unexported fields
}

ErrInvalidPrefixAndKey is returned when the prefix option is used with a key.

func (ErrInvalidPrefixAndKey) Error added in v0.6.0

func (e ErrInvalidPrefixAndKey) Error() string

type ErrRequired

type ErrRequired struct {
	// contains filtered or unexported fields
}

ErrRequired is returned when a required key is missing.

func (ErrRequired) Error added in v0.6.0

func (e ErrRequired) Error() string

type ErrUnknownFieldType

type ErrUnknownFieldType struct {
	// contains filtered or unexported fields
}

ErrUnknownFieldType is returned when the key type is not supported.

func (ErrUnknownFieldType) Error added in v0.6.0

func (e ErrUnknownFieldType) Error() string

type ErrUnknownTagOption

type ErrUnknownTagOption struct {
	// contains filtered or unexported fields
}

ErrUnknownTagOption is returned when an unknown tag option is used.

func (ErrUnknownTagOption) Error added in v0.6.0

func (e ErrUnknownTagOption) Error() string

type FieldTagName

type FieldTagName string

FieldTagName allows customising the struct tag name to use.

type Loader

type Loader interface {
	Load(ctx context.Context, key string) (string, error)
}

Loader defines the interface for a config loader.

type LoaderFunc

type LoaderFunc func(ctx context.Context, key string) (string, error)

LoaderFunc is a function that implements Loader.

func OSLoader

func OSLoader() LoaderFunc

OSLoader loads values from the OS environment.

func PrefixLoader

func PrefixLoader(prefix string, loader Loader) LoaderFunc

PrefixLoader wraps a loader and adds a prefix to all keys.

func SerialLoader

func SerialLoader(loaders ...Loader) LoaderFunc

SerialLoader loads values from multiple loaders. Last non-empty value wins.

func (LoaderFunc) Load

func (f LoaderFunc) Load(ctx context.Context, key string) (string, error)

Load returns the config value for the given key.

type MapLoader

type MapLoader map[string]string

MapLoader loads values from a map.

Can be used with xload.FlattenMap as an intermediate format when loading from various sources.

func (MapLoader) Load

func (m MapLoader) Load(_ context.Context, key string) (string, error)

Load fetches the value from the map.

type Option

type Option interface {
	// contains filtered or unexported methods
}

Option configures the xload behaviour.

func WithLoader

func WithLoader(loader Loader) Option

WithLoader allows customising the loader to use.

Directories

Path Synopsis
providers
cached Module
viper Module
yaml Module
Package xloadtype contains commonly used types for working with xload.Loader.
Package xloadtype contains commonly used types for working with xload.Loader.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL