depo

depo - Dependency orchestration for Go

• depo - Dependency orchestration for Go
  • Do You Need This?
  • How Is This Different From X?
  • How To Use
    • Make A Factory
      • The Factory Protocol In Depth
      • Middleware Factories And The Middleware Protocol
    • Wiring Dependencies Together
    • Graph Validation
    • Implementation Selection
    • Sorting And Ordering
    • Constructing Instances
    • Using The Orchestrator
  • Planned Features
  • Developing
    • Make targets
  • License

Do You Need This?

This project is intended to be a tool kit for building higher level dependency orchestration and injection tooling. This puts it in the league of uber-go/dig and google/wire.

While it is possible to use this library directly we don't really recommend that you do so. There are still several aspects of managing a system runtime that aren't covered by dependency orchestration and injection alone. You much more likely will benefit from using an application framework that includes dependency injection such as uber-go/fx, which is built from uber-go/dig, or stackopsd/app which is built from this library.

However, if you're looking to build your own higher level framework or want to know how stackopsd/app works then you're in the right place.

How Is This Different From X?

The uber-go/dig and google/wire projects are mostly stable and have pretty compelling feature sets for dependency injection tools. They both take a very different approach to implementation but deliver roughly the same feature set as each other. They are oriented around the idea that you will provide a constructor function in the form of:

func Constructor(dep Type1, dep2 *Type2, dep3 *Type3) (P1, *P2, error)

where each dependency is declared as a parameter of the constructor and each return value equates to a type that the constructor provides to the dependency injection system. This model is appealing because the code looks like standard Go code and all of the wiring is handled through inspection of the function signature rather than putting the burden on you, the user, to connect all the pieces. The constructor function model works well for simple use cases but leaves several features to be implemented by the user. Notably, neither uber-go/dig nor google/wire provide native support for loading drivers where the implementation is selected at runtime, integration with user configuration systems, or injection of middleware around loaded dependencies. It may be possible to implement these features on top of those libraries but the burden of managing the added complexity is placed on you.

This project is an attempt to provide a better balance of complexity and features. We replace constructor functions with factory structs, which are talked about in-depth in The Factory Protocol, and enable a suite of new features including:

• Native support for "drivers" where any number of implementations may be registered for a given dependency, or type, but only one is selected at runtime. Ex: Choose one of multiple DB backends for a system.

• Native support for "extensions" where zero or more implementations may be active simultaneously at runtime. Ex: Load a suite of callbacks that will be triggered when an event is received.

• Native support for "middleware" which are chains of zero or more functions that wrap or modify a dependency, or type. Ex: Enable or disable authentication for an HTTP handler during testing without changing code.

• Hooks for integrating with user facing configuration systems. Ex: Load a username and password before trying to create a database connection.

We believe these features are compelling enough to warrant a break from the existing libraries and that the added constraints are actually minimal enough that usage is not any more complex that other libraries.

How To Use

Make A Factory

All dependency management is orchestrated through the use of structs that act as factories in the "factory pattern" sense. The basics of making factories are:

// All factories must have a corresponding configuration struct even if the
// configuration struct is empty. The struct may be named anything you want
// and should contain any values that you need to get from a person before
// a system starts. Usernames, passwords, timeouts, etc. are all example of
// values a person might want to configure.
type Config struct {
	Timout time.Duration
}

// Each factory is a struct matching this general shape. The struct may be
// named anything you want. Each struct field is assumed to be a dependency
// you need injected.
type Factory struct {
	// Each field will be treated as a dependency that needs injection. These
	// are equivalent to the constructor function parameters in other libraries.
	// The field name may be anything you want. The type of the field must be
	// something that another factory in the system returns.
	Transport http.RoundTripper
}
// Each factory must have a method named `Config` that takes a context and
// returns and instance of the configuration struct. If your configuration has
// default values then you should set them here.
func (f *Factory) Config(ctx context.Context) *Config {
	return &Config{
		Timeout: 3*time.Second,
	}
}

// Each factory must have a method named `Make` that accepts a context and the
// configuration type returned by `Config`. The return type is the dependency
// that the factory provides. For example, another factory may require
// *http.Client which would match the type returned here.
func (f *Factory) Make(ctx context.Context, conf *Config) (*http.Client, error) {
	return &http.Client{
		Timeout: conf.Timeout,
		// The system will inject dependencies before calling make so you can
		// assume that all fields of your factory are populated.
		Transport: f.Transport,
	}
}

This is our equivalent of the constructor function. It's largely the same amount of code except we put things into a struct instead of only a method. We also add in the user configuration hooks with the Config method and configuration struct. If you can re-arrange your constructor functions into this shape then they will work with the dependency injection system.

The Factory Protocol In Depth

As demonstrated Make A Factory, our library replaces the typical constructor functions used in uber-go/dig and google/wire with structs that implement a factory pattern.

The Factory Protocol is an abstract interface we use to validate and operate all the factory structs that are given to the injection system. Unfortunately, Go generics are still some time away from inclusion in the language so we are unable to provide an actual Go interface that describes a factory in a way that preserves type safety so we are limited to describing the interface in words and pseudo-code:

type Protocol interface{
    Config(ctx context.Context) *C
    Make(ctx context.Context, conf *C) (T, error)
}

In this definition, C refers to a user defined struct that will act as a data container for configuration data. All input values needed by the Make method to construct an instance must be provided by the contents of C. The output of the Config method must be a non-nil pointer to an instance of C that contains any default values for configuration fields.

T refers to another user defined type that is being created by the factory. T may be defined as any valid Go type whether that be a primitive value, an instance of a struct, a pointer to a value, or an interface type, etc.

When creating factories, both C and T may be named anything and do not need to be called C and T. These names are merely placeholders. The important thing is that the output of Config is accepted as the second argument of Make. For example, here is a contrived implementation to help visualize:

type Configuration struct{
    Timeout time.Duration
}

type Factory struct {
	Transport http.RoundTripper
}

func(f *Factory) Config(ctx context.Context) *Configuration {
    return &Configuration{Timeout: 5*time.Second}
}

func(f *Factory) New(ctx context.Context, conf *Configuration) (*http.Client, error) {
    return &http.Client{Timeout: conf.Timeout, Transport: f.Transport}, nil
}

In the above example, the C placeholder is now Configuration and the T placeholder is now *http.Client. The types may be named anything and the T value may be any Go type.

Also in the example you can see usage of struct fields. Every field of a factory struct is assumed to be a dependency that needs injection. The type of each field must match up with a type returned by another factory somewhere. There are only two exceptions to this. One is that any field defined as a slice type is assumed to be receiving a list of extensions that all match the type of the slice element (ex: []T would receive the output of zero or more T types that were each created by a different factory). The other exception is if the field is an embedded type. Embedded types are assumed to be factories, themselves, that the current factory is wrapping or extending. For example, if you wanted to install the same factory twice then you would need to do something like this:

type FactoryExtension struct {
	*Factory
}

This results in a copy of all the same dependencies and methods of the original factory but under a new name. Additional dependencies may be added if needed.

Using this structure provides for several important aspects of this project. For one, it provides a formal mechanism by which configuration values may be managed. In order to construct an instance of the factory output any consumer must first call the Config method to obtain a default configuration. From there, any number of configuration loading mechanisms may be used to populate the configuration struct with new values. It also organizes the dependencies required to create something is a structured way.

Another important aspect of the Factory Protocol is that is is entire detached from any library or framework. All factories are valid Go code regardless of whether or not a library like this one is being used. We strongly value loose coupling and this structure provides for strong decoupling of application code from anything that consumes it.

Middleware Factories And The Middleware Protocol

Middleware are functions that take an instance of something, wrap it in some added behaviors, and then return the wrapped version. These are commonly used in HTTP stacks to add features like authentication and input validation. The middleware pattern, however, is generally useful for isolating independent layers of behavior that can be added or removed without affecting the core logic of some component. We generalize the middleware function pattern using what we call the Middleware Protocol. Like the Factory Protocol, The Middleware Protocol would generally be implemented using generics in other languages but is, instead, defined as documentation and enforced through reflection in Go.

The protocol defines an instance of a middleware as a function that accepts a type and returns a value of an equivalent type. The explicit purpose of middleware functions is to wrap types with additional functionality. Each middleware may implement any functionality such as adding logging, telemetry, automated retries, or HTTP request input validation, etc. Each middleware function may defined in one of several forms:

func(T) T
func(T) (T, erro)
func(context.Context, T) T
func(context.Context, T) (T, error)

In these definitions T refers to any type that is being wrapped by the middleware. Middleware functions may optionally return an error if there is a possibility of failure or accept a context if there is need for cancellation support.

It is generally expected that the input and output types will be exactly the same. That is, if a middleware accepts http.RoundTripper as in the method input then it would almost always return http.RoundTripper as the output. However, it is possible to return a different type for the output so long as it is convertible to the input type. For example, the method could appear as:

func(http.RoundTripper) MyRoundTripperWrapper

so long as the MyRoundTripperWrapper type implements the http.RoundTripper interface. This allows middleware functions to return concrete types but remain compatible with other middleware that may only accept and return the interface type.

Functions that satisfy the middleware protocol may be return by a Middleware Factory. Middleware factory are exactly the same as any other factory except that the type returned by Make matches the middleware protocol:

type MiddlewareConfig struct {}
type MiddlewareFactory struct {}
func(f *MiddlewareFactory) Config(context.Context) *MiddlewareConfig {
	return &MiddlewareConfig{}
}
func(f *MiddlewareFactory) Make(context.Context, *MiddlewareConfig) (func(http.RoundTripper) http.RoundTripper, error) {
	return func(r http.RoundTripper) http.RoundTripper {
		return r
	}, nil
}

All of the same rules for factories apply as they relate to declaring dependencies and defining configuration options. The only difference is that the type inference during factory registration is middleware aware and will select the input type of the middleware function rather than the function type directly.

Wiring Dependencies Together

This project provides a registry system that is used to collect all the dependencies of a system:

registry := depo.NewRegistry()

You can add your factories to the registry in two ways. The first is the "simple" method:

registry.Add(depo.TypeDriver, new(Factory))

The simple method accepts the kind of dependency you are adding and an instance of your factory. You do not need to initialize anything on the factory so you can use new to create an instance or the &Factory{} style if you prefer. The available dependency types are TypeDriver, TypeExtension, and TypeMiddleware. Drivers have one or more implementations but only one will be selected at runtime. Extensions have any number of implementations and will have zero or more selected at runtime. Middleware are used to wrap types with extra behavior and may have zero or more selected at runtime. All selected middleware will be automatically applied to the target type before it is injected into a system. See Middleware Factories And The Middleware Protocol for in-depth documentation on middleware.

Registering a factory using Add will infer the type it provides by inspecting the signature of the Make function. This can be overridden using the more "advanced" method for registration:

registry.AddAs(depo.TypeDriver, new(Factory), new(*http.Client))

The advanced registration allows you to override the automatic type detection. This is useful for cases where your factory returns a concrete type but you want it to be registered for an interface type.

Graph Validation

After registering all dependencies you should run the set through validation. There is some minimal validation performed at registration time but a much more extensive validation suite can only be run after everything is registered. Validation rules are implemented with the following interface:

type Validator interface {
	Validate(ctx context.Context, ds []depo.Dependency) error
}

Included with the project we provide validation for:

• Missing dependencies
• Missing implementations
• Dependency cycles
• Type equivalence

Type equivalence validation is roughly what the compiler would provide you if this project used code generation instead of reflection. The type equivalence validator ensures that every factory's fields may be satisfied by every other factory registered for the requested type and that all middleware are compatible with every factory that produces their target type. All of the built-in validators are available as a set through the depo.NewValidator() method:

validator := depo.NewValidator()
if err := validator.Validate(context.Background(), registry.Dependencies()); err != nil {
	panic(err)
}

Implementation Selection

Once we have a valid set of dependencies we need to filter out the implementations that will not be loaded at runtime. Implementation selectors are implemented with the following interface:

type Selector interface{
	Select(ctx context.Context, ds []depo.Dependency) ([]depo.Dependency, error)
}

Each implementation must take in a populated dependency graph and return a new version of that graph that only contains the implementations that should be used. This project offers a rudimentary implementation, called SelectorBasice that selects the first implementation registered for any driver, all instances of an extension, and all middleware. However, this is not greatly useful and this is one of the primary components that frameworks should re-implement.

Sorting And Ordering

After applying a selection process to the graph we need to order the graph for loading. In order to properly load implementations we must guarantee that every dependency of that implementation is already loaded. This is often referred to as "linearization" or "topological sorting". Sorters that perform this function are implemented with the following interface:

type Sorter interface{
	Sort(ctx context.Context, ds []depo.Dependency) ([]depo.Dependency, error)
}

Each implementation must return a new slice of dependencies that are ordered according to when they should be loaded. This project provides a depth-first implementation of this called SorterTopological that is generally useful for all cases.

Constructing Instances

Finally, the sorted sequence of dependencies must be actually loaded. Loaders implement the following interface:

type Loader interface {
	Load(ctx context.Context, ds []depo.Dependency) ([]depo.LoadedDependency, error)
}

Additional documentation and details of the LoadedDependency are available in the Go docs.

Each Loader is responsible for constructing an instance of every implementation that remains after filtering and sorting. Constructing an instance of an implementation means leveraging the Factory Protocol. Included in the project is a loader called LoaderBasic that calls the factory Make method with the default configuration. This is not greatly useful but serves as a good example for how you might implement your own loader that integrates with your configuration loading system.

Using The Orchestrator

To make it a little easier to correctly process a dependency set through all the major components we've included a type called Orchestrator that takes in one of each of the above interfaces and performs the processing chain as described.

Planned Features

We're still incubating this library so that we can better understand where it is deficient. However, there is no further roadmap for this library in terms of additional features. This is because all higher level features are being implemented in https://github.com/stackopsd/app where we are including a number of quality-of-life features in addition to providing integrations with other tools in our ecosystem like https://github.com/stackopsd/config.

Developing

Make targets

This project includes a Makefile that makes it easier to develop on the project. The following are the included targets:

• fmt

  Format all Go code using goimports.

• generate

  Regenerate any generated code. See gen.go for code generation commands.

• lint

  Run golangci-lint using the included linter configurations.

• test

  Run all unit tests for the project and generate a coverage report.

• integration

  Run all integration tests for the project and generate a coverage report.

• coverage

  Generate a combined coverage report from the test and integration target outputs.

• update

  Update all project dependencies to their latest versions.

• tools

  Generate a local bin/ directory that contains all of the tools used to lint, test, and format, etc.

• updatetools

  Update all project ools to their latest versions.

• vendor

  Generate a vendor directory.

• clean/cleancoverage/cleantools/cleanvendor

  Remove files created by the Makefile. This does not affect any code changes you may have made.

License

Copyright 2019 Kevin Conway

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.