app

app - A dependency orchestration and application framework

• app - A dependency orchestration and application framework
  • Do You Need This?
  • How To Use
    • Wrapping Code With Factories
      • Factories With Dependencies
    • Defining Dependencies
      • Kinds Of Dependencies
    • Runtime Hooks
    • Combining Everything Into An Application
    • Applying Configuration Values
      • Using Environment Variables
      • Using YAML And JSON
    • Generating Sample Configurations
  • Extending The Project
    • Building A Selector
    • Building A Loader
    • Building A HooksExecutor
  • Best Practices
    • Loose Coupling
    • Use A Dedicated Packages
    • Add Validation To Your Build
      • Validating Factories
      • Validating Hooks
      • Validating Applications
  • Planned Features
  • Developing
    • Make targets
  • License

Do You Need This?

This project is designed to help system, or service, developers to create and manage a runtime. Specifically, this projects provides for:

• Dependency orchestration and injection
• Runtime hooks for custom start and stop behaviors
• Integration with user configuration systems
• Signal handling

If you are building a library then you do not need these features as they are specific to managing runtimes.

How To Use

Wrapping Code With Factories

This project is designed to be applied as a set of layers on top of otherwise functioning Go code. We value loose coupling and have built this framework such that no amount of this framework should be present in core application code. To that end we have developed a structure we call the Factory Protocol that can be applied around any existing code that constructs a value you want to have injected as a dependency.

The Factory Protocol is an abstract interface that would normally be defined using generics or templates in other languages. However, Go generics are still some time away from inclusion in the language so we cannot define the protocol using an actual type definition. Instead we will use pseudo-code:

type Protocol interface{
    Config(ctx context.Context) *C
    Make(ctx context.Context, conf *C) (T, error)
}

In this definition C refers to a struct you will define that acts as a data container for all configuration data needed to create the instance. The Config method must return a pointer to an instance of C that contains any default configuration values.

The instance of C returned from Config will be processed using a configuration loading system. By default we use stackopsd/config but the section titled Extending The Project details how other configuration systems may be used. Once the configuration processing is complete the Make method is called with the instance of C to obtain an instance of T. The type T is any type your code constructs whether that is a primitive value, such as a string, or an instance of a struct, pointer to a struct, or an interface type, etc.

The names and definitions of C and T are entirely up to you. For example, if you wanted to create a factory that generates *http.Client instances then you might do the following:

type Configuration struct{
    Timeout time.Duration
}

type Factory struct {}

func(*Factory) Config(ctx context.Context) *Configuration {
    return &Configuration{Timeout: 5*time.Second}
}

func(*Factory) Make(ctx context.Context, conf *Configuration) (*http.Client, error) {
    return &http.Client{Timeout: conf.Timeout, Transport: http.DefaultTransport}, nil
}

This example defines a configuration struct called Configuration that contains an input value needed by the Make method. The Make method constructs an instance of *http.Client. Having a configuration struct and a constructor method is a fairly common practice in Go. This is merely a structure that encapsulates those behaviors in a way that can be integrated with configuration loading system.

Factories With Dependencies

If your factory requires things beyond configuration values then those dependencies should be defined as attributes of the factory struct. To illustrate we will modify the previous factory to depend on an http.RoundTripper rather than using the Go default:

type Configuration struct{
    Timeout time.Duration
}

type Factory struct {
	Transport http.RoundTripper
}

func(*Factory) Config(ctx context.Context) *Configuration {
    return &Configuration{Timeout: 5*time.Second}
}

func(f *Factory) Make(ctx context.Context, conf *Configuration) (*http.Client, error) {
    return &http.Client{Timeout: conf.Timeout, Transport: f.Transport}, nil
}

The factory is nearly identical except that it now references a Transport value that is attached to the factory. The name of the field does not matter but the type of the field will act as a request for a dependency that matches. Another factory will need to exist somewhere that generates values matching the requested type.

For more on the Factory Protocol see https://github.com/stackopsd/depo#the-factory-protocol-in-depth and https://github.com/stackopsd/depo#middleware-factories-and-the-middleware-protocol.

Defining Dependencies

After defining a set of factories that wrap all of your dependencies it's time to then actually declare the dependencies for the system. To do this we recommend defining a function like:

func RegisterDependencies(reg app.Registry) error {
	if err := reg.Add(app.TypeDriver, new(Factory)); err != nil {
		return err
	}
	return nil
}

The system will inspect your factory and automatically determine the type that it produces based on the signature of the Make method. For example, a make function like Make(context.Context, Config) (*http.Client, error) will result in a factory being recorded as providing the *http.Client type. Any other factory registered that has *http.Client as a field may receive the output of this factory.

If you need to register your factory for a type other than what is returned from Make then you can use the Registry.AddAs() method. For example:

func RegisterDependencies(reg app.Registry) error {
	if err := reg.AddAs(app.TypeDriver, new(Factory), new(*http.Client)); err != nil {
		return err
	}
	return nil
}

This is useful when your factory returns a concrete implementation but needs to be registered as an interface type that is requested elsewhere.

Kinds Of Dependencies

There are three kinds of dependencies that may be registered with the system: drivers, extensions, and middleware. These are selected by passing app.TypeDriver, app.TypeExtension, and app.TypeMiddleware, respectively, to the Add or AddAs methods of the registry.

Use the driver type when you have one or more factories that produce the same type but you only want one of them active at runtime. This is the most common type and enables you to, for example, provide two or three database backends for an end-user of your system to choose from.

Use the extension type when you have a variable number of factories that return the same type, you want zero or more of them active at runtime, and you want to receive a slice of all active selections. This choice is useful when you want to have a set of related behaviors that are enabled at runtime. If your system receives events that you want delivered to a number of external channels, for example, then you might use this feature to allow an end-user to enable or disable the email, slack, SMS, or push notifications, etc.

Use the middleware type when you want to wrap, or decorate, the output of a factory. This is useful when you want to layer on functionality without changing the core logic of a component. If you have a factory that returns an HTTP client, for example, you may want to wrap the client in a layer that logs requests, another layer that emits metrics, and another layer that automatically retries requests, etc. Note that middleware factories must return functions that match the Middleware Protocol which is described in detail here: https://github.com/stackopsd/depo#middleware-factories-and-the-middleware-protocol.

Runtime Hooks

The factories used to generate dependencies are not supposed to create side-effects when they run. For example, a factory may create an HTTP server but it should not start the server listening for requests. Instead, side-effects should be handled using runtime hooks.

Defining runtime hooks is a lot like defining dependencies. Each hook is a factory that produces a read-only error channel (<-chan error). The Make function of the factory should perform any required side-effects and may do so blocking or in a background goroutine. The error channel returned by Make will be read from and any value, nil or non-nil, will be considered a signal to stop the system. The difference is that nil signals indicate an expected, or graceful, shutdown while non-nil signals indicate critical system failure. In both cases, the system will execute shutdown hooks. Ex:

type ServerStarterConfig struct {}
type ServerStarterFactory struct {
	Server *http.Server
}
func (*ServerStarterFactory) Config(context.Context) *ServerStarterConfig {
	return &ServerStarterConfig{}
}
func (f *ServerStarterFactory) Make(context.Context, *ServerStarterConfig) (<-chan error, error) {
	ch := make(chan error)
	go func() {
		err := f.Server.ListenAndServe()
		if err == ErrServerClosed {
			ch <- nil
			return
		}
		ch <- err
	} ()
	return ch, nil
}

By making hooks into factories we gain the ability to provide for both dependency injection and configuration in the same way we expect for implementations.

Hook registration is done with the AddOnStart and AddOnStop methods of the registry.

Combining Everything Into An Application

Finally, all of the defined dependencies and hooks are brought together in an Application instance:

package main

import (
	"github.com/stackopsd/app"

	"github.com/myuser/myproject/pkg/runtime" // the package where you defined everything
)

func main() {
	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	reg := app.NewRegistry()
	if err := runtime.RegisterDeps(reg); err != nil {
		panic(err)
	}
	if err := runtime.RegisterHooks(reg); err != nil {
		panic(err)
	}
	a, err := app.NewApplication(ctx,reg)
	if err != nil {
		panic(err)
	}

	if err = a.Start(ctx); err != nil {
		panic(err)
	}
	if err = <-a.Wait(); err != nil {
		panic(err)
	}
	if err = a.Stop(ctx); err != nil {
		panic(err)
	}
}

Applying Configuration Values

Because it is a part of our ecosystem, we ship this project with built-in support for using https://github.com/stackopsd/config for managing the loading of configuration values. The default application will use environment variables as the source of configuration. If you want to use config but with a JSON or YAML file instead then you can swap out the defaults using the OptionConfigLoader feature:

confLoader, err := config.NewLoaderFile("config.json_or_yaml")
if err != nil {
    panic(err)
}

app, err := depo.NewApplication(
	context.Background(),
	registry,
    depo.OptionConfigLoader(confLoader),
)
if err != nil {
    panic(err)
}

Using Environment Variables

The framework uses configuration to both select which implementations are active at runtime and provide factories with user settings. The default source is the environment

APP__DEP_ONE__DRIVER="implementation_name"
APP__DEP_ONE__IMPLEMENTATION_NAME__CONFIG_VALUE_ONE="something"
APP__DEP_ONE__IMPLEMENTATION_NAME__CONFIG_VALUE_TWO="something2"
APP__DEP_ONE__MIDDLEWARE__0="wrapper_one"
APP__DEP_ONE__WRAPPER_ONE__CONFIG_VALUE_ONE="something"
APP__EXTENSION_ONE__ENABLED__0="implementation_one"
APP__EXTENSION_ONE__ENABLED__1="implementation_two"
APP__EXTENSION_ONE__ENABLED__2="implementation_three"
APP__EXTENSION_ONE__IMPLEMENTATION_ONE__CONFIG_VALUE_ONE="something"
APP__EXTENSION_ONE__IMPLEMENTATION_ONE__CONFIG_VALUE_TWO="something2"
APP__EXTENSION_ONE__IMPLEMENTATION_TWO__CONFIG_VALUE_ONE="something"
APP__EXTENSION_ONE__IMPLEMENTATION_TWO__CONFIG_VALUE_TWO="something2"
APP__EXTENSION_ONE__IMPLEMENTATION_THREE__CONFIG_VALUE_ONE="something"
APP__EXTENSION_ONE__IMPLEMENTATION_THREE__CONFIG_VALUE_TWO="something2"
APP__ON_START__HOOK_ONE__CONFIG_VALUE_ONE="something"
APP__ON_STOP__HOOK_ONE__CONFIG_VALUE_ONE="something"

The environment based configuration comes with a few warts because the character set is constrained to only ASCII alpha-numeric values and underscores. The easiest and least risky approach is to use the HelpENV method to generate a sample set of environment variables. This will give you all of the possible environment variables as well as any available documentation on what each value does. This is much easier to do than determining the variable names by following the naming scheme.

If, however, you want to understand the naming scheme then read on.

The general formula is that double underscore (__) is a hierarchical separator and any section containing only numeric values is considered a slice offset value. With these two conventions we can model both map/struct types as well as slice/array types.

The names of all the values are automatically generated based on the factories registered. The order for names works like:

<RETURN_TYPE><FACTORY_NAME>__<CONFIG_VALUE>

The prefix value defaults to APP but can be changed using by installing your own loader.

The return type value is in the form <PKG>_<NAME> where the name follows the canonicalization rules defined in https://github.com/stackopsd/config#configuration-value-names. Primitive types have no package name so a factory that returns a string will result in STRING. A factory that returns http.RoundTripper will result in HTTP_ROUND_TRIPPER.

The factory type is also in the form of <PKG>_<NAME> and the name also follows the canonicalization rules defined in https://github.com/stackopsd/config#configuration-value-names. A factory in package runtime called HTTPServerFactory will render as RUNTIME_HTTP_SERVER_FACTORY.

All configuration values are rendered according to the canonicalization rules.

Using YAML And JSON

YAML and JSON have an advantage over environment variables by having an internal structure.

dep_one:
	driver: "implementation_name"
	implementation_name:
		config_value_one: "something"
		config_value_two: "something2"
	middleware:
		- "wrapper_1"
		- "wrapper_2"
	wrapper_1:
		config_value_one: "something"
list_one:
	enabled:
		- "implementation_one"
		- "implementation_two"
		- "implementation_three"
	implementation_one:
		config_value_one: "something"
		config_value_two: "something2"
	implementation_two:
		config_value_one: "something"
		config_value_two: "something2"
	implementation_three:
		config_value_one: "something"
		config_value_two: "something2"
	middleware: []
on_start:
	hook_one:
		config_value_one: "something"
on_stop:
	hook_one:
		config_value_one: "something"

All top level keys except on_start and on_stop represent a type that the dependency system is managing because it is returned by at least one factory. The key name is derived as <PKG>_<NAME> where PKG is the name of the source package for the type and NAME is the name of the type after going through the canonicalization rules defined in https://github.com/stackopsd/config#configuration-value-names. The exceptions to the rule are primitive types, such as string, which do not have a package name to include so <PKG>_ is left off.

The configuration for each implementation is nested under the type that it produces. The key for each implementation is generated in the same way as the key for the type.

You are encouraged to use the HelpYAML method to generate a starting configuration with all the possible options rather than writing one by hand.

Generating Sample Configurations

This project comes bundled with methods called HelpENV and HelpYAML that generate sample configuration files for an application using the structure that will be read by the https://github.com/stackopsd/config loader that is also bundled.

Extending The Project

In addition to having a low impact on application code, we also want this project to be extensible to specific project needs. We do bundle in support for http://github.com/stackopsd/config because that's our configuration loading system but we never want that to be the only configuration system possible. Likewise we make some opinionated decisions about how driver and list implementations are selected that may not be correct in all cases. For example, what if you wanted to have list items enabled by default?

To enable modifications of all these behaviors we provide three interfaces that need to be implemented: The Selector, the Loader, and the HooksExecutor.

Building A Selector

A Selector is a component used by the system to filter out any registered implementations that will not be loaded when the system starts. The Selector interface is:

// Selector filters the implementations of each Dependency to only those that
// will be loaded at runtime. Any selection process may be used so long as the
// Type rules are maintained such that each TypeDriver has exactly one
// implementation, and each TypeExtension has zero or more implementations.
// Likewise, zero ore more TypeMiddleware may be enabled. Whichever remains in
// the implementations lists after this component processes them will be loaded.
type Selector interface {
	Select(ctx context.Context, ds []depo.Dependency) ([]depo.Dependency, error)
}

Like the code documentation says, any method may be used to filter the implementations so long as the rules for each kind of dependency are preserved. This is the component to create if you want to change how implementations are selected. For example, this is where you would implement extensions that default to "enable all" or drivers that automatically select the first implementation if there is only one in the list. See the LoaderConfig implementation in this project for an example of how filtering may be done.

Building A Loader

A Loader is the component that actually constructs an instance of the factory output for all implementations. A Loader must implement this interface:

// Loader implementations construct instances of all the implementations
// requested.
type Loader interface {
	Load(ctx context.Context, ds []depo.Dependency) ([]depo.LoadedDependency, error)
}

The dependency set given to the Loader will already be filtered to selected implementations and ordered such that each implementation is guaranteed to come after anything it depends on. This is the component to create if you want to change how factory configuration structs are populated. See the LoaderConfig for examples.

Building A HooksExecutor

A HooksExecutor is the component that constructs each hook and loads its configuration. A HooksExecutor must implement this interface:

// HooksExecutor implementations are responsible for executing a set of hooks.
type HooksExecutor interface {
	ExecuteHooks(ctx context.Context, hooks []*depo.ReflectedFactory, loaded []depo.LoadedDependency) (<-chan error, error)
}

The loaded value will contain all loaded dependencies and hooks will contain all hooks for a given lifecycle event. See the HooksExecutorConfig implementation in this project for a detailed example.

The application accepts a HooksExecutor for each event through the OptionStartExecutor and OptionStopExecutor modifiers when creating an Application.

Best Practices

These are our current recommendations for how to most effectively use this, and any, dependency orchestration and injection system. They are likely to evolve over time so check back periodically.

Loose Coupling

This library should be used as a layer on top of otherwise valid Go code. This library should never be imported or referenced outside of code that is setting up a runtime, such as main.go. Your base code, your system design, and even your factory implementations should be free of any imports or references to this library.

Use A Dedicated Packages

We recommend using a dedicated package, such as /internal, /pkg/internal, or /pkg/runtime, etc., to contain the dependency orchestration code. This will both help prevent tight coupling inside application code and provide a clear place where new system dependencies are registered and organized. We recommend also breaking up the orchestration into multiple steps that can be leveraged by 3rd parties and tests. To illustrate, we recommend your runtime package have the following exports:

package runtime // or other name as desired

// RegisterDependencies centralizes where all factories are added to the system.
func RegisterDependencies(r app.Registry) error {
	if err := r.Add(app.TypeDriver, new(Factory)); err != nil {
		return err
	}
	// ...
	return nil
}

// RegisterHooks centralizes where side-effects are installed.
func RegisterHooks(r app.Registry) error {
	if err := r.AddOnStart(new(Hook)); err != nil {
		return err
	}
	// ...
	return nil
}

// NewApplication wraps app.NewApplication and applies all project specific
// options such as custom configuration loaders or selection rules.
func NewApplication(ctx context.Context, r app.Registry) (app.Application, error) {
	return app.NewApplication(
		ctx, r,
		// Any custom options here.
	)
}

The purpose for this structure is to enable easier modifications to the runtime for specialized purposes such as testing or creating custom builds of an application. A caller could use these methods to construct a full copy of your application, inject their own custom implementations for drivers, etc., and then run the custom build.

Add Validation To Your Build

This project uses reflection to perform most of its complex functions which means we've had to sacrifice a large amount of compile time protection. To account for this we also provide a fairly extensive suite of runtime protections by recreating many of the checks that the compiler would have performed for us. However, because our tooling operates at runtime it means you must have something that executes the validation code in order to report on pass or fail. We recommend running validation as a unit test.

Validating Factories

func TestFactoryVerify(t *testing.T) {
	if err := app.VerifyFactory(new(Factory)); err != nil {
		t.Error(err)
	}
}

The VerifyFactory tool will report on whether or not the given instance satisfies the Factory Protocol and give details on any aspect of the instance that does not.

Validating Hooks

func TestHooksVerify(t *testing.T) {
	if err := app.VerifyHook(new(Hook)); err != nil {
		t.Error(err)
	}
}

The VerifyHook tool will check that your factory both implements the factory protocol and correctly implements the expected hook variant by returning a read-only error channel.

Validating Applications

Finally, there is some validation that can only be done when all the pieces are available together. Notably, validation of dependency cycles, missing dependencies, and advanced type protection can only happen when all values are together. To help with this we provide a method on the Application called Validate that performs all runtime validation:

func TestApplication(t *testing.T) {
	reg := app.NewRegistry()
	if err := runtime.RegisterDependencies(reg); err != nil {
		t.Fatal(err)
	}
	if err := runtime.RegisterHooks(reg); err != nil {
		t.Fatal(err)
	}
	a, err := runtime.NewApplication(context.Background(), reg)
	if err != nil {
		t.Fatal(err)
	}
	if err := app.Validate(); err != nil {
		t.Fatal(err)
	}
}

If an entire application passes validation then the only untested failure case left is misconfiguration during startup.

Planned Features

This project is still very young and there are a few features we'd like to deliver in the future:

• Dependency graph generation.

  Especially for large, complex projects and when debugging cycles, it would be useful to visualize the dependency graph of a system. Because this feature would likely bring in a number of unrelated dependencies we may implement this as a separate module.

• Code generation.

  Even though we've re-implemented most of the compiler type checks, it may still be a useful feature to have an option of generating the final orchestration code.

Developing

Make targets

This project includes a Makefile that makes it easier to develop on the project. The following are the included targets:

• fmt

  Format all Go code using goimports.

• generate

  Regenerate any generated code. See gen.go for code generation commands.

• lint

  Run golangci-lint using the included linter configurations.

• test

  Run all unit tests for the project and generate a coverage report.

• integration

  Run all integration tests for the project and generate a coverage report.

• coverage

  Generate a combined coverage report from the test and integration target outputs.

• update

  Update all project dependencies to their latest versions.

• tools

  Generate a local bin/ directory that contains all of the tools used to lint, test, and format, etc.

• updatetools

  Update all project ools to their latest versions.

• vendor

  Generate a vendor directory.

• clean/cleancoverage/cleantools/cleanvendor

  Remove files created by the Makefile. This does not affect any code changes you may have made.

License

Copyright 2019 Kevin Conway

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.