config

Overview ¶

Package config is used to configure software systems. A configuration managed by package config is called a profile. The binary that loads a profile declares a set of named, global objects through the APIs in this package. A profile configures these objects (objects may depend on each other, forming a DAG) and lets the user retrieve configured objects through its API.

The semantics of profiles provide the kind of flexibility that is often required in operational contexts. Profiles define a principled overriding so that a base configuration can be extended by the user, either by composing multiple configuration or by editing the configuration through a command-line integration. Profiles may also derive multiple instances from the same base instance in order to provide small variations on instance configuration. Profiles define a concrete syntax so that they may be stored (e.g., centrally) or transmitted over a network connection (e.g., to bootstrap a remote binary with a particular configuration). Profiles are also self-documenting in the manner of Go's flag package. Profiles are resolved lazily, and thus maintain configuration for unknown instances, so long as these are never retrieved. This permits a single profile to be reused across many binaries without concern for compatibility.

Profile syntax ¶

A profile contains a set of clauses, or directives. Each clause either declares a new instance or configures an existing instance. Clauses are interpreted in order, top-to-bottom, and later configurations override earlier configurations. These semantics accommodate for "overlays", where for example a user profile is loaded after a base profile to provide customization. Within GRAIL, a base profile is declared in the standard package github.com/grailbio/base/grail, which also loads a user profile from $HOME/grail/profile.

A parameter is set by the directive param. For example, the following sets the parallelism parameter on the instance bigslice to 1024:

param bigslice parallelism = 1024

The values supported by profiles are: integers, strings, booleans, floats, and indirections (naming other instances). The following shows an example of each:

param bigslice load-factor = 0.8
param bigmachine/ec2system username = "marius"
param bigmachine/ec2system on-demand = false
param s3 retries = 8

As a shortcut, parameters for the same instance may be grouped together. For example, the two parameters on the instance bigmachine/ec2system may be grouped together as follows:

param bigmachine/ec2system (
	username = "marius"
	on-demand = false
)

Instances may refer to each other by name. The following configures the aws/ticket instance to use a particular ticket path and region; it then configures bigmachine/ec2system to use this AWS session.

param aws/ticket (
	path = "eng/dev/aws"
	region = "us-west-2"
)

param bigmachine/ec2system aws = aws/ticket

Profiles may also define new instances with different configurations. This is done via the instance directive. For example, if we wanted to declare a new bigmachine/ec2system that used on-demand instances instead of spot instances, we could define a profile as follows:

instance bigmachine/ec2ondemand bigmachine/ec2system

param bigmachine/ec2ondemand on-demand = false

Since it is common to declare an instance and configure it, the profile syntax provides an affordance for combining the two, also through grouping. The above is equivalent to:

instance bigmachine/ec2ondemand bigmachine/ec2system (
	on-demand = false
	username = "marius-ondemand"
	// (any other configuration to be changed from the base)
)

New instances may depend on any instance. For example, the above may be further customized as follows.

instance bigmachine/ec2ondemand-anonymous bigmachine/ec2ondemand (
	username = "anonymous"
)

Customization through flags ¶

Profile parameters may be adjusted via command-line flags. Profile provides utility methods to register flags and interpret them. See the appropriate methods for more details. Any parameter may be set through the provided command-line flags by specifying the path to the parameter. As an example, the following invocations customize aspects of the above profile.

# Override the ticket path and the default ec2system username.
# -set flags are interpreted in order, and the following is equivalent
# to the clauses
# 	param aws/ticket path = "eng/prod/aws"
#	param bigmachine/ec2system username = "anonymous"
$ program -set aws/ticket.path=eng/prod/aws -set bigmachine/ec2system.username=anonymous

# User the aws/env instance instead of aws/ticket, as above.
# The type of a flag is interpreted based on underlying type, so
# the following is equivalent to the clause
# 	param bigmachine/ec2system aws = aws/env
$ program -set bigmachine/ec2system.aws=aws/env

Default profile ¶

Package config also defines a default profile and a set of package-level methods that operate on this profile. Most users should make use only of the default profile. This package also exports an http handler on the path /debug/profile on the default (global) ServeMux, which returns the global profile in parseable form.