components

package
v0.0.0-...-a1948fa Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 17, 2020 License: Apache-2.0 Imports: 19 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var ComponentTypes = map[string]bool{
	Service: true,
	Task:    true,
}

ComponentTypes is a set (of keys) enumerating the types of components that shnorky respects

View Source 
var DefaultSpecificationFileName = "component.json"

DefaultSpecificationFileName - this is the name of the file inside the component directory representing the shnorky specification of the component.

View Source 
var DockerImagePrefix = "shnorky/"

DockerImagePrefix is the prefix that shnorky attaches to each docker image name

View Source 
var ErrBuildNotFound = errors.New("Could not find the specified build")

ErrBuildNotFound - signifies that a single row lookup against the builds table in a state database returned no rows

View Source 
var ErrComponentNotFound = errors.New("Could not find the specified component")

ErrComponentNotFound - signifies that a single row lookup against a state database returned no rows

View Source 
var ErrEmptyBuildID = errors.New("BuildID must be a non-empty string")

ErrEmptyBuildID signifies that a caller attempted to create execution metadata in which the BuildID string was the empty string

View Source 
var ErrEmptyComponentID = errors.New("ComponentID must be a non-empty string")

ErrEmptyComponentID signifies that a caller attempted to create build or execution metadata in which the ComponentID string was the empty string

View Source 
var ErrEmptyComponentPath = errors.New("ComponentPath must be a non-empty string")

ErrEmptyComponentPath signifies that a caller attempted to create component metadata in which the ComponentPath string was the empty string

View Source 
var ErrEmptyID = errors.New("ID must be a non-empty string")

ErrEmptyID signifies that a caller attempted to create component metadata in which the ID string was the empty string

View Source 
var ErrInvalidComponentType = errors.New("Invalid ComponentType")

ErrInvalidComponentType signifies that a caller attempted to create component metadata with a component type which wasn't included in the ComponentTypes map

View Source 
var ErrInvalidMountMethod = errors.New("Invalid mount method in component mount configuration: must be one of \"bind\", \"volume\", \"tmpfs\"")

ErrInvalidMountMethod signifies that there was an error parsing a mount in a component mount configuration. It indicates that the value for the Method member was invalid.

View Source 
var ErrInvalidMountType = errors.New("Invalid mount type in component mount specification: must be one of \"file\", \"dir\"")

ErrInvalidMountType signifies that there was an error parsing a component mount specification. Specifically, that the MountType member did not have a valid value.

View Source 
var Service = "service"

Service is a component type which represents a long-running service that must be available as part of a shnorky data processing flow

View Source 
var SpecialPrefixEnv = "env:"

SpecialPrefixEnv denotes that a value in a specification refers to the environment variable whose name is its suffix.

View Source 
var SpecialPrefixUsername = "user:"

SpecialPrefixUsername denotes that a value in a specification refers to a username, its suffix.

View Source 
var Task = "task"

Task is a component type which represents a process that must be run to completion as part of a shnorky data processing flow

View Source 
var ValidMountMethods = map[string]dockerMount.Type{
	"bind":   dockerMount.TypeBind,
	"volume": dockerMount.TypeVolume,
	"tmpfs":  dockerMount.TypeTmpfs,
}

ValidMountMethods defines the values for the MountConfiguration Method member

View Source 
var ValidMountTypes = map[string]MountType{
	"file": MountTypeFile,
	"dir":  MountTypeDir,
}

ValidMountTypes is a map whose keys are the valid values for the Type member in a MountSpecification. This is here to make it easier to create a MountSpecification JSON document.

Functions

func DeleteComponentByID 

func DeleteComponentByID(db *sql.DB, id string) error

DeleteComponentByID creates a new row in the components table with the given component information.

func InsertBuild 

func InsertBuild(db *sql.DB, buildMetadata BuildMetadata) error

InsertBuild inserts the build represented by the given build metadata into the given shnorky state database

func InsertComponent 

func InsertComponent(db *sql.DB, component ComponentMetadata) error

InsertComponent creates a new row in the components table with the given component information.

func InsertExecution 

func InsertExecution(db *sql.DB, executionMetadata ExecutionMetadata) error

InsertExecution inserts an execution row into the state database

func ListBuilds 

func ListBuilds(db *sql.DB, builds chan<- BuildMetadata, componentID string) error

ListBuilds streams builds one by one from the given state database into the given builds channel. This function closes the builds channel when it is finished.

func ListComponents 

func ListComponents(db *sql.DB, components chan<- ComponentMetadata) error

ListComponents streams components one by one from the given state database into the given components channel. This function closes the components channel when it is finished.

func MaterializeEnv 

func MaterializeEnv(rawValue string) string

MaterializeEnv checks if a string is prefixed with "env:". If it is, it returns the value of the environment variable whose name is the remainder of the string. If not, it returns the input value.

func MaterializeUsername 

func MaterializeUsername(rawValue string) (string, error)

MaterializeUsername returns a "uid:gid" string for the user with the given name if the user exists, otherwise it returns an error

func RemoveComponent 

func RemoveComponent(db *sql.DB, id string) error

RemoveComponent removes the component with the given id from the given state database

Types

type BuildMetadata 

type BuildMetadata struct {
	ID          string    `json:"id"`
	ComponentID string    `json:"component_id"`
	CreatedAt   time.Time `json:"created_at"`
}

BuildMetadata - the metadata about a component build that gets stored in the state database

func CreateBuild 

func CreateBuild(ctx context.Context, db *sql.DB, dockerClient *docker.Client, outstream io.Writer, componentID string) (BuildMetadata, error)

CreateBuild creates a new build for the component with the given componentID

func GenerateBuildMetadata 

func GenerateBuildMetadata(componentID string) (BuildMetadata, error)

GenerateBuildMetadata creates a BuildMetadata instance representing a fresh (as yet unbuilt) build of the component specified by the given componentID.

func SelectBuildByID 

func SelectBuildByID(db *sql.DB, id string) (BuildMetadata, error)

SelectBuildByID gets build metadata from the given state database using the given ID. If no build with the given ID is found, returns ErrBuildNotFound in the error position.

func SelectMostRecentBuildForComponent 

func SelectMostRecentBuildForComponent(db *sql.DB, componentID string) (BuildMetadata, error)

SelectMostRecentBuildForComponent gets build metadata from the given state database for the most recent build for the component with the given componentID

type BuildSpecification 

type BuildSpecification struct {
	// Path to context directory (used to build docker image)
	Context string `json:"context"`

	// Path to Dockerfile to be used to build the component - should be relative to the context
	// path
	Dockerfile string `json:"Dockerfile"`
}

BuildSpecification - struct specifying how a component of a shnorky data processing flow should be built; all paths are assumed to be paths relative to the component path (i.e. the directory containing the implementation of the component)

type ComponentMetadata 

type ComponentMetadata struct {
	ID                string    `json:"id"`
	ComponentType     string    `json:"component_type"`
	ComponentPath     string    `json:"component_path"`
	SpecificationPath string    `json:"specification_path"`
	CreatedAt         time.Time `json:"created_at"`
}

ComponentMetadata - the metadata about a component that gets stored in the state database

func AddComponent 

func AddComponent(db *sql.DB, id, componentType, componentPath, specificationPath string) (ComponentMetadata, error)

AddComponent registers a component (by metadata) against a shnorky state database. It applies reasonable defaults where possible (e.g. on SpecificationPath). This is the handler for `shnorky components add`

func GenerateComponentMetadata 

func GenerateComponentMetadata(id, componentType, componentPath, specificationPath string) (ComponentMetadata, error)

GenerateComponentMetadata creates a ComponentMetadata instance from the specified parameters, applying defaults as required and reasonable. It also performs validation on its inputs and returns an error describing the reasons for rejection of invalid component metadata. Component metadata requires that: 1. id be non-null (ErrEmptyID returned otherwise) 2. componentType be one of the keys of the ComponentTypes map (ErrInvalidComponentType returned otherwise) 3. componentPath be non-empty (ErrEmptyComponentPath returned otherwise)

func SelectComponentByID 

func SelectComponentByID(db *sql.DB, id string) (ComponentMetadata, error)

SelectComponentByID gets component metadata from the given state database using the given ID. If no component with the given ID is found, returns ErrComponentNotFound in the error position.

type ComponentSpecification 

type ComponentSpecification struct {
	Build BuildSpecification `json:"build"`
	Run   RunSpecification   `json:"run"`
}

ComponentSpecification - struct specifying how a component of a shnorky data processing flow should be built and executed

func MaterializeComponentSpecification 

func MaterializeComponentSpecification(rawSpecification ComponentSpecification) (ComponentSpecification, error)

MaterializeComponentSpecification applies all run-time substitutions to the given ComponentSpecification For example, it replaces all "env:..." values with values of the corresponding environment variables in the invoking process.

func ReadSingleSpecification 

func ReadSingleSpecification(reader io.Reader) (ComponentSpecification, error)

ReadSingleSpecification reads a single ComponentSpecification JSON document and returns the corresponding ComponentSpecification struct. It returns an error if there was an issue parsing the specification into the struct.

type ExecutionMetadata 

type ExecutionMetadata struct {
	ID          string    `json:"id"`
	BuildID     string    `json:"build_id"`
	ComponentID string    `json:"component_id"`
	CreatedAt   time.Time `json:"created_at"`
	FlowID      string    `json:"flow_id"`
}

ExecutionMetadata - the metadata about a component build execution that gets stored in the state database

func Execute 

func Execute(
	ctx context.Context,
	db *sql.DB,
	dockerClient *docker.Client,
	buildID string,
	flowID string,
	mounts []MountConfiguration,
	env map[string]string,
) (ExecutionMetadata, error)

Execute runs a container corresponding to the given build of the given component. TODO(nkashy1): Maybe take build metadata instead of build ID? This will reduce the number of database lookups that happen in flow execution.

func GenerateExecutionMetadata 

func GenerateExecutionMetadata(build BuildMetadata, flowID string) (ExecutionMetadata, error)

GenerateExecutionMetadata creates an ExecutionMetadata instance representing a potential execution of the build specified by the given build metadata.

type MountConfiguration 

type MountConfiguration struct {
	Source string `json:"source"`
	Target string `json:"target"`
	Method string `json:"method"`
}

MountConfiguration - describes the run-time mount configuration for a shnorky component

func MaterializeMountConfiguration 

func MaterializeMountConfiguration(rawConfig MountConfiguration) (MountConfiguration, error)

MaterializeMountConfiguration validates the members of its input mount configuration, applies the required substitutions, and returns the resulting values in a new MountConfiguration struct.

func ReadMountConfiguration 

func ReadMountConfiguration(reader io.Reader) ([]MountConfiguration, error)

ReadMountConfiguration reads a single MountConfiguration JSON document from the given reader, validates it, and returns it as a MountConfiguration struct. Returns error (in the error position) if the MountConfiguration document is invalid or if there is an error reading it from the reader. If an error is returned, the offending mount configuration object is returned in a singleton array.

type MountSpecification 

type MountSpecification struct {
	// Can be one of the keys of the ValidMountTypes map.
	MountType  string `json:"mount_type"`
	Mountpoint string `json:"mountpoint"`
	ReadOnly   bool   `json:"read_only"`
	Required   bool   `json:"required"`
}

MountSpecification - specifies a mount point within a shnorky component, how it should be mounted on the container side, and whether or not it is required to be mounted at runtime TODO(nkashy1): It does not make sense to specify this kind of mount type in the MountSpecification - the mount type (e.g. whether it is a bind mount or a docker volume mount) is the responsibility of the caller. What does make sense is for MountType to specify the type of filesystem object that the mountpoint expects (e.g. file vs. directory)

type MountType 

type MountType int

MountType is an enum representing the valid mount types for mount specifications 

const (
	// MountTypeFile - mount point is a file
	MountTypeFile MountType = iota + 1
	// MountTypeDir - mount point is a directory
	MountTypeDir
)

type RunSpecification 

type RunSpecification struct {
	// Mapping of environment variable names to values to be set in component container at runtime
	// Special keys:
	//
	// Special values:
	// "env:<VARIABLE_NAME>" - specifies that the value of the environment variable denoted by
	// VARIABLE_NAME in the shnorky process should be interpolated into the specification; if the
	// environment variable is not set in the shnorky process, it will use the empty string "" as
	// the value
	Env map[string]string `json:"env"`

	// Entrypoint override for containers representing this component
	Entrypoint []string `json:"entrypoint"`

	// Command to be invoked when starting component container at runtime
	Cmd []string `json:"cmd"`

	// Mountpoint specify paths inside each container (for this component) that can accept data
	Mountpoints []MountSpecification `json:"mountpoints"`

	// User specifies the uid (and optionally guid that the container should run as) - format the
	// string as "<uid>:<guid>".
	// Special values:
	// "" - container runs as root
	// "env:<VARIABLE_NAME>" - container runs as user specified by environment variable; use
	// "env:UID" to use the user running the current shnorky process, for example
	// "user:<username>" - container runs as the user with the given username
	User string `json:"user"`
}

RunSpecification - struct specifying how a component of a shnorky data processing flow should be executed

func MaterializeRunSpecification 

func MaterializeRunSpecification(rawSpecification RunSpecification) (RunSpecification, error)

MaterializeRunSpecification applies all run-time substitutions to the given RunSpecification

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.