pipeline

package
v0.0.0-...-988b307 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: May 3, 2024 License: BSD-3-Clause Imports: 11 Imported by: 5

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func FindFs

func FindFs(name string) (fs.FS, bool)

func RegisterFs

func RegisterFs(name string, fsys fs.FS) error

RegisterFs adds a named file system to the registry. Nodes can make use of named filesystem for, example, in loading operations.

func RegisterNode

func RegisterNode(name string, newfunc NewNodeFunc) error

func RunNode

func RunNode(node Node, input RunInput, output *RunOutput) error

RunNode is a convenience to run a single node.

Types

type ClonePolicy

type ClonePolicy int
const (
	SmartClone  ClonePolicy = iota // A clone is made when one payload is going to two ore more RunInputs, othwerwise no clone.
	AlwaysClone                    // A clone is always made.
	NeverClone                     // A clone is never made.
)

type Cloner

type Cloner interface {
	Clone() Cloner
}

Cloner is a copying interface.

type ContentData

type ContentData struct {
	Name   string
	Data   string
	Format string
}

ContentData provides a generic content string.

func (*ContentData) Clone

func (d *ContentData) Clone() Cloner

type Flusher

type Flusher interface {
	// Flush any data in the node.
	// There's no reason to reallocate or replace the RunOutput.Pins,
	// just append to what's there.
	Flush(*State, *RunOutput) error
}

Flusher implements a flush operation.

type NewNodeFunc

type NewNodeFunc func() Node

type Node

type Node interface {
	Runner
}

Node is a single function in the pipeline graph. It runs an operation on some input, optionally providing some output.

Node implementations should be thread-safe. The framework uses a NodeData pattern to accomplish this in a simple way: Nodes have a parallel NodeData struct that stores their data. A node anonymously includes its NodeData, and also sets it to State.NodeData in the Start() func. THe data is then passed back in State.Node data during the Run() and Flush().

Optional interfaces: * Starter * Flusher

type Pin

type Pin struct {
	Name    string
	Payload Cloner
	Policy  ClonePolicy
}

type Pipeline

type Pipeline struct {
	// contains filtered or unexported fields
}

func Compile

func Compile(expr string) (*Pipeline, error)

Compile converts an expression into a Pipeline.

func (Pipeline) Env

func (p Pipeline) Env() map[string]any

Env answers the contents of the env() term in the initial expression, if any. This has no functional impact, but serves as a form of documentation to let clients discover what env vars the pipeline supports.

type RunInput

type RunInput struct {
	Pins []Pin
}

RunInput transports pin data into a Run function.

func NewRunInput

func NewRunInput(pins ...Pin) RunInput

NewRunInput answers a new RunInput on the given pins.

func (RunInput) NewRunOutput

func (r RunInput) NewRunOutput(pins []Pin) *RunOutput

NewRunOutput answers a new RunOutput on the given pins.

type RunOutput

type RunOutput struct {
	Pins []Pin
}

RunOutput transports pin data out of a Run function.

func Run

func Run(p *Pipeline, input *RunInput, env map[string]any) (*RunOutput, error)

func RunExpr

func RunExpr(expr string, input *RunInput, env map[string]any) (*RunOutput, error)

type Runner

type Runner interface {
	// Run the supplied pins, converting the input to output.
	// There's no reason to reallocate or replace the RunOutput.Pins,
	// just append to what's there.
	Run(*State, RunInput, *RunOutput) error
}

Runner processes input.

type StartInput

type StartInput interface {
	SetNodeData(any)
}

type Starter

type Starter interface {
	// One-time notification that the node is starting a run.
	// Clients that want to use the NodeData pattern can assign
	// the NodeData field and it will be available during the Run.
	Start(StartInput) error
}

Starter is called at the start of a pipeline run. Implementing starter, and placing all run state in a node data object, will make your node thread-safe.

type State

type State struct {
	// The data set in Starter.Start.
	NodeData any
}

State provides current state data to a running node.

type StructData

type StructData struct {
	// The name of the source struct.
	Name string

	// All exported fields in the struct.
	Fields []StructField

	// All unexported fields in the struct. It should be rare
	// for a client to want access to these fields, so we keep
	// them in a separate slice to avoid polluting the fields list.
	UnexportedFields []StructField
}

StructData provides information about a single struct from source data.

func NewStructData

func NewStructData(name string, fields []StructField, unexportedFields []StructField) *StructData

func (*StructData) Clone

func (d *StructData) Clone() Cloner

type StructField

type StructField struct {
	// The name of the field in the original source.
	Name string
	// The Go type of the field (string, float64, etc.)
	Type string
	// Tag data for the field.
	Tag string
}

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL