phly

package module
v0.0.0-...-e362888 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 7, 2023 License: BSD-3-Clause Imports: 23 Imported by: 0

README

phly

A tool for running pipeline operations.

WARNING: Currently in development, API will change.

DOUBLE WARNING: This is currently being rewritten, is totally torn up, and close to non-functional.

A pipeline processing framework. Phly has two main pieces:

  • The phly framework that manages registering and instantiating nodes, and loading and running pipelines.
  • The phly app, a lightweight application designed to make it easy to compile in additional nodes.

Building

  • Install Go 1.10.3 or later.
  • From the command line, enter directory phly\phly.
  • Type go get to get all dependencies.
  • Type go build to build the app.

Alternatively, the phly library can be compiled into other Go apps.

Use

The work so far has been on the framework. The actual application currently does nothing but scale images. To that end, running the app will load the data/scale_image.json pipeline, which loads an example image and scales it.

Examples (compiled for Windows):

  • phly.exe. Run the app, which currently defaults to running the data/scale_image.json pipeline.
  • phly.exe -nodes. Display all installed nodes.
  • phly.exe -markdown. Generate markdown for all installed nodes.
  • phly.exe -vars. Display all node-defined variables.

Nodes

  • Batch (phly/batch). Perform multiple actions in parallel.
  • Files (phly/files). Create file lists from file names and folders. Produce a single doc with a single page.
    • cfg value. A value directly entered into the cfg file. Use this if no cla or env are present.
    • cfg env. A value from the environment variables. Use this if no cla is available.
    • cfg cla. A value from the command line arguments.
    • cfg expand. (true or false). When true, folders are expanded to the file contents.
    • output out. The file list.
  • Pipeline (phly/pipeline). Run an internal pipeline.
  • Text (phly/text). Acquire text from the cfg values. If a cla is available use that. If no cla, use the env. If no env, use the value.
    • cfg value. A value directly entered into the cfg file. Use this if no cla or env are present.
    • cfg env. A value from the environment variables. Use this if no cla is available.
    • cfg cla. A value from the command line arguments.
    • output out. The text output.

Documentation

Index

Constants

View Source
const (
	BadRequestErrCode = 1000 + iota
	IllegalErrCode
	MissingErrCode
	ParseErrCode
)
View Source
const (
	WhatPins = "pins"
	WhatStop = "stop"
)

Variables

View Source
var (
	BadRequestErr = errors.New("Bad request")
)

Functions

func ErrorsEqual

func ErrorsEqual(a, b error) bool

ErrorsEqual() returns true if the errors are equivalent.

func MergeErrors

func MergeErrors(err ...error) error

MergeErrors() answers the first non-nil error in the list.

func NewBadRequestError

func NewBadRequestError(msg string) error

func NewIllegalError

func NewIllegalError(msg string) error

func NewMissingError

func NewMissingError(msg string) error

func NewParseError

func NewParseError(err error) error

func Register

func Register(fac NodeFactory) error

func RegisterVar

func RegisterVar(name, descr string, optional_value interface{})

RegisterVar() registers a variable with the system, which becomes part of the help system. Multiple variables with the same name can be registered, since each package can have its own non-conflicting variables. You can supply an optional value and the var wil be including when replacing vars through the Environment, although only a single var with the same name can have a value.

func StringPinsEqual

func StringPinsEqual(a, b Pins) bool

func StringPinsToJson

func StringPinsToJson(pins Pins) string

func WalkItems

func WalkItems(pins Pins, channel string, fn ItemFunc)

WalkItems iterates over each item on the channel

func WalkStringItems

func WalkStringItems(pins Pins, channel string, fn StringItemFunc)

WalkStringItems iterates over each string item on the channel

Types

type BuildPinsCmd

type BuildPinsCmd int
const (
	PbsChan BuildPinsCmd // The following command must be a string. It will be used to
	// start a new pin with the given channel name.
	PbsDoc // Start a new doc. Can be ommitted after starting a channel. Only necessary

)

type CfgDescr

type CfgDescr struct {
	Name    string
	Purpose string
}

type Doc

type Doc struct {
	Header
	MimeType string
	Items    []interface{}
}

Doc describes a single abstract document. It includes a user-defined header, an optional content type, and optional pages.

func NewStringDoc

func NewStringDoc(n ...string) *Doc

Create a new doc on string items

func (*Doc) AllItem

func (d *Doc) AllItem(index int) interface{}

func (*Doc) AllItems

func (d *Doc) AllItems() []interface{}

func (*Doc) AppendItem

func (d *Doc) AppendItem(item interface{})

func (*Doc) StringItem

func (d *Doc) StringItem(index int) string

func (*Doc) StringItems

func (d *Doc) StringItems() []string

type Docs

type Docs struct {
	Docs []*Doc
}

Docs is a wrapper on a slice of Doc structs, used to add convenience management functions.

func NewDocs

func NewDocs(docs ...*Doc) *Docs

func (Docs) AllItem

func (d Docs) AllItem(index int) interface{}

func (Docs) AllItems

func (d Docs) AllItems() []interface{}

func (Docs) StringItem

func (d Docs) StringItem(index int) string

func (Docs) StringItems

func (d Docs) StringItems() []string

type Environment

type Environment interface {
	// FindFile() answers the full path to the phlyp by searching paths for name.
	FindFile(name string) string
	// FindReader() answers a reader for the given name.
	FindReader(name string) io.Reader

	// Utility for replacing strings with a collection of my vars and supplied pairs.
	ReplaceVars(s string, pairs ...interface{}) string
}

Environment provides access to the system environment.

type Header struct {
	Values interface{}
}

func (*Header) GetInt

func (h *Header) GetInt(path string) (int, bool)

func (*Header) GetString

func (h *Header) GetString(path string) (string, bool)

func (*Header) SetHeader

func (h *Header) SetHeader(values interface{})

func (*Header) SetInt

func (h *Header) SetInt(path string, value int) error

func (*Header) SetString

func (h *Header) SetString(path, value string) error

type InstantiateArgs

type InstantiateArgs struct {
	Env Environment
}

InstantiateArgs provides information during the instantiation phase.

type ItemFunc

type ItemFunc func(channel string, doc *Doc, index int, item interface{})

ItemFunc is a callback for a single item in a pin doc.

type Items

type Items interface {
	// Get collections in various formats
	AllItems() []interface{}
	StringItems() []string

	// Get at index in various formats
	AllItem(index int) interface{}
	StringItem(index int) string
}

Items provides common behaviour on an abstract collection.

type Msg

type Msg struct {
	What    string
	Payload interface{}
}

Msg is an abstract node message.

func MsgFromPins

func MsgFromPins(pins Pins) Msg

func MsgFromStop

func MsgFromStop(err error) Msg

type Node

type Node interface {
	Describe() NodeDescr

	// Process processes input, sending any results to output.
	// stage provides minimal lifecycle information: whether this node is starting or not.
	// input is the collection of inputs to process.
	// output is used to send any output to the calling graph. It is also used to
	// request StopNode() be called if processing should end.
	// error is used to report any errors; not that it will immediately stop all
	// graph processing.
	Process(args ProcessArgs, stage NodeStage, input Pins, output NodeOutput) error

	// Request to stop the node.
	StopNode(args StoppedArgs) error
}

Node performs abstract document processing.

type NodeDescr

type NodeDescr struct {
	Id          string
	Name        string
	Purpose     string
	Cfgs        []CfgDescr
	StartupPins []PinDescr
	InputPins   []PinDescr
	OutputPins  []PinDescr
}

NodeDescr describes a node.

func (*NodeDescr) ClaString

func (n *NodeDescr) ClaString() string

func (*NodeDescr) FindInput

func (n *NodeDescr) FindInput(name string) *PinDescr

func (*NodeDescr) FindOutput

func (n *NodeDescr) FindOutput(name string) *PinDescr

func (*NodeDescr) MarkdownString

func (n *NodeDescr) MarkdownString() string

type NodeFactory

type NodeFactory interface {
	Describe() NodeDescr
	Instantiate(args InstantiateArgs, tree interface{}) (Node, error)
}

NodeFactory instantiates a node. All nodes should have their factories installed in the bootstrap.

type NodeOutput

type NodeOutput interface {
	SendPins(Pins)
	SendMsg(Msg)
}

NodeOutput sends data from the node to its parent runner.

type NodeStage

type NodeStage string
const (
	NodeStarting NodeStage = "starting"
	NodeRunning            = "running"
)

type PhlyError

type PhlyError struct {
	// contains filtered or unexported fields
}

func (*PhlyError) Error

func (e *PhlyError) Error() string

func (*PhlyError) ErrorCode

func (e *PhlyError) ErrorCode() int

type PinBuilder

type PinBuilder struct {
	// contains filtered or unexported fields
}

func (PinBuilder) Add

func (b PinBuilder) Add(name string, item *Doc) PinBuilder

func (PinBuilder) Pins

func (b PinBuilder) Pins() Pins

type PinDescr

type PinDescr struct {
	Name    string
	Purpose string
}

type Pins

type Pins interface {
	GetPin(name string) Docs
	WalkPins(fn PinsWalkFunc)
}

Pins describes a collection of documents attached to named pins. It is meant for clients that are consuming pins.

func BuildPins

func BuildPins(cmds ...interface{}) (Pins, error)

BuildPins() builds a new Pins object according to the command stream. See the tokens for the rules. Any cmd that is not a token or made use of by the last token becomes a new item in the current doc. Example. The first command defaults to being the channel (so it must be a string).

BuildPins("input", "item1", "item2")

Example. You can use the channel command to create a new channel at any point.

BuildPins(PbsChannel, "input", "item1", "item2")

Example. You can use the doc command to create a new doc at any point.

BuildPins(PbsChannel, "input", "doc1_item1", PbsDoc, "doc2_item1")

func MustBuildPins

func MustBuildPins(cmds ...interface{}) Pins

MustBuildPins() is identical to BuildPins but it will panic on an error. Intended only for testing

func RunApp

func RunApp() (Pins, error)

type PinsWalkFunc

type PinsWalkFunc func(name string, docs Docs)

PinsWalkFunc iterates over each pin in a Pins collection.

type Pipeline

type Pipeline interface {
	Run(args StartArgs, input Pins) error
	Start(args StartArgs, input Pins) error
	Stop() error
	Wait() error
}

func LoadPipeline

func LoadPipeline(name string) (Pipeline, error)

func ReadPipeline

func ReadPipeline(r io.Reader) (Pipeline, error)

type ProcessArgs

type ProcessArgs struct {
	// contains filtered or unexported fields
}

ProcessArgs provides arguments to the node during processing.

func (*ProcessArgs) ClaValue

func (r *ProcessArgs) ClaValue(name string) string

ClaValue() answers the command line argument value for the given name.

func (*ProcessArgs) Env

func (r *ProcessArgs) Env() Environment

func (*ProcessArgs) Filename

func (r *ProcessArgs) Filename(rel string) string

Filename() answers an absolute filename for the supplied filename. Absolute filenames are returned as-is. Relative filenames are made relative to the cfg that generated the pipeline.

type SortNodeFactory

type SortNodeFactory []NodeFactory

func (SortNodeFactory) Len

func (s SortNodeFactory) Len() int

func (SortNodeFactory) Less

func (s SortNodeFactory) Less(i, j int) bool

func (SortNodeFactory) Swap

func (s SortNodeFactory) Swap(i, j int)

type SortVars

type SortVars []varDescr

func (SortVars) Len

func (s SortVars) Len() int

func (SortVars) Less

func (s SortVars) Less(i, j int) bool

func (SortVars) Swap

func (s SortVars) Swap(i, j int)

type StartArgs

type StartArgs struct {
	Cla map[string]string // Command line arguments
	// contains filtered or unexported fields
}

StartArgs provides arguments when starting the pipeline.

type StopPayload

type StopPayload struct {
	Err error
}

type StoppedArgs

type StoppedArgs struct {
}

StoppedArgs provides information as nodes are stopping.

type StringItemFunc

type StringItemFunc func(channel string, doc *Doc, index int, item string)

StringItemFunc is a callback for a single string item in a pin doc.

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL