base

package
v0.18.0-beta Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 17, 2024 License: MIT Imports: 19 Imported by: 43

Documentation

Index

Constants

View Source 
const OpenAPITemplate = `` /* 1538-byte string literal not displayed */
View Source 
const SecretKeyword = "__INSTILL_SECRET"

SecretKeyword is a keyword to reference a secret in a component configuration. When a component detects this value in a configuration parameter, it will used the pre-configured value, injected at initialization.

Variables

View Source 
var InstillAcceptFormatsMeta = jsonschema.MustCompileString("instillAcceptFormats.json", `{
	"properties" : {
		"instillAcceptFormats": {
			"type": "array",
			"items": {
				"type": "string"
			}
		}
	}
}`)
View Source 
var InstillFormatMeta = jsonschema.MustCompileString("instillFormat.json", `{
	"properties" : {
		"instillFormat": {
			"type": "string"
		}
	}
}`)

Functions

func CompileInstillAcceptFormats 

func CompileInstillAcceptFormats(sch *structpb.Struct) error

func CompileInstillFormat 

func CompileInstillFormat(sch *structpb.Struct) error

func ConvertFromStructpb 

func ConvertFromStructpb(from *structpb.Struct, to interface{}) error

ConvertFromStructpb converts from structpb.Struct to a struct

func ConvertToStructpb 

func ConvertToStructpb(from interface{}) (*structpb.Struct, error)

ConvertToStructpb converts from a struct to structpb.Struct

func FormatErrors 

func FormatErrors(inputPath string, e jsonschema.Detailed, errors *[]string)

func NewUnresolvedSecret 

func NewUnresolvedSecret(key string) error

NewUnresolvedSecret returns an end-user error signaling that the component configuration references a global secret that wasn't injected into the component.

func ReadFromSecrets 

func ReadFromSecrets(key string, secrets map[string]any) string

ReadFromSecrets reads a component secret from a secret map that comes from environment variable configuration.

Connection parameters are defined with snake_case, but the environment variable configuration loader replaces underscores by dots, so we can't use the parameter key directly. TODO using camelCase in configuration fields would fix this issue.

func RenderJSON 

func RenderJSON(tasksJSONBytes []byte, additionalJSONBytes map[string][]byte) ([]byte, error)

func TaskIDToTitle 

func TaskIDToTitle(id string) string

TaskIDToTitle builds a Task title from its ID. This is used when the `title` key in the task definition isn't present.

func TrimBase64Mime 

func TrimBase64Mime(b64 string) string

func Validate 

func Validate(data []*structpb.Struct, jsonSchema string, target string) error

Validate the input and output format

Types

type Connector 

type Connector struct {
	Logger *zap.Logger
	// contains filtered or unexported fields
}

Connector implements the common connector methods.

func (*Connector) GetConnectorDefinition 

func (c *Connector) GetConnectorDefinition(sysVars map[string]any, component *pipelinePB.ConnectorComponent) (*pipelinePB.ConnectorDefinition, error)

func (*Connector) GetID 

func (c *Connector) GetID() string

func (*Connector) GetLogger 

func (c *Connector) GetLogger() *zap.Logger

func (*Connector) GetTaskInputSchemas 

func (c *Connector) GetTaskInputSchemas() map[string]string

func (*Connector) GetTaskOutputSchemas 

func (c *Connector) GetTaskOutputSchemas() map[string]string

func (*Connector) GetUID 

func (c *Connector) GetUID() uuid.UUID

func (*Connector) IsSecretField 

func (c *Connector) IsSecretField(target string) bool

IsSecretField checks if the target field is secret field

func (*Connector) ListSecretFields 

func (c *Connector) ListSecretFields() ([]string, error)

ListSecretFields lists the secret fields by definition id

func (*Connector) LoadConnectorDefinition 

func (c *Connector) LoadConnectorDefinition(definitionJSONBytes []byte, tasksJSONBytes []byte, additionalJSONBytes map[string][]byte) error

LoadConnectorDefinition loads the connector definitions from json files

func (*Connector) UsageHandlerCreator 

func (c *Connector) UsageHandlerCreator() UsageHandlerCreator

UsageHandlerCreator returns a function to initialize a UsageHandler.

type ConnectorExecution 

type ConnectorExecution struct {
	Connector       IConnector
	SystemVariables map[string]any
	Connection      *structpb.Struct
	Task            string
}

ConnectorExecution implements the common methods for connector execution.

func (*ConnectorExecution) GetConnection 

func (e *ConnectorExecution) GetConnection() *structpb.Struct

func (*ConnectorExecution) GetConnector 

func (e *ConnectorExecution) GetConnector() IConnector

func (*ConnectorExecution) GetLogger 

func (e *ConnectorExecution) GetLogger() *zap.Logger

func (*ConnectorExecution) GetSystemVariables 

func (e *ConnectorExecution) GetSystemVariables() map[string]any

func (*ConnectorExecution) GetTask 

func (e *ConnectorExecution) GetTask() string

func (*ConnectorExecution) GetTaskInputSchema 

func (e *ConnectorExecution) GetTaskInputSchema() string

func (*ConnectorExecution) GetTaskOutputSchema 

func (e *ConnectorExecution) GetTaskOutputSchema() string

func (*ConnectorExecution) UsageHandlerCreator 

func (e *ConnectorExecution) UsageHandlerCreator() UsageHandlerCreator

UsageHandlerCreator returns a function to initialize a UsageHandler.

func (*ConnectorExecution) UsesSecret 

func (e *ConnectorExecution) UsesSecret() bool

UsesSecret indicates wether the connector execution is configured with global secrets. Components should override this method when they have the ability to be executed with global secrets.

type ExecutionWrapper 

type ExecutionWrapper struct {
	Execution IExecution
}

ExecutionWrapper performs validation and usage collection around the execution of a component.

func (*ExecutionWrapper) Execute 

func (e *ExecutionWrapper) Execute(ctx context.Context, inputs []*structpb.Struct) ([]*structpb.Struct, error)

Execute wraps the execution method with validation and usage collection.

type IComponent 

type IComponent interface {
	GetID() string
	GetUID() uuid.UUID
	GetLogger() *zap.Logger
	GetTaskInputSchemas() map[string]string
	GetTaskOutputSchemas() map[string]string

	UsageHandlerCreator() UsageHandlerCreator
}

IComponent is the interface that wraps the basic component methods. All component need to implement this interface.

type IConnector 

type IConnector interface {
	IComponent

	LoadConnectorDefinition(definitionJSON []byte, tasksJSON []byte, additionalJSONBytes map[string][]byte) error

	// Note: Some content in the definition JSON schema needs to be generated
	// by sysVars or component setting.
	GetConnectorDefinition(sysVars map[string]any, component *pipelinePB.ConnectorComponent) (*pipelinePB.ConnectorDefinition, error)

	CreateExecution(sysVars map[string]any, connection *structpb.Struct, task string) (*ExecutionWrapper, error)
	Test(sysVars map[string]any, connection *structpb.Struct) error

	IsSecretField(target string) bool
}

IConnector defines the methods of a connector component.

type IExecution 

type IExecution interface {
	GetTask() string
	GetLogger() *zap.Logger
	GetTaskInputSchema() string
	GetTaskOutputSchema() string
	GetSystemVariables() map[string]any

	UsesSecret() bool
	UsageHandlerCreator() UsageHandlerCreator

	Execute(context.Context, []*structpb.Struct) ([]*structpb.Struct, error)
}

IExecution allows components to be executed.

type IOperator 

type IOperator interface {
	IComponent

	LoadOperatorDefinition(definitionJSON []byte, tasksJSON []byte, additionalJSONBytes map[string][]byte) error

	// Note: Some content in the definition JSON schema needs to be generated
	// by sysVars or component setting.
	GetOperatorDefinition(sysVars map[string]any, component *pipelinePB.OperatorComponent) (*pipelinePB.OperatorDefinition, error)

	CreateExecution(sysVars map[string]any, task string) (*ExecutionWrapper, error)
}

IOperator defines the methods of an operator component.

type InstillAcceptFormatsCompiler 

type InstillAcceptFormatsCompiler struct{}

func (InstillAcceptFormatsCompiler) Compile 

func (InstillAcceptFormatsCompiler) Compile(ctx jsonschema.CompilerContext, m map[string]interface{}) (jsonschema.ExtSchema, error)

type InstillAcceptFormatsSchema 

type InstillAcceptFormatsSchema []string

func (InstillAcceptFormatsSchema) Validate 

func (s InstillAcceptFormatsSchema) Validate(ctx jsonschema.ValidationContext, v interface{}) error

type InstillFormatCompiler 

type InstillFormatCompiler struct{}

func (InstillFormatCompiler) Compile 

func (InstillFormatCompiler) Compile(ctx jsonschema.CompilerContext, m map[string]interface{}) (jsonschema.ExtSchema, error)

type InstillFormatSchema 

type InstillFormatSchema string

func (InstillFormatSchema) Validate 

func (s InstillFormatSchema) Validate(ctx jsonschema.ValidationContext, v interface{}) error

type Operator 

type Operator struct {
	Logger *zap.Logger
	// contains filtered or unexported fields
}

Operator implements the common operator methods.

func (*Operator) GetID 

func (o *Operator) GetID() string

func (*Operator) GetLogger 

func (o *Operator) GetLogger() *zap.Logger

func (*Operator) GetOperatorDefinition 

func (o *Operator) GetOperatorDefinition(sysVars map[string]any, component *pipelinePB.OperatorComponent) (*pipelinePB.OperatorDefinition, error)

func (*Operator) GetTaskInputSchemas 

func (o *Operator) GetTaskInputSchemas() map[string]string

func (*Operator) GetTaskOutputSchemas 

func (o *Operator) GetTaskOutputSchemas() map[string]string

func (*Operator) GetUID 

func (o *Operator) GetUID() uuid.UUID

func (*Operator) LoadOperatorDefinition 

func (o *Operator) LoadOperatorDefinition(definitionJSONBytes []byte, tasksJSONBytes []byte, additionalJSONBytes map[string][]byte) error

LoadOperatorDefinition loads the operator definitions from json files

func (*Operator) UsageHandlerCreator 

func (o *Operator) UsageHandlerCreator() UsageHandlerCreator

UsageHandlerCreator returns a no-op usage handler initializer. For the moment there are no use cases for usage collection in operators.

type OperatorExecution 

type OperatorExecution struct {
	Operator        IOperator
	SystemVariables map[string]any
	Task            string
}

OperatorExecution implements the common methods for operator execution.

func (*OperatorExecution) GetLogger 

func (e *OperatorExecution) GetLogger() *zap.Logger

func (*OperatorExecution) GetOperator 

func (e *OperatorExecution) GetOperator() IOperator

func (*OperatorExecution) GetSystemVariables 

func (e *OperatorExecution) GetSystemVariables() map[string]any

func (*OperatorExecution) GetTask 

func (e *OperatorExecution) GetTask() string

func (*OperatorExecution) GetTaskInputSchema 

func (e *OperatorExecution) GetTaskInputSchema() string

func (*OperatorExecution) GetTaskOutputSchema 

func (e *OperatorExecution) GetTaskOutputSchema() string

func (*OperatorExecution) UsageHandlerCreator 

func (e *OperatorExecution) UsageHandlerCreator() UsageHandlerCreator

UsageHandlerCreator returns a function to initialize a UsageHandler.

func (*OperatorExecution) UsesSecret 

func (e *OperatorExecution) UsesSecret() bool

UsesSecret indicates wether the operator execution is configured with global secrets. Components should override this method when they have the ability to be executed with global secrets.

type UsageHandler 

type UsageHandler interface {
	Check(ctx context.Context, inputs []*structpb.Struct) error
	Collect(ctx context.Context, inputs, outputs []*structpb.Struct) error
}

UsageHandler allows the component execution wrapper to add checks and collect usage metrics around a component execution.

func NewNoopUsageHandler 

func NewNoopUsageHandler(IExecution) (UsageHandler, error)

NewNoopUsageHandler is a no-op usage handler initializer.

type UsageHandlerCreator 

type UsageHandlerCreator func(IExecution) (UsageHandler, error)

UsageHandlerCreator returns a function to initialize a UsageHandler.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.