core

package
v0.2.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 16, 2019 License: Apache-2.0 Imports: 14 Imported by: 0

Documentation

Index

Examples

Constants

View Source
const DefaultPhaseVersion = uint32(0)

Variables

View Source
var PhaseInfoUndefined = PhaseInfo{/* contains filtered or unexported fields */}

Undefined entity, associated with an error

Unknown/Undefined transition. To be returned when an error is observed

Functions

This section is empty.

Types

type AllocationStatus

type AllocationStatus string
const (
	// This is the enum returned when there's an error
	AllocationUndefined AllocationStatus = "ResourceGranted"

	// Go for it
	AllocationStatusGranted AllocationStatus = "ResourceGranted"

	// This means that no resources are available globally.  This is the only rejection message we use right now.
	AllocationStatusExhausted AllocationStatus = "ResourceExhausted"

	// We're not currently using this - but this would indicate that things globally are okay, but that your
	// own namespace is too busy
	AllocationStatusNamespaceQuotaExceeded AllocationStatus = "NamespaceQuotaExceeded"
)

type EnqueueOwner

type EnqueueOwner func(id types.NamespacedName) error

When a change is observed, the owning entity with id types.NamespacedName can be triggered for re-validation

type EventsRecorder

type EventsRecorder interface {
	RecordRaw(ctx context.Context, ev PhaseInfo) error
}

Task events recorder, which get stored in the Admin. If this is invoked multiple times, multiple events will be sent to Admin. It is not recommended that one uses this interface, a transition will trigger an auto event to admin

type KubeClient

type KubeClient interface {
	// GetClient returns a client configured with the Config
	GetClient() client.Client

	// GetCache returns a cache.Cache
	GetCache() cache.Cache
}

TODO we may not want to expose this? A friendly controller-runtime client that gets passed to executors

type Phase

type Phase int8
const (
	// Does not mean an error, but simply states that we dont know the state in this round, try again later. But can be used to signal a system error too
	PhaseUndefined Phase = iota
	PhaseNotReady
	// Indicates plugin is not ready to submit the request as it is waiting for resources
	PhaseWaitingForResources
	// Indicates plugin has submitted the execution, but it has not started executing yet
	PhaseQueued
	// The system has started the pre-execution process, like container download, cluster startup etc
	PhaseInitializing
	// Indicates that the task has started executing
	PhaseRunning
	// Indicates that the task has completed successfully
	PhaseSuccess
	// Indicates that the Failure is recoverable, by re-executing the task if retries permit
	PhaseRetryableFailure
	// Indicate that the failure is non recoverable even if retries exist
	PhasePermanentFailure
)

func PhaseString

func PhaseString(s string) (Phase, error)

PhaseString retrieves an enum value from the enum constants string name. Throws an error if the param is not part of the enum.

func PhaseValues

func PhaseValues() []Phase

PhaseValues returns all values of the enum

func (Phase) IsAPhase

func (i Phase) IsAPhase() bool

IsAPhase returns "true" if the value is listed in the enum definition. "false" otherwise

func (Phase) IsFailure

func (p Phase) IsFailure() bool

func (Phase) IsSuccess

func (p Phase) IsSuccess() bool

func (Phase) IsTerminal

func (p Phase) IsTerminal() bool

Returns true if the given phase is failure, retryable failure or success

func (Phase) String

func (i Phase) String() string

type PhaseInfo

type PhaseInfo struct {
	// contains filtered or unexported fields
}

Additional info that should be sent to the front end. The Information is sent to the front-end if it meets certain criterion, for example currently, it is sent only if an event was not already sent for

func PhaseInfoFailed

func PhaseInfoFailed(p Phase, err *core.ExecutionError, info *TaskInfo) PhaseInfo

func PhaseInfoFailure

func PhaseInfoFailure(code, reason string, info *TaskInfo) PhaseInfo

func PhaseInfoInitializing

func PhaseInfoInitializing(t time.Time, version uint32, reason string) PhaseInfo

func PhaseInfoNotReady

func PhaseInfoNotReady(t time.Time, version uint32, reason string) PhaseInfo

Return in the case the plugin is not ready to start

func PhaseInfoQueued

func PhaseInfoQueued(t time.Time, version uint32, reason string) PhaseInfo

func PhaseInfoRetryableFailure

func PhaseInfoRetryableFailure(code, reason string, info *TaskInfo) PhaseInfo

func PhaseInfoRunning

func PhaseInfoRunning(version uint32, info *TaskInfo) PhaseInfo

func PhaseInfoSuccess

func PhaseInfoSuccess(info *TaskInfo) PhaseInfo

func PhaseInfoWaitingForResources

func PhaseInfoWaitingForResources(t time.Time, version uint32, reason string) PhaseInfo

Return in the case the plugin is not ready to start

func (PhaseInfo) Err

func (p PhaseInfo) Err() *core.ExecutionError

func (PhaseInfo) Info

func (p PhaseInfo) Info() *TaskInfo

func (PhaseInfo) Phase

func (p PhaseInfo) Phase() Phase

func (PhaseInfo) Reason

func (p PhaseInfo) Reason() string

func (PhaseInfo) String

func (p PhaseInfo) String() string

func (PhaseInfo) Version

func (p PhaseInfo) Version() uint32

type Plugin

type Plugin interface {
	// Unique ID for the plugin, should be ideally the same the ID in PluginEntry
	GetID() string
	// Properties desired by the plugin from the available set
	GetProperties() PluginProperties
	// The actual method that is invoked for every single task execution. The method should be a non blocking method.
	// It maybe invoked multiple times and hence all actions should be idempotent. If idempotency is not possible look at
	// Transitions to get some system level guarantees
	Handle(ctx context.Context, tCtx TaskExecutionContext) (Transition, error)
	// Called when the task is to be killed/aborted, because the top level entity was aborted or some other failure happened.
	// Abort should always be idempotent
	Abort(ctx context.Context, tCtx TaskExecutionContext) error
	// Finalize is always called, after Handle or Abort. Finalize should be an idempotent operation
	Finalize(ctx context.Context, tCtx TaskExecutionContext) error
}

Interface for the core Flyte plugin

type PluginEntry

type PluginEntry struct {
	// System wide unique identifier for the plugin
	ID TaskType
	// A list of all the task types for which this plugin is applicable.
	RegisteredTaskTypes []TaskType
	// A Lazy loading function, that will load the plugin. Plugins should be initialized in this method. It is guaranteed
	// that the plugin loader will be called before any Handle/Abort/Finalize functions are invoked
	LoadPlugin PluginLoader
	// Boolean that indicates if this plugin can be used as the default for unknown task types. There can only be
	// one default in the system
	IsDefault bool
}

An entry that identifies the CorePlugin

type PluginLoader

type PluginLoader func(ctx context.Context, iCtx SetupContext) (Plugin, error)

A Lazy loading function, that will load the plugin. Plugins should be initialized in this method. It is guaranteed that the plugin loader will be called before any Handle/Abort/Finalize functions are invoked

type PluginProperties

type PluginProperties struct {
	// Instructs the execution engine to not attempt to cache lookup or write for the node.
	DisableNodeLevelCaching bool
}

System level properties that this Plugin supports

type PluginStateReader

type PluginStateReader interface {
	// Retrieve state version that is currently stored
	GetStateVersion() uint8
	// Retrieve the typed state in t from the stored value. It also returns the stateversion.
	// If there is no state, t will be zero value, stateversion will be 0
	Get(t interface{}) (stateVersion uint8, err error)
}

Read previously written plugin state (previous round)

type PluginStateWriter

type PluginStateWriter interface {
	// Only the last call to this method is recorded. All previous calls are overwritten
	// This data is also not accessible until the next round.
	Put(stateVersion uint8, v interface{}) error
	// Resets the state to empty or zero value
	Reset() error
}

Write new plugin state for a plugin

type ResourceManager

type ResourceManager interface {
	AllocateResource(ctx context.Context, namespace ResourceNamespace, allocationToken string) (AllocationStatus, error)
	ReleaseResource(ctx context.Context, namespace ResourceNamespace, allocationToken string) error
}

Resource Manager manages a single resource type, and each allocation is of size one

type ResourceNamespace

type ResourceNamespace string

func (ResourceNamespace) CreateSubNamespace

func (r ResourceNamespace) CreateSubNamespace(namespace ResourceNamespace) ResourceNamespace

type ResourceRegistrar

type ResourceRegistrar interface {
	RegisterResourceQuota(ctx context.Context, namespace ResourceNamespace, quota int) error
}

type SecretManager

type SecretManager interface {
	Get(ctx context.Context, key string) (string, error)
}

type SetupContext

type SetupContext interface {
	// returns a callback mechanism that indicates that (workflow, task) is ready to be re-evaluated
	EnqueueOwner() EnqueueOwner
	// provides a k8s specific owner kind
	OwnerKind() string
	// a metrics scope to publish stats under
	MetricsScope() promutils.Scope
	// A kubernetes client to the bound cluster
	KubeClient() KubeClient
	// Returns a secret manager that can retrieve configured secrets for this plugin
	SecretManager() SecretManager
	// Returns a resource negotiator that the plugin can register resource quota against
	ResourceRegistrar() ResourceRegistrar
}

Passed to the Loader function when setting up a plugin

type SignalAsync

type SignalAsync func(ctx context.Context)

A simple fire-and-forget func

type TaskExecutionContext

type TaskExecutionContext interface {
	// Returns a resource manager that can be used to create reservations for limited resources
	ResourceManager() ResourceManager

	// Returns a secret manager that can retrieve configured secrets for this plugin
	SecretManager() SecretManager

	// Returns a method that allows a plugin to indicate that the task has a new update and can be invoked again to check for updates
	TaskRefreshIndicator() SignalAsync

	// Returns the max allowed dataset size that the outputwriter will accept
	MaxDatasetSizeBytes() int64

	// Returns a handle to the currently configured storage backend that can be used to communicate with the tasks or write metadata
	DataStore() *storage.DataStore

	// Returns a reader that retrieves previously stored plugin internal state. the state itself is immutable
	PluginStateReader() PluginStateReader

	// Returns a TaskReader, to retrieve task details
	TaskReader() TaskReader

	// Returns an input reader to retrieve input data
	InputReader() io.InputReader

	// Returns a handle to the Task's execution metadata.
	TaskExecutionMetadata() TaskExecutionMetadata

	// Provides an output sync of type io.OutputWriter
	OutputWriter() io.OutputWriter

	// Get a handle to the PluginStateWriter. Any mutations to the plugins internal state can be persisted using this
	// These mutation will be visible in the next round
	PluginStateWriter() PluginStateWriter

	// Get a handle to catalog client
	Catalog() catalog.AsyncClient

	// Returns a handle to the Task events recorder, which get stored in the Admin.
	EventsRecorder() EventsRecorder
}

An interface that is passed to every plugin invocation. It carries all meta and contextual information for the current task execution

type TaskExecutionID

type TaskExecutionID interface {
	GetGeneratedName() string
	GetID() core.TaskExecutionIdentifier
}

Simple Interface to expose the ExecutionID of the running Task

type TaskExecutionMetadata

type TaskExecutionMetadata interface {
	// The owning Kubernetes object
	GetOwnerID() types.NamespacedName
	// A specially generated task execution id, that is guaranteed to be unique and consistent for subsequent calls
	GetTaskExecutionID() TaskExecutionID
	GetNamespace() string
	GetOwnerReference() v12.OwnerReference
	GetOverrides() TaskOverrides
	GetLabels() map[string]string
	GetAnnotations() map[string]string
	GetK8sServiceAccount() string
}

TaskContext represents any execution information for a Task. It is used to communicate meta information about the execution or any previously stored information

type TaskInfo

type TaskInfo struct {
	// log information for the task execution
	Logs []*core.TaskLog
	// Set this value to the intended time when the status occurred at. If not provided, will be defaulted to the current
	// time at the time of publishing the event.
	OccurredAt *time.Time
	// Custom Event information that the plugin would like to expose to the front-end
	CustomInfo *structpb.Struct
}

func (*TaskInfo) String

func (t *TaskInfo) String() string

type TaskOverrides

type TaskOverrides interface {
	GetResources() *v1.ResourceRequirements
	GetConfig() *v1.ConfigMap
}

Interface to expose any overrides that have been set for this task (like resource overrides etc)

type TaskReader

type TaskReader interface {
	// Returns the core TaskTemplate
	Read(ctx context.Context) (*core.TaskTemplate, error)
}

An interface to access the TaskInformation

type TaskType

type TaskType = string

type Transition

type Transition struct {
	// contains filtered or unexported fields
}

A Plugin Handle method returns a Transition. This transition indicates to the Flyte framework that if the plugin wants to continue "Handle"ing this task, or if wants to move the task to success, attempt a retry or fail. The transition automatically sends an event to Admin service which shows the plugin provided information in the Console/cli etc The information to be published is in the PhaseInfo structure. Transition Type indicates the type of consistency for subsequent handle calls in case the phase info results in a non terminal state. the PhaseInfo structure is very important and is used to record events in Admin. Only if the Phase + PhaseVersion was not previously observed, will an event be published to Admin there are only a configurable number of phase-versions usable. Usually it is preferred to be a monotonically increasing sequence

func DoTransition

func DoTransition(info PhaseInfo) Transition

Same as DoTransition, but TransitionTime is always Ephemeral

func DoTransitionType

func DoTransitionType(ttype TransitionType, info PhaseInfo) Transition

Creates and returns a new Transition based on the PhaseInfo.Phase Phases: PhaseNotReady, PhaseQueued, PhaseInitializing, PhaseRunning will cause the system to continue invoking Handle

func (Transition) Info

func (t Transition) Info() PhaseInfo

func (Transition) String

func (t Transition) String() string
Example
trns := DoTransitionType(TransitionTypeBarrier, PhaseInfoUndefined)
fmt.Println(trns.String())
Output:

TransitionTypeBarrier,Phase<PhaseUndefined:0 <nil> Reason:>

func (Transition) Type

func (t Transition) Type() TransitionType

type TransitionType

type TransitionType int

Type of Transition, refer to Transition to understand what transition means

const (
	// The transition is eventually consistent. For all the state written may not be visible in the next call, but eventually will persist
	// Best to use when the plugin logic is completely idempotent. This is also the most performant option.
	TransitionTypeEphemeral TransitionType = iota
	// This transition tries its best to make the latest state visible for every consecutive read. But, it is possible
	// to go back in time, i.e. monotonic consistency is violated (in rare cases).
	TransitionTypeBarrier
)

func TransitionTypeString

func TransitionTypeString(s string) (TransitionType, error)

TransitionTypeString retrieves an enum value from the enum constants string name. Throws an error if the param is not part of the enum.

func TransitionTypeValues

func TransitionTypeValues() []TransitionType

TransitionTypeValues returns all values of the enum

func (TransitionType) IsATransitionType

func (i TransitionType) IsATransitionType() bool

IsATransitionType returns "true" if the value is listed in the enum definition. "false" otherwise

func (TransitionType) String

func (i TransitionType) String() string

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL