taskcluster-worker: github.com/taskcluster/taskcluster-worker/plugins Index | Files | Directories

package plugins

import "github.com/taskcluster/taskcluster-worker/plugins"

Package plugins defines interfaces to be implemented by feature plugins.

A plugin is an object that gets called for each step in the task execution flow. At each step the plugin will get access to engine and runtime object, such as the SandboxBuilder, the Sandbox, the SandboxContext, etc. A plugin is then supposed to implement a specific feature using these objects, this could be live logging, artifact uploading, attachment of proxies, mouting of caches, archival of caches, and many other things.

A plugin does not get any plugin specific task.payload space, instead the options required by a plugin must be part of the task.payload. Hence, all plugins must be available on all platforms. To facilitate this, plugins should not use platform specific APIs, instead they should rely on interface offered by the engine objects to do so, fail gracefully if an engine method is not supported in a given configuration and returns ErrFeatureNotSupported.

When a plugin encounters an unsupported feature that it needs, it may either return a MalformedPayloadError, or simply ignore the error and workaround it.

Plugin packages should provide a method:

NewXXXPluginFactory(engine.Engine,*runtime.EngineContext) PluginFactory

and have it registered in pluginmanager.go

At high-level, PluginFactory stores the global state owned by the plugin, and Plugin stores the task-specific state owned by the plugin. A new Plugin instance will be created for each task.

In summary, plugins are not really "plugins", they are merely abstractions that allows us to implement features in complete isolation. Maybe they will become more flexible in the future, but there is no need to design for this at this point.


Package Files

doc.go plugin.go pluginmanager.go pluginprovider.go

func PluginManagerConfigSchema Uses

func PluginManagerConfigSchema() schematypes.Object

PluginManagerConfigSchema returns configuration for PluginOptions.Config for NewPluginManager.

func Plugins Uses

func Plugins() map[string]PluginProvider

Plugins returns map from plugin name to PluginProviders.

func Register Uses

func Register(name string, provider PluginProvider)

Register will register a PluginProvider. This is meant to be called from init(), and will panic if name is already used by another plugin.

type Plugin Uses

type Plugin interface {
    // Documentation returns a list of sections with end-user documentation.
    // These sections will be combined with documentation sections from all
    // enabled plugins in-order to form end-user documentation.
    Documentation() []runtime.Section

    // PayloadSchema returns a schematypes.Object with the properties for
    // for the TaskPluginOptions.Payload property.
    // Note: this will be merged with payload schemas from engine and other
    // plugins, thus, it cannot contain conflicting properties. Furthermore the
    // metadata will be discarded and additionalProperties will not be allowed.
    PayloadSchema() schematypes.Object

    // NewTaskPlugin method will be called once for each task. The TaskPlugin
    // instance returned will be called for each stage in the task execution.
    // NewTaskPlugin will be called in parallel with NewSandboxBuilder(), making
    // it a good place to start long-running operations, you then have to take
    // care to clean-up in Dispose() if they are still running.
    // You should wait for your long-running operations to finish in
    //  BuildSandbox() or whatever hook you need them in.
    // Plugins implementing logging should not return an error here, as it
    // naturally follows that such an error can't be logged if no logging plugin
    // is created.
    // Implementors may return TaskPluginBase{}, if the plugin doesn't have any
    // hooks for the given tasks.
    // Non-fatal errors: MalformedPayloadError
    NewTaskPlugin(options TaskPluginOptions) (TaskPlugin, error)

    // ReportIdle is called if the worker is idle prior to polling for tasks.
    // The durationSinceBusy is the time since the worker as last busy.
    // This is a perfect time to initiate a graceful stop the worker by
    // calling runtime.Environment.Worker.StopGracefully(). A plugin can call
    // StopGracefully() at any time, causing the worker to stop polling.
    // This hook is mainly useful as a way to gracefully stop the worker without
    // entering a new billing cycle, which might happen if you gracefully stop
    // while the worker is processing one or more tasks.
    // Warning: This hook is called as part of the worker claim loop, hence, it
    // strongly to execution time in this method
    ReportIdle(durationSinceBusy time.Duration)

    // ReportNonFatalError is called if the worker encountered a non-fatal error.
    // Plugins can use this to implement a heuristic for shutting down after
    // certain number of non-fatal errors, or purging caches. This is mainly
    // intended to allow a configurable plugin to decide if the worker should
    // stop in response to a non-fatal error.
    // The specific non-fatal error message is not available to the heuristic.
    // If a special heuristic is desired for a special non-fatal error, then this
    // should be handled in the plugin/engine where the error origins.

    // Dispose is called when the worker is stopping, a plugin should free all
    // resources and halt all background processes.
    // This is important for testing, but also for relevant when running on
    // bare metal machines that aren't re-imaged between worker restarts.
    Dispose() error

Plugin is a plugin to the worker, for each task NewTaskPlugin is created. The Plugin instance is responsible for creating these objects and managing data shared between TaskPlugin instances.

All methods on this interface must be thread-safe.

type PluginBase Uses

type PluginBase struct{}

PluginBase is a base implementation of the Plugin interface, it just handles all methods and does nothing.

Implementors should embed this to ensure forward compatibility when we add new optional methods.

func (PluginBase) Dispose Uses

func (PluginBase) Dispose() error

Dispose does nothing

func (PluginBase) Documentation Uses

func (PluginBase) Documentation() []runtime.Section

Documentation returns no documentation.

func (PluginBase) NewTaskPlugin Uses

func (PluginBase) NewTaskPlugin(TaskPluginOptions) (TaskPlugin, error)

NewTaskPlugin returns TaskPluginBase{} which ignores all the stages.

func (PluginBase) PayloadSchema Uses

func (PluginBase) PayloadSchema() schematypes.Object

PayloadSchema returns a schema for an empty object for plugins that doesn't take any payload.

func (PluginBase) ReportIdle Uses

func (PluginBase) ReportIdle(time.Duration)

ReportIdle does nothing

func (PluginBase) ReportNonFatalError Uses

func (PluginBase) ReportNonFatalError()

ReportNonFatalError does nothing

type PluginManager Uses

type PluginManager struct {
    // contains filtered or unexported fields

A PluginManager combines a collection of plugins and implements an interface similar to Plugin. The interface is not exactly the same, notably, if there is an error, the PluginManager still returns a TaskPlugin wrapping plugins that didn't fail, such that artifacts and logs can be uploaded.

Furthermore, the PluginManager doesn't require that the task payload matches the PayloadSchema(), as such a requirement would preclude running logging plugins to upload logs for the task. Which is important as errors in the task log makes no sense, if the task log isn't uploaded.

func NewPluginManager Uses

func NewPluginManager(options PluginOptions) (*PluginManager, error)

NewPluginManager loads all plugins not disabled in configuration and returns a Plugin implementation that wraps all of the plugins.

This expects options.Config satisfying schema from PluginManagerConfigSchema().

Note. While the PluginManager implements the Plugin interface, it does have different semantics for NewTaskPlugin(). This is because we need to run other TaskPlugins (such as logging), even if one of the TaskPlugins fails to be created or there is a schema validation error.

func (*PluginManager) Dispose Uses

func (pm *PluginManager) Dispose() error

Dispose calls Dispose on all the managed plugins.

func (*PluginManager) Documentation Uses

func (pm *PluginManager) Documentation() []runtime.Section

Documentation will collect documentation from all managed plugins.

func (*PluginManager) NewTaskPlugin Uses

func (pm *PluginManager) NewTaskPlugin(options TaskPluginOptions) (TaskPlugin, error)

NewTaskPlugin constructs a TaskPlugin wrapping all the managed plugins whose PayloadSchema is satisfied by options.Payload.

This method always returns a TaskPlugin, even if there is errors. Because plugins that don't fail still want their Exception hook invoked.

func (*PluginManager) PayloadSchema Uses

func (pm *PluginManager) PayloadSchema() schematypes.Object

PayloadSchema returns the 'task.payload' schema expected by plugins.

func (*PluginManager) ReportIdle Uses

func (pm *PluginManager) ReportIdle(durationSinceBusy time.Duration)

ReportIdle call ReportIdle on all the managed plugins.

func (*PluginManager) ReportNonFatalError Uses

func (pm *PluginManager) ReportNonFatalError()

ReportNonFatalError calls ReportNonFatalError on all the managed plugins.

type PluginOptions Uses

type PluginOptions struct {
    Environment *runtime.Environment
    Engine      engines.Engine
    Monitor     runtime.Monitor
    Config      interface{}

PluginOptions is a wrapper for the arguments/options given when instantiating a Plugin using PluginProvider.

We wrap all arguments so that we can add additional properties without breaking source compatibility with older plugins.

type PluginProvider Uses

type PluginProvider interface {
    NewPlugin(options PluginOptions) (Plugin, error)

    // ConfigSchema returns schema for the PluginOptions.Config property.
    // May return nil, if no configuration should be given.
    ConfigSchema() schematypes.Schema

The PluginProvider interface must be implemented and registered by anyone implementing a Plugin.

If an implementor can determine that a plugin isn't available at compile-time it is preferred not to register the plugin.

type PluginProviderBase Uses

type PluginProviderBase struct{}

PluginProviderBase is a base struct that provides empty implementations of some methods for PluginProvider

Implementors of PluginProvider should embed this struct to ensure forward compatibility when we add new optional method to PluginProvider.

func (PluginProviderBase) ConfigSchema Uses

func (PluginProviderBase) ConfigSchema() schematypes.Schema

ConfigSchema returns a nil schema.

type TaskPlugin Uses

type TaskPlugin interface {
    // BuildSandbox is called once NewSandboxBuilder() has returned.
    // This is the place to wait for downloads and other expensive operations to
    // finished, before mounting caches, proxies, etc. and returning.
    // Non-fatal errors: MalformedPayloadError
    BuildSandbox(sandboxBuilder engines.SandboxBuilder) error

    // Started is called once the sandbox has started execution. This is a good
    // place to hook if you want to do interactive things.
    // Non-fatal errors: MalformedPayloadError
    Started(sandbox engines.Sandbox) error

    // Stopped is called once the sandbox has terminated.
    // This is a good place to upload artifacts, logs, check exit code, and start
    // to clean-up resources if such clean-up is expected to take a while.
    // This will return false if the operation could not be completed successfully.
    // Such as artifact upload failure or files not existing.  If this returns false
    // it shall be assumed that the task has failed and should be reported as a failure.
    // Non-fatal errors: MalformedPayloadError
    Stopped(result engines.ResultSet) (bool, error)

    // Finished is called once the sandbox has terminated and Stopped() have been
    // called.
    // At this stage the task-specific log is closed, and attempts to log data
    // using the previous TaskContext will fail. That makes this a good place to
    // upload logs and any processing of the logs. In fact logging is the primary
    // motivation for this stage.
    // As there is no logging in this method, it's not recommend to do anything
    // that may fail here.
    Finished(success bool) error

    // Exception is called once the task is resolved exception. This may happen
    // instead of calls to Prepare(), BuildSandbox(), Started(), Stopped(), or
    // Finished().
    // This is a good place for best-effort to upload artifacts and logs that you
    // wish to persist. Naturally, log messages written at this stage will be
    // dropped and all error messages will be fatal.
    // Implementors should be aware that additional reasons may be added in the
    // future. Therefore they must handle the default case, if switching on the
    // reason parameter.
    Exception(reason runtime.ExceptionReason) error

    // Dispose is called once everything is done and it's time for clean-up.
    // This method will be invoked following Finished() or Exception(). It is then
    // the responsibility of the implementor to abort or wait for any long-running
    // processes and clean-up any resources held.
    Dispose() error

TaskPlugin holds the task-specific state for a plugin

Each method on this interface represents stage in the task execution and will be called when this stage is reached. The methods are allowed to take significant amounts of time, as they will run asynchronously.

These methods does not have to be thread-safe, we will never call the next method, before the previous method has returned.

Implementors of this interface should be sure to embed TaskPluginBase. This will do absolutely nothing, but provide empty implementations for any current and future methods that isn't implemented.

The methods are called in the order listed here with the exception of Exception() which may be called following any method, and Dispose() which will always be called as a final step allowing you to clean up.

type TaskPluginBase Uses

type TaskPluginBase struct{}

TaskPluginBase is a base implementation of the TaskPlugin interface, it just handles all methods and does nothing.

Implementors should embed this to ensure forward compatibility when we add new optional methods.

func (TaskPluginBase) BuildSandbox Uses

func (TaskPluginBase) BuildSandbox(engines.SandboxBuilder) error

BuildSandbox ignores the sandbox building stage.

func (TaskPluginBase) Dispose Uses

func (TaskPluginBase) Dispose() error

Dispose ignores the stage where resources are disposed.

func (TaskPluginBase) Exception Uses

func (TaskPluginBase) Exception(reason runtime.ExceptionReason) error

Exception ignores the stage where a task is resolved exception

func (TaskPluginBase) Finished Uses

func (TaskPluginBase) Finished(success bool) error

Finished ignores the stage where a task has been finished

func (TaskPluginBase) Started Uses

func (TaskPluginBase) Started(engines.Sandbox) error

Started ignores the stage where the sandbox has started

func (TaskPluginBase) Stopped Uses

func (TaskPluginBase) Stopped(engines.ResultSet) (bool, error)

Stopped ignores the stage where the sandbox has returned a ResultSet, and returns true saying the task was successful, as not to poison the water.

type TaskPluginOptions Uses

type TaskPluginOptions struct {
    TaskInfo    *runtime.TaskInfo
    TaskContext *runtime.TaskContext
    Payload     map[string]interface{}
    Monitor     runtime.Monitor

The TaskPluginOptions is a wrapper for the set of arguments given to NewTaskPlugin.

We wrap the arguments in a single argument to maintain source compatibility when introducing additional arguments.


artifactsPackage artifacts provides a taskcluster-worker plugin that uploads artifacts when sandbox execution has stopped.
cachePackage cache provides a cache plugin for taskcluster-worker
envPackage env provides a taskcluster-worker plugin that injects environment variables into the task environments.
interactivePackage interactive implements the plugin that serves the interactive display and shell sessions over websockets.
interactive/displayclientPackage displayclient provides a golang implementation of websockify, transforming a websocket connection to an ioext.ReadWriteCloser object.
interactive/shellclientPackage shellclient provides a wrapper for demuxing a shell websocket and exposing the stdout/stderr streams as well as offering a way to provide the stdin stream.
interactive/shellconstsPackage shellconsts contains constants shared between shell server and client which is split into different packages to reduce the binary size of potential commandline clients.
livelogPackage livelog provides a taskcluster-worker plugin that makes the task log available as a live log during task execution and finally uploads it as a static log.
logprefixPackage logprefix provides a taskcluster-worker plugin that prefixes all task logs with useful debug information such as taskId, workerType, as well as configurable constants.
maxruntimePackage maxruntime provides a plugin for taskcluster-worker which can enforce a maximum runtime upon tasks.
rebootPackage reboot provides a taskcluster-worker plugin that stops the worker after certain number of tasks or given amount of time.
relengapiPackage relengapi provides a taskcluster-worker plugin that exposes a proxy that forward requests to relengapi.
stoponerrorPackage stoponerror implements a very simple plugin that stops the worker gracefully if an non-fatal error is encountered.
successPackage success implements a very simple plugin that looks that the ResultSet.Success() value to determine if the process from the sandbox exited successfully.
tasklogPackage tasklog provides a taskcluster-worker plugin that uploads a static task.log when the task is finished.
tcproxyPackage tcproxy provides a taskcluster-worker plugin that exposes a proxy that signs requests with taskcluster credentials matching task.scopes.
watchdogPackage watchdog provides a taskcluster-worker plugin that pokes a watchdog whenever a task makes progress or the worker reports that it's idle.

Package plugins imports 8 packages (graph) and is imported by 17 packages. Updated 2018-06-14. Refresh now. Tools for package owners.