automation

package
v0.14.5 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 30, 2023 License: Apache-2.0 Imports: 18 Imported by: 0

Documentation

Overview

Package automation contains code to execute high-level cluster operations (e.g. resharding) as a series of low-level operations (e.g. vtctl, shell commands, ...).

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func AddMissingTaskID

func AddMissingTaskID(tc []*automationpb.TaskContainer, taskIDGenerator *IDGenerator)

AddMissingTaskID assigns a task id to each task in "tc".

func AddTask

func AddTask(t *automationpb.TaskContainer, taskName string, parameters map[string]string)

AddTask adds a new task to an existing task container.

func ExecuteVtctl

func ExecuteVtctl(ctx context.Context, server string, args []string) (string, error)

ExecuteVtctl runs vtctl using vtctlclient. The stream of Event messages is concatenated into one output string. Additionally, the start and the end of the command will be logged to make it easier to debug which command was executed and how long it took.

func ExecuteVtworker

func ExecuteVtworker(ctx context.Context, server string, args []string) (string, error)

ExecuteVtworker executes the vtworker command in "args" via an RPC to "server". The output of the RPC, a stream of LoggerEvent messages, is concatenated into one output string. If a retryable error is encountered (e.g. the vtworker process is already executing another command), this function will keep retrying infinitely until "ctx" is cancelled.

func MarkTaskFailed

func MarkTaskFailed(t *automationpb.Task, output string, err error)

MarkTaskFailed marks the task as failed.

func MarkTaskSucceeded

func MarkTaskSucceeded(t *automationpb.Task, output string)

MarkTaskSucceeded marks the task as done.

func NewTask

func NewTask(taskName string, parameters map[string]string) *automationpb.Task

NewTask creates a new task protobuf message for "taskName" with "parameters".

func NewTaskContainer

func NewTaskContainer() *automationpb.TaskContainer

NewTaskContainer creates an empty task container. Use AddTask() to add tasks to it.

func NewTaskContainerWithSingleTask

func NewTaskContainerWithSingleTask(taskName string, parameters map[string]string) *automationpb.TaskContainer

NewTaskContainerWithSingleTask creates a new task container with exactly one task.

Types

type ClusterOperationInstance

type ClusterOperationInstance struct {
	*automationpb.ClusterOperation
	// contains filtered or unexported fields
}

ClusterOperationInstance is a runtime type which enhances the protobuf message "ClusterOperation" with runtime specific data. Unlike the protobuf message, the additional runtime data will not be part of a checkpoint. Methods of this struct are not thread-safe.

func NewClusterOperationInstance

func NewClusterOperationInstance(clusterOpID string, initialTask *automationpb.TaskContainer, taskIDGenerator *IDGenerator) ClusterOperationInstance

NewClusterOperationInstance creates a new cluster operation instance with one initial task.

func (ClusterOperationInstance) Clone

Clone creates a deep copy of the inner protobuf. Other elements e.g. taskIDGenerator are not deep-copied.

func (*ClusterOperationInstance) InsertTaskContainers

func (c *ClusterOperationInstance) InsertTaskContainers(newTaskContainers []*automationpb.TaskContainer, pos int)

InsertTaskContainers inserts "newTaskContainers" at pos in the current list of task containers. Existing task containers will be moved after the new task containers.

type CopySchemaShardTask

type CopySchemaShardTask struct {
}

CopySchemaShardTask runs vtctl CopySchemaShard to copy the schema from one shard to another.

func (*CopySchemaShardTask) OptionalParameters

func (t *CopySchemaShardTask) OptionalParameters() []string

OptionalParameters is part of the Task interface.

func (*CopySchemaShardTask) RequiredParameters

func (t *CopySchemaShardTask) RequiredParameters() []string

RequiredParameters is part of the Task interface.

func (*CopySchemaShardTask) Run

func (t *CopySchemaShardTask) Run(parameters map[string]string) ([]*automationpb.TaskContainer, string, error)

Run is part of the Task interface.

type HorizontalReshardingTask

type HorizontalReshardingTask struct {
}

HorizontalReshardingTask is a cluster operation which allows to increase the number of shards.

func (*HorizontalReshardingTask) OptionalParameters

func (t *HorizontalReshardingTask) OptionalParameters() []string

OptionalParameters is part of the Task interface.

func (*HorizontalReshardingTask) RequiredParameters

func (t *HorizontalReshardingTask) RequiredParameters() []string

RequiredParameters is part of the Task interface.

func (*HorizontalReshardingTask) Run

Run is part of the Task interface.

type IDGenerator

type IDGenerator struct {
	// contains filtered or unexported fields
}

IDGenerator generates unique task and cluster operation IDs.

func (*IDGenerator) GetNextID

func (ig *IDGenerator) GetNextID() string

GetNextID returns an ID which wasn't returned before.

type MigrateServedFromTask

type MigrateServedFromTask struct {
}

MigrateServedFromTask runs vtctl MigrateServedFrom to let vertically split out tables get served from the new destination keyspace.

func (*MigrateServedFromTask) OptionalParameters

func (t *MigrateServedFromTask) OptionalParameters() []string

OptionalParameters is part of the Task interface.

func (*MigrateServedFromTask) RequiredParameters

func (t *MigrateServedFromTask) RequiredParameters() []string

RequiredParameters is part of the Task interface.

func (*MigrateServedFromTask) Run

Run is part of the Task interface.

type MigrateServedTypesTask

type MigrateServedTypesTask struct {
}

MigrateServedTypesTask runs vtctl MigrateServedTypes to migrate a serving type from the source shard to the shards that it replicates to.

func (*MigrateServedTypesTask) OptionalParameters

func (t *MigrateServedTypesTask) OptionalParameters() []string

OptionalParameters is part of the Task interface.

func (*MigrateServedTypesTask) RequiredParameters

func (t *MigrateServedTypesTask) RequiredParameters() []string

RequiredParameters is part of the Task interface.

func (*MigrateServedTypesTask) Run

Run is part of the Task interface.

type RebuildKeyspaceGraphTask

type RebuildKeyspaceGraphTask struct {
}

RebuildKeyspaceGraphTask runs vtctl RebuildKeyspaceGraph to migrate a serving type from the source shard to the shards that it replicates to.

func (*RebuildKeyspaceGraphTask) OptionalParameters

func (t *RebuildKeyspaceGraphTask) OptionalParameters() []string

OptionalParameters is part of the Task interface.

func (*RebuildKeyspaceGraphTask) RequiredParameters

func (t *RebuildKeyspaceGraphTask) RequiredParameters() []string

RequiredParameters is part of the Task interface.

func (*RebuildKeyspaceGraphTask) Run

Run is part of the Task interface.

type Scheduler

type Scheduler struct {
	automationservice.UnimplementedAutomationServer
	// contains filtered or unexported fields
}

Scheduler executes automation tasks and maintains the execution state.

func NewScheduler

func NewScheduler() (*Scheduler, error)

NewScheduler creates a new instance.

func (*Scheduler) Checkpoint

func (s *Scheduler) Checkpoint(clusterOp ClusterOperationInstance)

Checkpoint should be called every time the state of the cluster op changes. It is used to update the copy of the state in activeClusterOperations.

func (*Scheduler) EnqueueClusterOperation

EnqueueClusterOperation can be used to start a new cluster operation.

func (*Scheduler) GetClusterOperationDetails

GetClusterOperationDetails can be used to query the full details of active or finished operations.

func (*Scheduler) Run

func (s *Scheduler) Run()

Run processes queued cluster operations.

func (*Scheduler) ShutdownAndWait

func (s *Scheduler) ShutdownAndWait()

ShutdownAndWait shuts down the scheduler and waits infinitely until all pending cluster operations have finished.

type SplitCloneTask

type SplitCloneTask struct {
}

SplitCloneTask runs SplitClone on a remote vtworker to split an existing shard.

func (*SplitCloneTask) OptionalParameters

func (t *SplitCloneTask) OptionalParameters() []string

OptionalParameters is part of the Task interface.

func (*SplitCloneTask) RequiredParameters

func (t *SplitCloneTask) RequiredParameters() []string

RequiredParameters is part of the Task interface.

func (*SplitCloneTask) Run

func (t *SplitCloneTask) Run(parameters map[string]string) ([]*automationpb.TaskContainer, string, error)

Run is part of the Task interface.

type SplitDiffTask

type SplitDiffTask struct {
}

SplitDiffTask runs SplitDiff on a remote vtworker to compare the old shard against its new split shards.

func (*SplitDiffTask) OptionalParameters

func (t *SplitDiffTask) OptionalParameters() []string

OptionalParameters is part of the Task interface.

func (*SplitDiffTask) RequiredParameters

func (t *SplitDiffTask) RequiredParameters() []string

RequiredParameters is part of the Task interface.

func (*SplitDiffTask) Run

func (t *SplitDiffTask) Run(parameters map[string]string) ([]*automationpb.TaskContainer, string, error)

Run is part of the Task interface.

type Task

type Task interface {
	// Run executes the task using the key/values from parameters.
	// "newTaskContainers" contains new tasks which the task can emit. They'll be inserted in the cluster operation directly after this task. It may be "nil".
	// "output" may be empty. It contains any text which maybe must e.g. to debug the task or show it in the UI.
	Run(parameters map[string]string) (newTaskContainers []*automationpb.TaskContainer, output string, err error)

	// RequiredParameters() returns a list of parameter keys which must be provided as input for Run().
	RequiredParameters() []string

	// OptionalParameters() returns a list of parameter keys which are optional input for Run().
	OptionalParameters() []string
}

Task implementations can be executed by the scheduler.

type VerticalSplitCloneTask

type VerticalSplitCloneTask struct {
}

VerticalSplitCloneTask runs VerticalSplitClone on a remote vtworker to split out tables from an existing keyspace to a different keyspace.

func (*VerticalSplitCloneTask) OptionalParameters

func (t *VerticalSplitCloneTask) OptionalParameters() []string

OptionalParameters is part of the Task interface.

func (*VerticalSplitCloneTask) RequiredParameters

func (t *VerticalSplitCloneTask) RequiredParameters() []string

RequiredParameters is part of the Task interface.

func (*VerticalSplitCloneTask) Run

Run is part of the Task interface.

type VerticalSplitDiffTask

type VerticalSplitDiffTask struct {
}

VerticalSplitDiffTask runs VerticalSplitDiff on a remote vtworker to compare the split out tables against the source keyspace.

func (*VerticalSplitDiffTask) OptionalParameters

func (t *VerticalSplitDiffTask) OptionalParameters() []string

OptionalParameters is part of the Task interface.

func (*VerticalSplitDiffTask) RequiredParameters

func (t *VerticalSplitDiffTask) RequiredParameters() []string

RequiredParameters is part of the Task interface.

func (*VerticalSplitDiffTask) Run

Run is part of the Task interface.

type VerticalSplitTask

type VerticalSplitTask struct {
}

VerticalSplitTask is a cluster operation to split out specific tables of one keyspace to a different keyspace.

func (*VerticalSplitTask) OptionalParameters

func (t *VerticalSplitTask) OptionalParameters() []string

OptionalParameters is part of the Task interface.

func (*VerticalSplitTask) RequiredParameters

func (t *VerticalSplitTask) RequiredParameters() []string

RequiredParameters is part of the Task interface.

func (*VerticalSplitTask) Run

func (t *VerticalSplitTask) Run(parameters map[string]string) ([]*automationpb.TaskContainer, string, error)

Run is part of the Task interface.

type WaitForFilteredReplicationTask

type WaitForFilteredReplicationTask struct {
}

WaitForFilteredReplicationTask runs vtctl WaitForFilteredReplication to block until the destination primary (i.e. the receiving side of the filtered replication) has caught up to max_delay with the source shard.

func (*WaitForFilteredReplicationTask) OptionalParameters

func (t *WaitForFilteredReplicationTask) OptionalParameters() []string

OptionalParameters is part of the Task interface.

func (*WaitForFilteredReplicationTask) RequiredParameters

func (t *WaitForFilteredReplicationTask) RequiredParameters() []string

RequiredParameters is part of the Task interface.

func (*WaitForFilteredReplicationTask) Run

Run is part of the Task interface.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL