cockroach: github.com/cockroachdb/cockroach/pkg/jobs Index | Files | Directories

package jobs

import "github.com/cockroachdb/cockroach/pkg/jobs"

Index

Package Files

executor_impl.go helpers.go job_scheduler.go jobs.go metrics.go progress.go registry.go scheduled_job.go scheduled_job_executor.go update.go

Constants

const InlineExecutorName = "inline"

InlineExecutorName is the name associated with scheduled job executor which runs jobs "inline" -- that is, it doesn't spawn external system.job to do its work.

const PreventAdoptionFile = "DISABLE_STARTING_BACKGROUND_JOBS"

PreventAdoptionFile is the name of the file which, if present in the first on-disk store, will prevent the adoption of background jobs by that node.

Variables

var DefaultAdoptInterval = 30 * time.Second

DefaultAdoptInterval is a reasonable interval at which to poll system.jobs for jobs with expired leases.

DefaultAdoptInterval is mutable for testing. NB: Updates to this value after Registry.Start has been called will not have any effect.

var DefaultCancelInterval = base.DefaultTxnHeartbeatInterval

DefaultCancelInterval is a reasonable interval at which to poll this node for liveness failures and cancel running jobs.

var FakeNodeID = func() *base.NodeIDContainer {
    nodeID := base.NodeIDContainer{}
    nodeID.Reset(1)
    return &nodeID
}()

FakeNodeID is a dummy node ID for use in tests. It always stores 1.

var (

    // LeniencySetting is the amount of time to defer any attempts to
    // reschedule a job.  Visible for testing.
    LeniencySetting = settings.RegisterDurationSetting(
        "jobs.registry.leniency",
        "the amount of time to defer any attempts to reschedule a job",
        defaultLeniencySetting)
)
var MakeChangefeedMetricsHook func(time.Duration) metric.Struct

MakeChangefeedMetricsHook allows for registration of changefeed metrics from ccl code.

var ProdJobSchedulerEnv jobSchedulerEnv = &prodJobSchedulerEnvImpl{}

ProdJobSchedulerEnv is a jobSchedulerEnv implementation suitable for production.

var ProgressUpdateOnly func(context.Context, jobspb.ProgressDetails)

ProgressUpdateOnly is for use with NewChunkProgressLogger to just update job progress fraction (ie. when a custom func with side-effects is not needed).

func DefaultHandleFailedRun Uses

func DefaultHandleFailedRun(schedule *ScheduledJob, jobID int64, err error)

DefaultHandleFailedRun is a default implementation for handling failed run (either system.job failure, or perhaps error processing the schedule itself).

func HasJobNotFoundError Uses

func HasJobNotFoundError(err error) bool

HasJobNotFoundError returns true if the error contains a JobNotFoundError.

func NewRetryJobError Uses

func NewRetryJobError(s string) error

NewRetryJobError creates a new error that, if returned by a Resumer, indicates to the jobs registry that the job should be restarted in the background.

func NotifyJobTermination Uses

func NotifyJobTermination(
    ctx context.Context,
    env jobSchedulerEnv,
    md *JobMetadata,
    scheduleID int64,
    ex sqlutil.InternalExecutor,
    txn *kv.Txn,
) error

NotifyJobTermination is invoked when the job triggered by specified schedule completes

The 'txn' transaction argument is the transaction the job will use to update its state (e.g. status, etc). If any changes need to be made to the scheduled job record, those changes are applied to the same transaction -- that is, they are applied atomically with the job status changes.

func RegisterConstructor Uses

func RegisterConstructor(typ jobspb.Type, fn Constructor)

RegisterConstructor registers a Resumer constructor for a certain job type.

func RegisterScheduledJobExecutorFactory Uses

func RegisterScheduledJobExecutorFactory(name string, factory ScheduledJobExecutorFactory)

RegisterScheduledJobExecutorFactory registers callback for creating ScheduledJobExecutor with the specified name.

func SimplifyInvalidStatusError Uses

func SimplifyInvalidStatusError(err error) error

SimplifyInvalidStatusError unwraps an *InvalidStatusError into an error message suitable for users. Other errors are returned as passed.

func StartJobSchedulerDaemon Uses

func StartJobSchedulerDaemon(
    ctx context.Context,
    stopper *stop.Stopper,
    sv *settings.Values,
    env jobSchedulerEnv,
    db *kv.DB,
    ex sqlutil.InternalExecutor,
)

StartJobSchedulerDaemon starts a daemon responsible for periodically scanning system.scheduled_jobs table to find and executing eligible scheduled jobs.

func TestingSetProgressThresholds Uses

func TestingSetProgressThresholds() func()

TestingSetProgressThresholds overrides batching limits to update more often.

func UnmarshalPayload Uses

func UnmarshalPayload(datum tree.Datum) (*jobspb.Payload, error)

UnmarshalPayload unmarshals and returns the Payload encoded in the input datum, which should be a tree.DBytes.

func UnmarshalProgress Uses

func UnmarshalProgress(datum tree.Datum) (*jobspb.Progress, error)

UnmarshalProgress unmarshals and returns the Progress encoded in the input datum, which should be a tree.DBytes.

type ChunkProgressLogger Uses

type ChunkProgressLogger struct {
    // contains filtered or unexported fields
}

ChunkProgressLogger is a helper for managing the progress state on a job. For a given job, it assumes there are some number of chunks of work to do and tracks the completion progress as chunks are reported as done (via Loop). It then updates the actual job periodically using a ProgressUpdateBatcher.

func NewChunkProgressLogger Uses

func NewChunkProgressLogger(
    j *Job,
    expectedChunks int,
    startFraction float32,
    progressedFn func(context.Context, jobspb.ProgressDetails),
) *ChunkProgressLogger

NewChunkProgressLogger returns a ChunkProgressLogger.

func (*ChunkProgressLogger) Loop Uses

func (jpl *ChunkProgressLogger) Loop(ctx context.Context, chunkCh <-chan struct{}) error

Loop calls chunkFinished for every message received over chunkCh. It exits when chunkCh is closed, when totalChunks messages have been received, or when the context is canceled.

type Constructor Uses

type Constructor func(job *Job, settings *cluster.Settings) Resumer

Constructor creates a resumable job of a certain type. The Resumer is created on the coordinator each time the job is started/resumed, so it can hold state. The Resume method is always ran, and can set state on the Resumer that can be used by the other methods.

type DescriptionUpdateFn Uses

type DescriptionUpdateFn func(ctx context.Context, description string) (string, error)

DescriptionUpdateFn is a callback that computes a job's description given its current one.

type FakeNodeLiveness Uses

type FakeNodeLiveness struct {

    // A non-blocking send is performed over these channels when the corresponding
    // method is called.
    SelfCalledCh          chan struct{}
    GetLivenessesCalledCh chan struct{}
    // contains filtered or unexported fields
}

FakeNodeLiveness allows simulating liveness failures without the full storage.NodeLiveness machinery.

func NewFakeNodeLiveness Uses

func NewFakeNodeLiveness(nodeCount int) *FakeNodeLiveness

NewFakeNodeLiveness initializes a new NodeLiveness with nodeCount live nodes.

func (*FakeNodeLiveness) FakeIncrementEpoch Uses

func (nl *FakeNodeLiveness) FakeIncrementEpoch(id roachpb.NodeID)

FakeIncrementEpoch increments the epoch for the node with the specified ID.

func (*FakeNodeLiveness) FakeSetExpiration Uses

func (nl *FakeNodeLiveness) FakeSetExpiration(id roachpb.NodeID, ts hlc.Timestamp)

FakeSetExpiration sets the expiration time of the liveness for the node with the specified ID to ts.

func (*FakeNodeLiveness) GetLivenesses Uses

func (nl *FakeNodeLiveness) GetLivenesses() (out []kvserverpb.Liveness)

GetLivenesses implements the implicit storage.NodeLiveness interface.

func (*FakeNodeLiveness) IsLive Uses

func (nl *FakeNodeLiveness) IsLive(roachpb.NodeID) (bool, error)

IsLive is unimplemented.

func (*FakeNodeLiveness) ModuleTestingKnobs Uses

func (*FakeNodeLiveness) ModuleTestingKnobs()

ModuleTestingKnobs implements base.ModuleTestingKnobs.

func (*FakeNodeLiveness) Self Uses

func (nl *FakeNodeLiveness) Self() (kvserverpb.Liveness, error)

Self implements the implicit storage.NodeLiveness interface. It uses NodeID as the node ID. On every call, a nonblocking send is performed over nl.ch to allow tests to execute a callback.

type FractionProgressedFn Uses

type FractionProgressedFn func(ctx context.Context, details jobspb.ProgressDetails) float32

FractionProgressedFn is a callback that computes a job's completion fraction given its details. It is safe to modify details in the callback; those modifications will be automatically persisted to the database record.

func FractionUpdater Uses

func FractionUpdater(f float32) FractionProgressedFn

FractionUpdater returns a FractionProgressedFn that returns its argument.

type HighWaterProgressedFn Uses

type HighWaterProgressedFn func(ctx context.Context, txn *kv.Txn, details jobspb.ProgressDetails) (hlc.Timestamp, error)

HighWaterProgressedFn is a callback that computes a job's high-water mark given its details. It is safe to modify details in the callback; those modifications will be automatically persisted to the database record.

type InvalidStatusError Uses

type InvalidStatusError struct {
    // contains filtered or unexported fields
}

InvalidStatusError is the error returned when the desired operation is invalid given the job's current status.

func (*InvalidStatusError) Error Uses

func (e *InvalidStatusError) Error() string

type Job Uses

type Job struct {
    // contains filtered or unexported fields
}

Job manages logging the progress of long-running system processes, like backups and restores, to the system.jobs table.

func (*Job) CheckStatus Uses

func (j *Job) CheckStatus(ctx context.Context) error

CheckStatus verifies the status of the job and returns an error if the job's status isn't Running or Reverting.

func (*Job) CheckTerminalStatus Uses

func (j *Job) CheckTerminalStatus(ctx context.Context) bool

CheckTerminalStatus returns true if the job is in a terminal status.

func (*Job) CurrentStatus Uses

func (j *Job) CurrentStatus(ctx context.Context) (Status, error)

CurrentStatus returns the current job status from the jobs table or error.

func (*Job) Details Uses

func (j *Job) Details() jobspb.Details

Details returns the details from the most recently sent Payload for this Job.

func (*Job) FractionCompleted Uses

func (j *Job) FractionCompleted() float32

FractionCompleted returns completion according to the in-memory job state.

func (*Job) FractionProgressed Uses

func (j *Job) FractionProgressed(ctx context.Context, progressedFn FractionProgressedFn) error

FractionProgressed updates the progress of the tracked job. It sets the job's FractionCompleted field to the value returned by progressedFn and persists progressedFn's modifications to the job's progress details, if any.

Jobs for which progress computations do not depend on their details can use the FractionUpdater helper to construct a ProgressedFn.

func (*Job) HighWaterProgressed Uses

func (j *Job) HighWaterProgressed(ctx context.Context, progressedFn HighWaterProgressedFn) error

HighWaterProgressed updates the progress of the tracked job. It sets the job's HighWater field to the value returned by progressedFn and persists progressedFn's modifications to the job's progress details, if any.

func (*Job) ID Uses

func (j *Job) ID() *int64

ID returns the ID of the job that this Job is currently tracking. This will be nil if Created has not yet been called.

func (*Job) MakeSessionBoundInternalExecutor Uses

func (j *Job) MakeSessionBoundInternalExecutor(
    ctx context.Context, sd *sessiondata.SessionData,
) sqlutil.InternalExecutor

MakeSessionBoundInternalExecutor makes an internal executor, for use in a job resumer, and sets it with the provided session data. See the comment on sessionBoundInternalExecutorFactory for a more detailed explanation of why this exists.

func (*Job) Payload Uses

func (j *Job) Payload() jobspb.Payload

Payload returns the most recently sent Payload for this Job.

func (*Job) Progress Uses

func (j *Job) Progress() jobspb.Progress

Progress returns the most recently sent Progress for this Job.

func (*Job) RunningStatus Uses

func (j *Job) RunningStatus(ctx context.Context, runningStatusFn RunningStatusFn) error

RunningStatus updates the detailed status of a job currently in progress. It sets the job's RunningStatus field to the value returned by runningStatusFn and persists runningStatusFn's modifications to the job's details, if any.

func (*Job) SetDescription Uses

func (j *Job) SetDescription(ctx context.Context, updateFn DescriptionUpdateFn) error

SetDescription updates the description of a created job.

func (*Job) SetDetails Uses

func (j *Job) SetDetails(ctx context.Context, details interface{}) error

SetDetails sets the details field of the currently running tracked job.

func (*Job) SetProgress Uses

func (j *Job) SetProgress(ctx context.Context, details interface{}) error

SetProgress sets the details field of the currently running tracked job.

func (*Job) Update Uses

func (j *Job) Update(ctx context.Context, updateFn UpdateFn) error

Update is used to read the metadata for a job and potentially update it.

The updateFn is called in the context of a transaction and is passed the current metadata for the job. It can choose to update parts of the metadata using the JobUpdater, causing them to be updated within the same transaction.

Sample usage:

err := j.Update(ctx, func(_ *client.Txn, md jobs.JobMetadata, ju *jobs.JobUpdater) error {
  if md.Status != StatusRunning {
    return errors.New("job no longer running")
  }
  md.UpdateStatus(StatusPaused)
  // <modify md.Payload>
  md.UpdatePayload(md.Payload)
}

Note that there are various convenience wrappers (like FractionProgressed) defined in jobs.go.

func (*Job) WithTxn Uses

func (j *Job) WithTxn(txn *kv.Txn) *Job

WithTxn sets the transaction that this Job will use for its next operation. If the transaction is nil, the Job will create a one-off transaction instead. If you use WithTxn, this Job will no longer be threadsafe.

type JobMetadata Uses

type JobMetadata struct {
    ID       int64
    Status   Status
    Payload  *jobspb.Payload
    Progress *jobspb.Progress
}

JobMetadata groups the job metadata values passed to UpdateFn.

func (*JobMetadata) CheckRunningOrReverting Uses

func (md *JobMetadata) CheckRunningOrReverting() error

CheckRunningOrReverting returns an InvalidStatusError if md.Status is not StatusRunning or StatusReverting.

type JobNotFoundError Uses

type JobNotFoundError struct {
    // contains filtered or unexported fields
}

JobNotFoundError is returned from load when the job does not exist.

func (*JobNotFoundError) Error Uses

func (e *JobNotFoundError) Error() string

Error makes JobNotFoundError an error.

type JobUpdater Uses

type JobUpdater struct {
    // contains filtered or unexported fields
}

JobUpdater accumulates changes to job metadata that are to be persisted.

func (*JobUpdater) UpdatePayload Uses

func (ju *JobUpdater) UpdatePayload(payload *jobspb.Payload)

UpdatePayload sets a new Payload (to be persisted).

WARNING: the payload can be large (resulting in a large KV for each version); it shouldn't be updated frequently.

func (*JobUpdater) UpdateProgress Uses

func (ju *JobUpdater) UpdateProgress(progress *jobspb.Progress)

UpdateProgress sets a new Progress (to be persisted).

func (*JobUpdater) UpdateStatus Uses

func (ju *JobUpdater) UpdateStatus(status Status)

UpdateStatus sets a new status (to be persisted).

type Metrics Uses

type Metrics struct {
    Changefeed metric.Struct
}

Metrics are for production monitoring of each job type.

func (*Metrics) InitHooks Uses

func (m *Metrics) InitHooks(histogramWindowInterval time.Duration)

InitHooks initializes the metrics for job monitoring.

func (Metrics) MetricStruct Uses

func (Metrics) MetricStruct()

MetricStruct implements the metric.Struct interface.

type PauseRequester Uses

type PauseRequester interface {
    Resumer

    // OnPauseRequest is called in the transaction that moves a job to PauseRequested.
    // If an error is returned, the pause request will fail. phs is a
    // sql.PlanHookState.
    OnPauseRequest(ctx context.Context, phs interface{}, txn *kv.Txn, details *jobspb.Progress) error
}

PauseRequester is an extension of Resumer which allows job implementers to inject logic during the transaction which moves a job to PauseRequested.

type ProgressUpdateBatcher Uses

type ProgressUpdateBatcher struct {
    // Report is the function called to record progress
    Report func(context.Context, float32) error

    syncutil.Mutex
    // contains filtered or unexported fields
}

ProgressUpdateBatcher is a helper for tracking progress as it is made and calling a progress update function when it has meaningfully advanced (e.g. by more than 5%), while ensuring updates also are not done too often (by default not less than 30s apart).

func (*ProgressUpdateBatcher) Add Uses

func (p *ProgressUpdateBatcher) Add(ctx context.Context, delta float32) error

Add records some additional progress made and checks there has been enough change in the completed progress (and enough time has passed) to report the new progress amount.

func (*ProgressUpdateBatcher) Done Uses

func (p *ProgressUpdateBatcher) Done(ctx context.Context) error

Done allows the batcher to report any meaningful unreported progress, without worrying about update frequency now that it is done.

type Record Uses

type Record struct {
    Description   string
    Statement     string
    Username      string
    DescriptorIDs sqlbase.IDs
    Details       jobspb.Details
    Progress      jobspb.ProgressDetails
    RunningStatus RunningStatus
    // NonCancelable is used to denote when a job cannot be canceled. This field
    // will not be respected in mixed version clusters where some nodes have
    // a version < 20.1, so it can only be used in cases where all nodes having
    // versions >= 20.1 is guaranteed.
    NonCancelable bool
}

Record bundles together the user-managed fields in jobspb.Payload.

type Registry Uses

type Registry struct {
    TestingResumerCreationKnobs map[jobspb.Type]func(Resumer) Resumer
    // contains filtered or unexported fields
}

Registry creates Jobs and manages their leases and cancelation.

Job information is stored in the `system.jobs` table. Each node will poll this table and establish a lease on any claimed job. Registry calculates its own liveness for a node based on the expiration time of the underlying node-liveness lease. This is because we want to allow jobs assigned to temporarily non-live (i.e. saturated) nodes to continue without being canceled.

When a lease has been determined to be stale, a node may attempt to claim the relevant job. Thus, a Registry must occasionally re-validate its own leases to ensure that another node has not stolen the work and cancel the local job if so.

Prior versions of Registry used the node's epoch value to determine whether or not a job should be stolen. The current implementation uses a time-based approach, where a node's last reported expiration timestamp is used to calculate a liveness value for the purpose of job scheduling.

Mixed-version operation between epoch- and time-based nodes works since we still publish epoch information in the leases for time-based nodes. From the perspective of a time-based node, an epoch-based node simply behaves as though its leniency period is 0. Epoch-based nodes will see time-based nodes delay the act of stealing a job.

func MakeRegistry Uses

func MakeRegistry(
    ac log.AmbientContext,
    stopper *stop.Stopper,
    clock *hlc.Clock,
    nl sqlbase.OptionalNodeLiveness,
    db *kv.DB,
    ex sqlutil.InternalExecutor,
    nodeID *base.SQLIDContainer,
    settings *cluster.Settings,
    histogramWindowInterval time.Duration,
    planFn planHookMaker,
    preventAdoptionFile string,
) *Registry

MakeRegistry creates a new Registry. planFn is a wrapper around sql.newInternalPlanner. It returns a sql.PlanHookState, but must be coerced into that in the Resumer functions.

func (*Registry) CancelRequested Uses

func (r *Registry) CancelRequested(ctx context.Context, txn *kv.Txn, id int64) error

CancelRequested marks the job as cancel-requested using the specified txn (may be nil).

func (*Registry) CreateAdoptableJobWithTxn Uses

func (r *Registry) CreateAdoptableJobWithTxn(
    ctx context.Context, record Record, txn *kv.Txn,
) (*Job, error)

CreateAdoptableJobWithTxn creates a job which will be adopted for execution at a later time by some node in the cluster.

func (*Registry) CreateAndStartJob Uses

func (r *Registry) CreateAndStartJob(
    ctx context.Context, resultsCh chan<- tree.Datums, record Record,
) (*Job, <-chan error, error)

CreateAndStartJob creates and asynchronously starts a job from record. An error is returned if the job type has not been registered with RegisterConstructor. The ctx passed to this function is not the context the job will be started with (canceling ctx will not cause the job to cancel).

func (*Registry) CreateJobWithTxn Uses

func (r *Registry) CreateJobWithTxn(ctx context.Context, record Record, txn *kv.Txn) (*Job, error)

CreateJobWithTxn creates a job to be started later with StartJob. It stores the job in the jobs table, marks it running and gives the current node a lease.

func (*Registry) CreateStartableJobWithTxn Uses

func (r *Registry) CreateStartableJobWithTxn(
    ctx context.Context, record Record, txn *kv.Txn, resultsCh chan<- tree.Datums,
) (*StartableJob, error)

CreateStartableJobWithTxn creates a job to be started later, after the creating txn commits. The method uses the passed txn to write the job in the jobs table, marks it pending and gives the current node a lease. It additionally registers the job with the Registry which will prevent the Registry from adopting the job after the transaction commits. The resultsCh will be connected to the output of the job and written to after the returned StartableJob is started.

The returned job is not associated with the user transaction. The intention is that the job will not be modified again in txn. If the transaction is committed, the caller must explicitly Start it. If the transaction is rolled back then the caller must call CleanupOnRollback to unregister the job from the Registry.

func (*Registry) CurrentlyRunningJobs Uses

func (r *Registry) CurrentlyRunningJobs() []int64

CurrentlyRunningJobs returns a slice of the ids of all jobs running on this node.

func (*Registry) Failed Uses

func (r *Registry) Failed(ctx context.Context, txn *kv.Txn, id int64, causingError error) error

Failed marks the job with id as failed.

func (*Registry) LoadJob Uses

func (r *Registry) LoadJob(ctx context.Context, jobID int64) (*Job, error)

LoadJob loads an existing job with the given jobID from the system.jobs table.

func (*Registry) LoadJobWithTxn Uses

func (r *Registry) LoadJobWithTxn(ctx context.Context, jobID int64, txn *kv.Txn) (*Job, error)

LoadJobWithTxn does the same as above, but using the transaction passed in the txn argument. Passing a nil transaction is equivalent to calling LoadJob in that a transaction will be automatically created.

func (*Registry) MetricsStruct Uses

func (r *Registry) MetricsStruct() *Metrics

MetricsStruct returns the metrics for production monitoring of each job type. They're all stored as the `metric.Struct` interface because of dependency cycles.

func (*Registry) NewJob Uses

func (r *Registry) NewJob(record Record) *Job

NewJob creates a new Job.

func (*Registry) PauseRequested Uses

func (r *Registry) PauseRequested(ctx context.Context, txn *kv.Txn, id int64) error

PauseRequested marks the job with id as paused-requested using the specified txn (may be nil).

func (*Registry) Run Uses

func (r *Registry) Run(ctx context.Context, ex sqlutil.InternalExecutor, jobs []int64) error

Run starts previously unstarted jobs from a list of scheduled jobs. Canceling ctx interrupts the waiting but doesn't cancel the jobs.

func (*Registry) SetSessionBoundInternalExecutorFactory Uses

func (r *Registry) SetSessionBoundInternalExecutorFactory(
    factory sqlutil.SessionBoundInternalExecutorFactory,
)

SetSessionBoundInternalExecutorFactory sets the SessionBoundInternalExecutorFactory that will be used by the job registry executor. We expose this separately from the constructor to avoid a circular dependency.

func (*Registry) Start Uses

func (r *Registry) Start(
    ctx context.Context, stopper *stop.Stopper, cancelInterval, adoptInterval time.Duration,
) error

Start polls the current node for liveness failures and cancels all registered jobs if it observes a failure. Otherwise it starts all the main daemons of registry that poll the jobs table and start/cancel/gc jobs.

func (*Registry) Succeeded Uses

func (r *Registry) Succeeded(ctx context.Context, txn *kv.Txn, id int64) error

Succeeded marks the job with id as succeeded.

func (*Registry) TestingNudgeAdoptionQueue Uses

func (r *Registry) TestingNudgeAdoptionQueue()

TestingNudgeAdoptionQueue is used by tests to tell the registry that there is a job to be adopted.

func (*Registry) Unpause Uses

func (r *Registry) Unpause(ctx context.Context, txn *kv.Txn, id int64) error

Unpause changes the paused job with id to running or reverting using the specified txn (may be nil).

type Resumer Uses

type Resumer interface {
    // Resume is called when a job is started or resumed. Sending results on the
    // chan will return them to a user, if a user's session is connected. phs
    // is a sql.PlanHookState.
    Resume(ctx context.Context, phs interface{}, resultsCh chan<- tree.Datums) error

    // OnFailOrCancel is called when a job fails or is cancel-requested.
    //
    // This method will be called when a registry notices the cancel request,
    // which is not guaranteed to run on the node where the job is running. So it
    // cannot assume that any other methods have been called on this Resumer
    // object.
    OnFailOrCancel(ctx context.Context, phs interface{}) error
}

Resumer is a resumable job, and is associated with a Job object. Jobs can be paused or canceled at any time. Jobs should call their CheckStatus() or Progressed() method, which will return an error if the job has been paused or canceled.

Resumers are created through registered Constructor functions.

type RunningStatus Uses

type RunningStatus string

RunningStatus represents the more detailed status of a running job in the system.jobs table.

type RunningStatusFn Uses

type RunningStatusFn func(ctx context.Context, details jobspb.Details) (RunningStatus, error)

RunningStatusFn is a callback that computes a job's running status given its details. It is safe to modify details in the callback; those modifications will be automatically persisted to the database record.

type ScheduledJob Uses

type ScheduledJob struct {
    // contains filtered or unexported fields
}

ScheduledJob is a representation of the scheduled job. This struct can marshal/unmarshal changes made to the underlying system.scheduled_job table.

func NewScheduledJob Uses

func NewScheduledJob(env jobSchedulerEnv) *ScheduledJob

NewScheduledJob creates and initializes ScheduledJob.

func (*ScheduledJob) AddScheduleChangeReason Uses

func (j *ScheduledJob) AddScheduleChangeReason(reasonFmt string, args ...interface{})

AddScheduleChangeReason adds change information to this job. Arguments are interpreted same as printf. If there are too many changes already recorded, trims older changes.

func (*ScheduledJob) Create Uses

func (j *ScheduledJob) Create(ctx context.Context, ex sqlutil.InternalExecutor, txn *kv.Txn) error

Create persists this schedule in the system.scheduled_jobs table. Sets j.scheduleID to the ID of the newly created schedule. Only the values initialized in this schedule are written to the specified transaction. If an error is returned, it is callers responsibility to handle it (e.g. rollback transaction).

func (*ScheduledJob) ExecutionArgs Uses

func (j *ScheduledJob) ExecutionArgs() *jobspb.ExecutionArguments

ExecutionArgs returns ExecutionArgs set for this schedule.

func (*ScheduledJob) ExecutorType Uses

func (j *ScheduledJob) ExecutorType() string

ExecutorType returns executor type for this schedule.

func (*ScheduledJob) HasRecurringSchedule Uses

func (j *ScheduledJob) HasRecurringSchedule() bool

HasRecurringSchedule returns true if this schedule job runs periodically.

func (*ScheduledJob) InitFromDatums Uses

func (j *ScheduledJob) InitFromDatums(datums []tree.Datum, cols []sqlbase.ResultColumn) error

InitFromDatums initializes this ScheduledJob object based on datums and column names.

func (*ScheduledJob) IsPaused Uses

func (j *ScheduledJob) IsPaused() bool

IsPaused returns true if this schedule is paused.

func (*ScheduledJob) NextRun Uses

func (j *ScheduledJob) NextRun() time.Time

NextRun returns the next time this schedule supposed to execute. A sentinel value of time.Time{} indicates this schedule is paused.

func (*ScheduledJob) Pause Uses

func (j *ScheduledJob) Pause(reason string)

Pause pauses this schedule.

func (*ScheduledJob) ScheduleDetails Uses

func (j *ScheduledJob) ScheduleDetails() *jobspb.ScheduleDetails

ScheduleDetails returns schedule configuration information.

func (*ScheduledJob) ScheduleID Uses

func (j *ScheduledJob) ScheduleID() int64

ScheduleID returns schedule ID.

func (*ScheduledJob) ScheduleNextRun Uses

func (j *ScheduledJob) ScheduleNextRun() error

ScheduleNextRun updates next run based on job schedule.

func (*ScheduledJob) SetExecutionDetails Uses

func (j *ScheduledJob) SetExecutionDetails(executor string, args jobspb.ExecutionArguments)

SetExecutionDetails sets execution specific fields for this schedule.

func (*ScheduledJob) SetNextRun Uses

func (j *ScheduledJob) SetNextRun(t time.Time)

SetNextRun updates next run time for this schedule.

func (*ScheduledJob) SetSchedule Uses

func (j *ScheduledJob) SetSchedule(scheduleExpr string) error

SetSchedule updates periodicity of this schedule, and updates this schedules next run time.

func (*ScheduledJob) SetScheduleDetails Uses

func (j *ScheduledJob) SetScheduleDetails(details jobspb.ScheduleDetails)

SetScheduleDetails updates schedule configuration.

func (*ScheduledJob) SetScheduleName Uses

func (j *ScheduledJob) SetScheduleName(name string)

SetScheduleName updates schedule name.

func (*ScheduledJob) Unpause Uses

func (j *ScheduledJob) Unpause(reason string) error

Unpause resumes running this schedule.

func (*ScheduledJob) Update Uses

func (j *ScheduledJob) Update(ctx context.Context, ex sqlutil.InternalExecutor, txn *kv.Txn) error

Update saves changes made to this schedule. If an error is returned, it is callers responsibility to handle it (e.g. rollback transaction).

type ScheduledJobExecutor Uses

type ScheduledJobExecutor interface {
    // Executes scheduled job;  Implementation may use provided transaction.
    // Modifications to the ScheduledJob object will be persisted.
    ExecuteJob(ctx context.Context, schedule *ScheduledJob, txn *kv.Txn) error

    // Notifies that the system.job started by the ScheduledJob completed.
    // Implementation may use provided transaction to perform any additional mutations.
    // Modifications to the ScheduledJob object will be persisted.
    NotifyJobTermination(ctx context.Context, md *JobMetadata, schedule *ScheduledJob, txn *kv.Txn) error
}

ScheduledJobExecutor is an interface describing execution of the scheduled job.

func NewScheduledJobExecutor Uses

func NewScheduledJobExecutor(
    name string, ex sqlutil.InternalExecutor,
) (ScheduledJobExecutor, error)

NewScheduledJobExecutor creates new ScheduledJobExecutor.

type ScheduledJobExecutorFactory Uses

type ScheduledJobExecutorFactory = func(ex sqlutil.InternalExecutor) (ScheduledJobExecutor, error)

ScheduledJobExecutorFactory is a callback to create a ScheduledJobExecutor.

type StartableJob Uses

type StartableJob struct {
    *Job
    // contains filtered or unexported fields
}

StartableJob is a job created with a transaction to be started later. See Registry.CreateStartableJob

func (*StartableJob) Cancel Uses

func (sj *StartableJob) Cancel(ctx context.Context) error

Cancel will mark the job as canceled and release its resources in the Registry.

func (*StartableJob) CleanupOnRollback Uses

func (sj *StartableJob) CleanupOnRollback(ctx context.Context) error

CleanupOnRollback will unregister the job in the case that the creating transaction has been rolled back.

func (*StartableJob) Run Uses

func (sj *StartableJob) Run(ctx context.Context) error

Run will resume the job and wait for it to finish or the context to be canceled. The transaction used to create the StartableJob must be committed. Results will be copied to the channel used to create this StartableJob even if job is canceled.

func (*StartableJob) Start Uses

func (sj *StartableJob) Start(ctx context.Context) (errCh <-chan error, err error)

Start will resume the job. The transaction used to create the StartableJob must be committed. If a non-nil error is returned, the job was not started and nothing will be send on errCh. Clients must not start jobs more than once.

type Status Uses

type Status string

Status represents the status of a job in the system.jobs table.

const (
    // StatusPending is for jobs that have been created but on which work has
    // not yet started.
    StatusPending Status = "pending"
    // StatusRunning is for jobs that are currently in progress.
    StatusRunning Status = "running"
    // StatusPaused is for jobs that are not currently performing work, but have
    // saved their state and can be resumed by the user later.
    StatusPaused Status = "paused"
    // StatusFailed is for jobs that failed.
    StatusFailed Status = "failed"
    // StatusReverting is for jobs that failed or were canceled and their changes are being
    // being reverted.
    StatusReverting Status = "reverting"
    // StatusSucceeded is for jobs that have successfully completed.
    StatusSucceeded Status = "succeeded"
    // StatusCanceled is for jobs that were explicitly canceled by the user and
    // cannot be resumed.
    StatusCanceled Status = "canceled"
    // StatusCancelRequested is for jobs that were requested to be canceled by
    // the user but may be still running Resume. The node that is running the job
    // will change it to StatusReverting the next time it runs maybeAdoptJobs.
    StatusCancelRequested Status = "cancel-requested"
    // StatusPauseRequested is for jobs that were requested to be paused by the
    // user but may be still resuming or reverting. The node that is running the
    // job will change its state to StatusPaused the next time it runs
    // maybeAdoptJobs and will stop running it.
    StatusPauseRequested Status = "pause-requested"
)

func (Status) Terminal Uses

func (s Status) Terminal() bool

Terminal returns whether this status represents a "terminal" state: a state after which the job should never be updated again.

type UpdateFn Uses

type UpdateFn func(txn *kv.Txn, md JobMetadata, ju *JobUpdater) error

UpdateFn is the callback passed to Job.Update. It is called from the context of a transaction and is passed the current metadata for the job. The callback can modify metadata using the JobUpdater and the changes will be persisted within the same transaction.

The function is free to modify contents of JobMetadata in place (but the changes will be ignored unless JobUpdater is used).

Directories

PathSynopsis
jobspb
jobsprotectedts

Package jobs imports 38 packages (graph) and is imported by 47 packages. Updated 2020-07-10. Refresh now. Tools for package owners.