beam: github.com/apache/beam/sdks/go/pkg/beam/runners/universal/runnerlib Index | Files

package runnerlib

import "github.com/apache/beam/sdks/go/pkg/beam/runners/universal/runnerlib"

Package runnerlib contains utilities for submitting Go pipelines to a Beam model runner.

Index

Package Files

compile.go execute.go job.go stage.go

func BuildTempWorkerBinary Uses

func BuildTempWorkerBinary(ctx context.Context) (string, error)

BuildTempWorkerBinary creates a local worker binary in the tmp directory for linux/amd64. Caller responsible for deleting the binary.

func BuildWorkerBinary Uses

func BuildWorkerBinary(ctx context.Context, filename string) error

BuildWorkerBinary creates a local worker binary for linux/amd64. It finds the filename by examining the call stack. We want the user entry (*), for example:

/Users/herohde/go/src/github.com/apache/beam/sdks/go/pkg/beam/runners/beamexec/main.go (skip: 2)

* /Users/herohde/go/src/github.com/apache/beam/sdks/go/examples/wordcount/wordcount.go (skip: 3)

/usr/local/go/src/runtime/proc.go (skip: 4)      // not always present
/usr/local/go/src/runtime/asm_amd64.s (skip: 4 or 5)

func Execute Uses

func Execute(ctx context.Context, p *pipepb.Pipeline, endpoint string, opt *JobOptions, async bool) (string, error)

Execute executes a pipeline on the universal runner serving the given endpoint. Convenience function.

func IsWorkerCompatibleBinary Uses

func IsWorkerCompatibleBinary() (string, bool)

IsWorkerCompatibleBinary returns the path to itself and true if running a linux-amd64 binary that can directly be used as a worker binary.

func Prepare Uses

func Prepare(ctx context.Context, client jobpb.JobServiceClient, p *pipepb.Pipeline, opt *JobOptions) (id, endpoint, stagingToken string, err error)

Prepare prepares a job to the given job service. It returns the preparation id artifact staging endpoint, and staging token if successful.

func Stage Uses

func Stage(ctx context.Context, id, endpoint, binary, st string) (retrievalToken string, err error)

Stage stages the worker binary and any additional files to the given artifact staging endpoint. It returns the retrieval token if successful.

func StageFile Uses

func StageFile(filename string, stream jobpb.ArtifactStagingService_ReverseArtifactRetrievalServiceClient) error

func StageViaLegacyApi Uses

func StageViaLegacyApi(ctx context.Context, cc *grpc.ClientConn, binary, st string) (retrievalToken string, err error)

func StageViaPortableApi Uses

func StageViaPortableApi(ctx context.Context, cc *grpc.ClientConn, binary, st string) error

func Submit Uses

func Submit(ctx context.Context, client jobpb.JobServiceClient, id, token string) (string, error)

Submit submits a job to the given job service. It returns a jobID, if successful.

func WaitForCompletion Uses

func WaitForCompletion(ctx context.Context, client jobpb.JobServiceClient, jobID string) error

WaitForCompletion monitors the given job until completion. It logs any messages and state changes received.

type JobOptions Uses

type JobOptions struct {
    // Name is the job name.
    Name string
    // Experiments are additional experiments.
    Experiments []string

    // Worker is the worker binary override.
    Worker string

    // RetainDocker is an option to pass to the runner.
    RetainDocker bool
}

JobOptions capture the various options for submitting jobs to universal runners.

Package runnerlib imports 23 packages (graph) and is imported by 2 packages. Updated 2020-09-20. Refresh now. Tools for package owners.