beam: github.com/apache/beam/sdks/go/pkg/beam/artifact Index | Files | Directories

package artifact

import "github.com/apache/beam/sdks/go/pkg/beam/artifact"

Package artifact contains utilities for staging and retrieving artifacts.

Index

Package Files

materialize.go stage.go

Constants

const (
    URNStagingTo      = "beam:artifact:role:staging_to:v1"
    NoArtifactsStaged = "__no_artifacts_staged__"
)

TODO(lostluck): 2018/05/28 Extract these from their enum descriptors in the pipeline_v1 proto

func Commit Uses

func Commit(ctx context.Context, client jobpb.LegacyArtifactStagingServiceClient, artifacts []*jobpb.ArtifactMetadata, st string) (string, error)

Commit commits a manifest with the given staged artifacts. It returns the staging token, if successful.

func LegacyMultiRetrieve Uses

func LegacyMultiRetrieve(ctx context.Context, client jobpb.LegacyArtifactRetrievalServiceClient, cpus int, list []*jobpb.ArtifactMetadata, rt string, dest string) error

LegacyMultiRetrieve is exported for testing.

func Materialize Uses

func Materialize(ctx context.Context, endpoint string, dependencies []*pipepb.ArtifactInformation, rt string, dest string) ([]*jobpb.ArtifactMetadata, error)

Materialize is a convenience helper for ensuring that all artifacts are present and uncorrupted. It interprets each artifact name as a relative path under the dest directory. It does not retrieve valid artifacts already present. TODO(BEAM-9577): Return a mapping of filename to dependency, rather than []*jobpb.ArtifactMetadata. TODO(BEAM-9577): Leverage richness of roles rather than magic names to understand artifacts.

func MultiRetrieve Uses

func MultiRetrieve(ctx context.Context, cpus int, list []retrievable, dest string) error

MultiRetrieve retrieves multiple artifacts concurrently, using at most 'cpus' goroutines. It retries each artifact a few times. Convenience wrapper.

func MultiStage Uses

func MultiStage(ctx context.Context, client jobpb.LegacyArtifactStagingServiceClient, cpus int, list []KeyedFile, st string) ([]*jobpb.ArtifactMetadata, error)

MultiStage stages a set of local files with the given keys. It returns the full artifact metadate. It retries each artifact a few times. Convenience wrapper.

func Retrieve Uses

func Retrieve(ctx context.Context, client jobpb.LegacyArtifactRetrievalServiceClient, a *jobpb.ArtifactMetadata, rt string, dest string) error

Retrieve checks whether the given artifact is already successfully retrieved. If not, it retrieves into the dest directory. It overwrites any previous retrieval attempt and may leave a corrupt/partial local file on failure.

func Stage Uses

func Stage(ctx context.Context, client jobpb.LegacyArtifactStagingServiceClient, key, filename, st string) (*jobpb.ArtifactMetadata, error)

Stage stages a local file as an artifact with the given key. It computes the SHA256 and returns the full artifact metadata.

func StageDir Uses

func StageDir(ctx context.Context, client jobpb.LegacyArtifactStagingServiceClient, src string, st string) ([]*jobpb.ArtifactMetadata, error)

StageDir stages a local directory with relative path keys. Convenience wrapper.

type KeyedFile Uses

type KeyedFile struct {
    Key, Filename string
}

KeyedFile is a key and filename pair.

Directories

PathSynopsis
gcsproxyPackage gcsproxy contains artifact staging and retrieval servers backed by GCS.

Package artifact imports 19 packages (graph) and is imported by 4 packages. Updated 2020-05-27. Refresh now. Tools for package owners.