beam: github.com/apache/beam/sdks/go/pkg/beam/artifact Index | Files | Directories

package artifact

import "github.com/apache/beam/sdks/go/pkg/beam/artifact"

Package artifact contains utilities for staging and retrieving artifacts.

Index

Package Files

materialize.go stage.go

func Commit Uses

func Commit(ctx context.Context, client pb.ArtifactStagingServiceClient, artifacts []*pb.ArtifactMetadata, st string) (string, error)

Commit commits a manifest with the given staged artifacts. It returns the staging token, if successful.

func Materialize Uses

func Materialize(ctx context.Context, endpoint string, rt string, dest string) ([]*pb.ArtifactMetadata, error)

Materialize is a convenience helper for ensuring that all artifacts are present and uncorrupted. It interprets each artifact name as a relative path under the dest directory. It does not retrieve valid artifacts already present.

func MultiRetrieve Uses

func MultiRetrieve(ctx context.Context, client pb.ArtifactRetrievalServiceClient, cpus int, list []*pb.ArtifactMetadata, rt string, dest string) error

MultiRetrieve retrieves multiple artifacts concurrently, using at most 'cpus' goroutines. It retries each artifact a few times. Convenience wrapper.

func MultiStage Uses

func MultiStage(ctx context.Context, client pb.ArtifactStagingServiceClient, cpus int, list []KeyedFile, st string) ([]*pb.ArtifactMetadata, error)

MultiStage stages a set of local files with the given keys. It returns the full artifact metadate. It retries each artifact a few times. Convenience wrapper.

func Retrieve Uses

func Retrieve(ctx context.Context, client pb.ArtifactRetrievalServiceClient, a *pb.ArtifactMetadata, rt string, dest string) error

Retrieve checks whether the given artifact is already successfully retrieved. If not, it retrieves into the dest directory. It overwrites any previous retrieval attempt and may leave a corrupt/partial local file on failure.

func Stage Uses

func Stage(ctx context.Context, client pb.ArtifactStagingServiceClient, key, filename, st string) (*pb.ArtifactMetadata, error)

Stage stages a local file as an artifact with the given key. It computes the SHA256 and returns the full artifact metadata.

func StageDir Uses

func StageDir(ctx context.Context, client pb.ArtifactStagingServiceClient, src string, st string) ([]*pb.ArtifactMetadata, error)

StageDir stages a local directory with relative path keys. Convenience wrapper.

type KeyedFile Uses

type KeyedFile struct {
    Key, Filename string
}

KeyedFile is a key and filename pair.

Directories

PathSynopsis
gcsproxyPackage gcsproxy contains artifact staging and retrieval servers backed by GCS.

Package artifact imports 17 packages (graph) and is imported by 3 packages. Updated 2019-05-05. Refresh now. Tools for package owners.