beam: github.com/apache/beam/sdks/go/pkg/beam/artifact/gcsproxy Index | Files

package gcsproxy

import "github.com/apache/beam/sdks/go/pkg/beam/artifact/gcsproxy"

Package gcsproxy contains artifact staging and retrieval servers backed by GCS.

Index

Package Files

retrieval.go staging.go

func ReadProxyManifest Uses

func ReadProxyManifest(ctx context.Context, object string) (*jobpb.ProxyManifest, error)

ReadProxyManifest reads and parses the proxy manifest from GCS.

type RetrievalServer Uses

type RetrievalServer struct {
    // contains filtered or unexported fields
}

RetrievalServer is a artifact retrieval server backed by Google Cloud Storage (GCS). It serves a single manifest and ignores the worker id. The server performs no caching or pre-fetching.

func NewRetrievalServer Uses

func NewRetrievalServer(md *jobpb.ProxyManifest) (*RetrievalServer, error)

NewRetrievalServer creates a artifact retrieval server for the given manifest. It requires that the locations are in GCS.

func (*RetrievalServer) GetArtifact Uses

func (s *RetrievalServer) GetArtifact(req *jobpb.LegacyGetArtifactRequest, stream jobpb.LegacyArtifactRetrievalService_GetArtifactServer) error

GetArtifact returns a given artifact.

func (*RetrievalServer) GetManifest Uses

func (s *RetrievalServer) GetManifest(ctx context.Context, req *jobpb.GetManifestRequest) (*jobpb.GetManifestResponse, error)

GetManifest returns the manifest for all artifacts.

type StagingServer Uses

type StagingServer struct {
    // contains filtered or unexported fields
}

StagingServer is a artifact staging server backed by Google Cloud Storage (GCS). It commits a single manifest and ignores the staging id.

func NewStagingServer Uses

func NewStagingServer(manifest string) (*StagingServer, error)

NewStagingServer creates a artifact staging server for the given manifest. It requires that the manifest is in GCS and will stage the supplied artifacts next to it.

func (*StagingServer) CommitManifest Uses

func (s *StagingServer) CommitManifest(ctx context.Context, req *jobpb.CommitManifestRequest) (*jobpb.CommitManifestResponse, error)

CommitManifest commits the given artifact manifest to GCS.

func (*StagingServer) PutArtifact Uses

func (s *StagingServer) PutArtifact(ps jobpb.LegacyArtifactStagingService_PutArtifactServer) error

PutArtifact stores the given artifact in GCS.

Package gcsproxy imports 13 packages (graph). Updated 2020-04-30. Refresh now. Tools for package owners.