beam: Index | Files

package pipelinex

import ""

Package pipelinex contains utilities for manipulating Beam proto pipelines. The utilities generally uses shallow copies and do not mutate their inputs.


Package Files

clone.go replace.go util.go

func Bounded Uses

func Bounded(p *pipepb.Pipeline) bool

Bounded returns true iff all PCollections are bounded.

func ContainerImages Uses

func ContainerImages(p *pipepb.Pipeline) []string

ContainerImages returns the set of container images used in the given pipeline.

func Normalize Uses

func Normalize(p *pipepb.Pipeline) (*pipepb.Pipeline, error)

Normalize recomputes derivative information in the pipeline, such as roots and input/output for composite transforms. It also ensures that unique names are so and topologically sorts each subtransform list.

func ShallowCloneFunctionSpec Uses

func ShallowCloneFunctionSpec(p *pipepb.FunctionSpec) *pipepb.FunctionSpec

ShallowCloneFunctionSpec makes a shallow copy of the given FunctionSpec.

func ShallowClonePTransform Uses

func ShallowClonePTransform(t *pipepb.PTransform) *pipepb.PTransform

ShallowClonePTransform makes a shallow copy of the given PTransform.

func ShallowCloneParDoPayload Uses

func ShallowCloneParDoPayload(p *pipepb.ParDoPayload) *pipepb.ParDoPayload

ShallowCloneParDoPayload makes a shallow copy of the given ParDoPayload.

func ShallowCloneSideInput Uses

func ShallowCloneSideInput(p *pipepb.SideInput) *pipepb.SideInput

ShallowCloneSideInput makes a shallow copy of the given SideInput.

func TopologicalSort Uses

func TopologicalSort(xforms map[string]*pipepb.PTransform, ids []string) []string

TopologicalSort returns a topologically sorted list of the given ids, generally from the same scope/composite. Assumes acyclic graph.

func TrimCoders Uses

func TrimCoders(coders map[string]*pipepb.Coder, ids ...string) map[string]*pipepb.Coder

TrimCoders returns the transitive closure of the given coders ids.

func Update Uses

func Update(p *pipepb.Pipeline, values *pipepb.Components) (*pipepb.Pipeline, error)

Update merges a pipeline with the given components, which may add, replace or delete its values. It returns the merged pipeline. The input is not modified.

Package pipelinex imports 6 packages (graph) and is imported by 3 packages. Updated 2020-06-23. Refresh now. Tools for package owners.