beam: github.com/apache/beam

Directories

PathSynopsis
sdks/go/cmd/beamctlbeamctl is a command line client for the Apache Beam portability services.
sdks/go/cmd/beamctl/cmdPackage cmd contains the commands for beamctl.
sdks/go/cmd/specializespecialize is a low-level tool to generate type-specialized code.
sdks/go/cmd/starcgenstarcgen is a tool to generate specialized type assertion shims to be used in Apache Beam Go SDK pipelines instead of the default reflection shim.
sdks/go/cmd/symtabPackage verifies that functions sym2addr and addr2sym work correctly.
sdks/go/container
sdks/go/examples/contains
sdks/go/examples/debugging_wordcountdebugging_wordcount is an example that verifies word counts in Shakespeare and includes Beam best practices.
sdks/go/examples/forestforest is an example that shows that pipeline construction is normal Go code -- the pipeline "forest" is created recursively and uses a global variable -- and that a pipeline may contain non-connected parts.
sdks/go/examples/grades
sdks/go/examples/minimal_wordcountminimal_wordcount is an example that counts words in Shakespeare.
sdks/go/examples/multioutmultiout is a wordcount variation that uses a multi-outout DoFn and writes 2 output files.
sdks/go/examples/pingpong
sdks/go/examples/readavroreadavro is a simple Avro read/write Example This example uses a 500 Byte sample avro file [twitter.avro] download here: https://s3-eu-west-1.amazonaws.com/daidokoro-dev/apache/twitter.avro
sdks/go/examples/streaming_wordcapstreaming_wordcap is a toy streaming pipeline that uses PubSub.
sdks/go/examples/stringsplitAn example of using a Splittable DoFn in the Go SDK with a portable runner.
sdks/go/examples/windowed_wordcountwindowed_wordcount counts words in text, and can run over either unbounded or bounded input collections.
sdks/go/examples/wordcountwordcount is an example that counts words in Shakespeare and includes Beam best practices.
sdks/go/examples/yatzyyatzy is an implementation of https://en.wikipedia.org/wiki/Yatzy that shows that pipeline construction is normal Go code.
sdks/go/pkg/beamPackage beam is an implementation of the Apache Beam (https://beam.apache.org) programming model in Go.
sdks/go/pkg/beam/artifactPackage artifact contains utilities for staging and retrieving artifacts.
sdks/go/pkg/beam/artifact/gcsproxyPackage gcsproxy contains artifact staging and retrieval servers backed by GCS.
sdks/go/pkg/beam/core/funcxPackage funcx contains functions and types used to perform type analysis of Beam functions.
sdks/go/pkg/beam/core/graphPackage graph is the internal representation of the Beam execution plan.
sdks/go/pkg/beam/core/graph/coderPackage coder contains coder representation and utilities.
sdks/go/pkg/beam/core/graph/mtimePackage mtime contains a millisecond representation of time.
sdks/go/pkg/beam/core/graph/windowPackage window contains window representation, windowing strategies and utilities.
sdks/go/pkg/beam/core/metricsPackage metrics implements the Beam metrics API, described at http://s.apache.org/beam-metrics-api
sdks/go/pkg/beam/core/runtimePackage runtime contains runtime hooks and utilities for pipeline options and type registration.
sdks/go/pkg/beam/core/runtime/coderxPackage coderx contains coders for primitive types that aren't included in the beam model.
sdks/go/pkg/beam/core/runtime/execPackage exec contains runtime plan representation and execution.
sdks/go/pkg/beam/core/runtime/exec/optimizedPackage optimized contains type-specialized shims for faster execution.
sdks/go/pkg/beam/core/runtime/graphxPackage graphx provides facilities to help with the serialization of pipelines into a serializable graph structure suitable for the worker.
sdks/go/pkg/beam/core/runtime/graphx/schemaPackage schema contains utility functions for relating Go types and Beam Schemas.
sdks/go/pkg/beam/core/runtime/graphx/v1Package v1 is a generated protocol buffer package.
sdks/go/pkg/beam/core/runtime/harnessPackage harness implements the SDK side of the Beam FnAPI.
sdks/go/pkg/beam/core/runtime/harness/initPackage init contains the harness initialization code defined by the FnAPI.
sdks/go/pkg/beam/core/runtime/harness/sessionPackage session is a generated protocol buffer package.
sdks/go/pkg/beam/core/runtime/pipelinexPackage pipelinex contains utilities for manipulating Beam proto pipelines.
sdks/go/pkg/beam/core/runtime/xlangx
sdks/go/pkg/beam/core/sdfPackage contains interfaces used specifically for splittable DoFns.
sdks/go/pkg/beam/core/typexPackage typex contains full type representation for PCollections and DoFns, and utilities for type checking.
sdks/go/pkg/beam/core/util/dotPackage dot produces DOT graphs from Beam graph representations.
sdks/go/pkg/beam/core/util/hooksPackage hooks allows runners to tailor execution of the worker harness.
sdks/go/pkg/beam/core/util/ioutilxPackage ioutilx contains additional io utilities.
sdks/go/pkg/beam/core/util/protoxPackage protox contains utilities for working with protobufs.
sdks/go/pkg/beam/core/util/reflectxPackage reflectx contains a set of reflection utilities and well-known types.
sdks/go/pkg/beam/core/util/stringxPackage stringx contains utilities for working with strings.
sdks/go/pkg/beam/core/util/symtabPackage symtab allows reading low-level symbol information from the symbol table.
sdks/go/pkg/beam/internal/errorsPackage errors contains functionality for creating and wrapping errors with improved formatting compared to the standard Go error functionality.
sdks/go/pkg/beam/io/avroioPackage avroio contains transforms for reading and writing avro files.
sdks/go/pkg/beam/io/bigqueryioPackage bigqueryio provides transformations and utilities to interact with Google BigQuery.
sdks/go/pkg/beam/io/databaseioPackage databaseio provides transformations and utilities to interact with generic database database/sql API.
sdks/go/pkg/beam/io/filesystemPackage filesystem contains an extensible file system abstraction.
sdks/go/pkg/beam/io/filesystem/gcsPackage gcs contains a Google Cloud Storage (GCS) implementation of the Beam file system.
sdks/go/pkg/beam/io/filesystem/localPackage local contains a local file implementation of the Beam file system.
sdks/go/pkg/beam/io/filesystem/memfsPackage memfs contains a in-memory Beam filesystem.
sdks/go/pkg/beam/io/pubsubioPackage pubsubio provides access to PubSub on Dataflow streaming.
sdks/go/pkg/beam/io/pubsubio/v1Package v1 is a generated protocol buffer package.
sdks/go/pkg/beam/io/rtrackers/offsetrangePackage offsetrange defines a restriction and restriction tracker for offset ranges.
sdks/go/pkg/beam/io/syntheticPackage synthetic contains transforms for creating synthetic pipelines.
sdks/go/pkg/beam/io/textioPackage textio contains transforms for reading and writing text files.
sdks/go/pkg/beam/logPackage log contains a re-targetable context-aware logging system.
sdks/go/pkg/beam/modelPackage model contains the portable Beam model contracts.
sdks/go/pkg/beam/model/fnexecution_v1
sdks/go/pkg/beam/model/jobmanagement_v1
sdks/go/pkg/beam/model/pipeline_v1
sdks/go/pkg/beam/options/gcpoptsPackage gcpopts contains shared options for Google Cloud Platform.
sdks/go/pkg/beam/options/joboptsPackage jobopts contains shared options for job submission.
sdks/go/pkg/beam/provisionPackage provision contains utilities for obtaining runtime provision, information -- such as pipeline options.
sdks/go/pkg/beam/runners/dataflowPackage dataflow contains the Dataflow runner for submitting pipelines to Google Cloud Dataflow.
sdks/go/pkg/beam/runners/dataflow/dataflowlibPackage dataflowlib translates a Beam pipeline model to the Dataflow API job model, for submission to Google Cloud Dataflow.
sdks/go/pkg/beam/runners/directPackage direct contains the direct runner for running single-bundle pipelines in the current process.
sdks/go/pkg/beam/runners/dotPackage dot is a Beam runner that "runs" a pipeline by producing a DOT graph of the execution plan.
sdks/go/pkg/beam/runners/flinkPackage flink contains the Flink runner.
sdks/go/pkg/beam/runners/session
sdks/go/pkg/beam/runners/sparkPackage spark contains the Spark runner.
sdks/go/pkg/beam/runners/universalPackage universal contains a general-purpose runner that can submit jobs to any portable Beam runner.
sdks/go/pkg/beam/runners/universal/extworkerPackage extworker provides an external worker service and related utilities.
sdks/go/pkg/beam/runners/universal/runnerlibPackage runnerlib contains utilities for submitting Go pipelines to a Beam model runner.
sdks/go/pkg/beam/runners/vetPackage vet is a Beam runner that "runs" a pipeline by producing generated code to avoid symbol table lookups and reflection in pipeline execution.
sdks/go/pkg/beam/runners/vet/testpipelinePackage testpipeline exports small test pipelines for testing the vet runner.
sdks/go/pkg/beam/testing/passertPackage passert contains verification transformations for testing pipelines.
sdks/go/pkg/beam/testing/ptestPackage ptest contains utilities for pipeline unit testing.
sdks/go/pkg/beam/transforms/filterPackage filter contains transformations for removing pipeline elements based on various conditions.
sdks/go/pkg/beam/transforms/statsPackage stats contains transforms for statistical processing.
sdks/go/pkg/beam/transforms/topPackage top contains transformations for finding the smallest (or largest) N elements based on arbitrary orderings.
sdks/go/pkg/beam/util/errorxPackage errorx contains utilities for handling errors.
sdks/go/pkg/beam/util/execxPackage execx contains wrappers and utilities for the exec package.
sdks/go/pkg/beam/util/gcsxPackage gcsx contains utilities for working with Google Cloud Storage (GCS).
sdks/go/pkg/beam/util/grpcxPackage grpcx contains utilities for working with gRPC.
sdks/go/pkg/beam/util/pubsubxPackage pubsubx contains utilities for working with Google PubSub.
sdks/go/pkg/beam/util/shimxPackage shimx specifies the templates for generating type assertion shims for Apache Beam Go SDK pipelines.
sdks/go/pkg/beam/util/starcgenxPackage starcgenx is a Static Analysis Type Assertion shim and Registration Code Generator which provides an extractor to extract types from a package, in order to generate approprate shimsr a package so code can be generated for it.
sdks/go/pkg/beam/x/beamxPackage beamx is a convenience package for beam.
sdks/go/pkg/beam/x/debugPackage debug contains pipeline components that may help in debugging pipeline issues.
sdks/go/pkg/beam/x/hooks/perfPackage perf is to add performance measuring hooks to a runner, such as cpu, heap, or trace profiles.
sdks/go/test/integrationThe integration driver provides a suite of tests to run against a registered runner.
sdks/go/test/integration/primitives
sdks/go/test/integration/syntheticPackage synthetic contains pipelines for testing synthetic steps and sources.
sdks/go/test/integration/wordcountPackage wordcount contains transforms for wordcount.
sdks/go/test/regressionPackage regression contains pipeline regression tests.

Updated 2020-09-14. Refresh now. Tools for package owners.