| Path | Synopsis |
|---|---|
| cmd/beamctl | beamctl is a command line client for the Apache Beam portability services. |
| cmd/beamctl/cmd | Package cmd contains the commands for beamctl. |
| cmd/specialize | specialize is a low-level tool to generate type-specialized code. |
| cmd/starcgen | starcgen is a tool to generate specialized type assertion shims to be used in Apache Beam Go SDK pipelines instead of the default reflection shim. |
| cmd/symtab | Package verifies that functions sym2addr and addr2sym work correctly. |
| container | |
| examples/contains | |
| examples/debugging_wordcount | debugging_wordcount is an example that verifies word counts in Shakespeare and includes Beam best practices. |
| examples/forest | forest is an example that shows that pipeline construction is normal Go code -- the pipeline "forest" is created recursively and uses a global variable -- and that a pipeline may contain non-connected parts. |
| examples/grades | |
| examples/minimal_wordcount | minimal_wordcount is an example that counts words in Shakespeare. |
| examples/multiout | multiout is a wordcount variation that uses a multi-outout DoFn and writes 2 output files. |
| examples/pingpong | |
| examples/readavro | readavro is a simple Avro read/write Example This example uses a 500 Byte sample avro file [twitter.avro] download here: https://s3-eu-west-1.amazonaws.com/daidokoro-dev/apache/twitter.avro |
| examples/streaming_wordcap | streaming_wordcap is a toy streaming pipeline that uses PubSub. |
| examples/stringsplit | An example of using a Splittable DoFn in the Go SDK with a portable runner. |
| examples/windowed_wordcount | windowed_wordcount counts words in text, and can run over either unbounded or bounded input collections. |
| examples/wordcount | wordcount is an example that counts words in Shakespeare and includes Beam best practices. |
| examples/yatzy | yatzy is an implementation of https://en.wikipedia.org/wiki/Yatzy that shows that pipeline construction is normal Go code. |
| pkg/beam | Package beam is an implementation of the Apache Beam (https://beam.apache.org) programming model in Go. |
| pkg/beam/artifact | Package artifact contains utilities for staging and retrieving artifacts. |
| pkg/beam/artifact/gcsproxy | Package gcsproxy contains artifact staging and retrieval servers backed by GCS. |
| pkg/beam/core/funcx | Package funcx contains functions and types used to perform type analysis of Beam functions. |
| pkg/beam/core/graph | Package graph is the internal representation of the Beam execution plan. |
| pkg/beam/core/graph/coder | Package coder contains coder representation and utilities. |
| pkg/beam/core/graph/mtime | Package mtime contains a millisecond representation of time. |
| pkg/beam/core/graph/window | Package window contains window representation, windowing strategies and utilities. |
| pkg/beam/core/metrics | Package metrics implements the Beam metrics API, described at http://s.apache.org/beam-metrics-api |
| pkg/beam/core/runtime | Package runtime contains runtime hooks and utilities for pipeline options and type registration. |
| pkg/beam/core/runtime/coderx | Package coderx contains coders for primitive types that aren't included in the beam model. |
| pkg/beam/core/runtime/exec | Package exec contains runtime plan representation and execution. |
| pkg/beam/core/runtime/exec/optimized | Package optimized contains type-specialized shims for faster execution. |
| pkg/beam/core/runtime/genx | Package genx is a convenience package to better support the code generator. |
| pkg/beam/core/runtime/graphx | Package graphx provides facilities to help with the serialization of pipelines into a serializable graph structure suitable for the worker. |
| pkg/beam/core/runtime/graphx/schema | Package schema contains utility functions for relating Go types and Beam Schemas. |
| pkg/beam/core/runtime/graphx/v1 | Package v1 is a generated protocol buffer package. |
| pkg/beam/core/runtime/harness | Package harness implements the SDK side of the Beam FnAPI. |
| pkg/beam/core/runtime/harness/init | Package init contains the harness initialization code defined by the FnAPI. |
| pkg/beam/core/runtime/harness/session | Package session is a generated protocol buffer package. |
| pkg/beam/core/runtime/metricsx | |
| pkg/beam/core/runtime/pipelinex | Package pipelinex contains utilities for manipulating Beam proto pipelines. |
| pkg/beam/core/runtime/xlangx | |
| pkg/beam/core/sdf | Package contains interfaces used specifically for splittable DoFns. |
| pkg/beam/core/typex | Package typex contains full type representation for PCollections and DoFns, and utilities for type checking. |
| pkg/beam/core/util/dot | Package dot produces DOT graphs from Beam graph representations. |
| pkg/beam/core/util/hooks | Package hooks allows runners to tailor execution of the worker harness. |
| pkg/beam/core/util/ioutilx | Package ioutilx contains additional io utilities. |
| pkg/beam/core/util/jsonx | Package jsonx contains utilities for working with JSON encoded data. |
| pkg/beam/core/util/protox | Package protox contains utilities for working with protobufs. |
| pkg/beam/core/util/reflectx | Package reflectx contains a set of reflection utilities and well-known types. |
| pkg/beam/core/util/stringx | Package stringx contains utilities for working with strings. |
| pkg/beam/core/util/symtab | Package symtab allows reading low-level symbol information from the symbol table. |
| pkg/beam/internal/errors | Package errors contains functionality for creating and wrapping errors with improved formatting compared to the standard Go error functionality. |
| pkg/beam/io/avroio | Package avroio contains transforms for reading and writing avro files. |
| pkg/beam/io/bigqueryio | Package bigqueryio provides transformations and utilities to interact with Google BigQuery. |
| pkg/beam/io/databaseio | Package databaseio provides transformations and utilities to interact with generic database database/sql API. |
| pkg/beam/io/filesystem | Package filesystem contains an extensible file system abstraction. |
| pkg/beam/io/filesystem/gcs | Package gcs contains a Google Cloud Storage (GCS) implementation of the Beam file system. |
| pkg/beam/io/filesystem/local | Package local contains a local file implementation of the Beam file system. |
| pkg/beam/io/filesystem/memfs | Package memfs contains a in-memory Beam filesystem. |
| pkg/beam/io/pubsubio | Package pubsubio provides access to PubSub on Dataflow streaming. |
| pkg/beam/io/pubsubio/v1 | Package v1 is a generated protocol buffer package. |
| pkg/beam/io/rtrackers/offsetrange | Package offsetrange defines a restriction and restriction tracker for offset ranges. |
| pkg/beam/io/synthetic | Package synthetic contains transforms for creating synthetic pipelines. |
| pkg/beam/io/textio | Package textio contains transforms for reading and writing text files. |
| pkg/beam/log | Package log contains a re-targetable context-aware logging system. |
| pkg/beam/model | Package model contains the portable Beam model contracts. |
| pkg/beam/model/fnexecution_v1 | |
| pkg/beam/model/jobmanagement_v1 | |
| pkg/beam/model/pipeline_v1 | |
| pkg/beam/options/gcpopts | Package gcpopts contains shared options for Google Cloud Platform. |
| pkg/beam/options/jobopts | Package jobopts contains shared options for job submission. |
| pkg/beam/provision | Package provision contains utilities for obtaining runtime provision, information -- such as pipeline options. |
| pkg/beam/runners/dataflow | Package dataflow contains the Dataflow runner for submitting pipelines to Google Cloud Dataflow. |
| pkg/beam/runners/dataflow/dataflowlib | Package dataflowlib translates a Beam pipeline model to the Dataflow API job model, for submission to Google Cloud Dataflow. |
| pkg/beam/runners/direct | Package direct contains the direct runner for running single-bundle pipelines in the current process. |
| pkg/beam/runners/dot | Package dot is a Beam runner that "runs" a pipeline by producing a DOT graph of the execution plan. |
| pkg/beam/runners/flink | Package flink contains the Flink runner. |
| pkg/beam/runners/session | |
| pkg/beam/runners/spark | Package spark contains the Spark runner. |
| pkg/beam/runners/universal | Package universal contains a general-purpose runner that can submit jobs to any portable Beam runner. |
| pkg/beam/runners/universal/extworker | Package extworker provides an external worker service and related utilities. |
| pkg/beam/runners/universal/runnerlib | Package runnerlib contains utilities for submitting Go pipelines to a Beam model runner. |
| pkg/beam/runners/vet | Package vet is a Beam runner that "runs" a pipeline by producing generated code to avoid symbol table lookups and reflection in pipeline execution. |
| pkg/beam/runners/vet/testpipeline | Package testpipeline exports small test pipelines for testing the vet runner. |
| pkg/beam/testing/passert | Package passert contains verification transformations for testing pipelines. |
| pkg/beam/testing/ptest | Package ptest contains utilities for pipeline unit testing. |
| pkg/beam/transforms/filter | Package filter contains transformations for removing pipeline elements based on various conditions. |
| pkg/beam/transforms/stats | Package stats contains transforms for statistical processing. |
| pkg/beam/transforms/top | Package top contains transformations for finding the smallest (or largest) N elements based on arbitrary orderings. |
| pkg/beam/util/errorx | Package errorx contains utilities for handling errors. |
| pkg/beam/util/execx | Package execx contains wrappers and utilities for the exec package. |
| pkg/beam/util/gcsx | Package gcsx contains utilities for working with Google Cloud Storage (GCS). |
| pkg/beam/util/grpcx | Package grpcx contains utilities for working with gRPC. |
| pkg/beam/util/pubsubx | Package pubsubx contains utilities for working with Google PubSub. |
| pkg/beam/util/shimx | Package shimx specifies the templates for generating type assertion shims for Apache Beam Go SDK pipelines. |
| pkg/beam/util/starcgenx | Package starcgenx is a Static Analysis Type Assertion shim and Registration Code Generator which provides an extractor to extract types from a package, in order to generate approprate shimsr a package so code can be generated for it. |
| pkg/beam/x/beamx | Package beamx is a convenience package for beam. |
| pkg/beam/x/debug | Package debug contains pipeline components that may help in debugging pipeline issues. |
| pkg/beam/x/hooks/perf | Package perf is to add performance measuring hooks to a runner, such as cpu, heap, or trace profiles. |
| test/integration | The integration driver provides a suite of tests to run against a registered runner. |
| test/integration/primitives | |
| test/integration/synthetic | Package synthetic contains pipelines for testing synthetic steps and sources. |
| test/integration/wordcount | Package wordcount contains transforms for wordcount. |
| test/regression | Package regression contains pipeline regression tests. |
Updated 2021-01-19. Refresh now. Tools for package owners.