Documentation
¶
Overview ¶
Package beamio provides Beam transformations for common IO patterns.
Index ¶
- func ComputeShards(s beam.Scope, kv beam.PCollection, opts stats.Opts) beam.PCollection
- func EncodeKeyValues(s beam.Scope, tables ...beam.PCollection) beam.PCollection
- func ReadEntries(ctx context.Context, s beam.Scope, fileOrDir string) (beam.PCollection, error)
- func WriteLevelDB(s beam.Scope, path string, opts stats.Opts, tables ...beam.PCollection)
- type KeyValue
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func ComputeShards ¶ added in v0.0.50
func ComputeShards(s beam.Scope, kv beam.PCollection, opts stats.Opts) beam.PCollection
ComputeShards assigns shards to KeyValues. Input is a PCollection of beamio.KeyValue, output is (int, beamio.KeyValue).
func EncodeKeyValues ¶ added in v0.0.30
func EncodeKeyValues(s beam.Scope, tables ...beam.PCollection) beam.PCollection
EncodeKeyValues encodes each PCollection of KVs into encoded KeyValues and flattens all entries into a single PCollection.
func ReadEntries ¶
ReadEntries reads a set of *spb.Entry messages into a PCollection from the given file, or files stored in a directory. The file can be part of any filesystem registered with the beam/io/filesystem package and can either be a delimited protobuf stream or a Riegeli file.
func WriteLevelDB ¶ added in v0.0.28
WriteLevelDB writes a set of PCollections containing KVs to a new LevelDB at the given path. Each KV is serialized and stored as a single LevelDB key-value entry according to their enclosing PCollection's beam.Coder. Each table may have different KV types. Keys must be unique across all PCollections.
Source Files