morebeam: github.com/bramp/morebeam Index | Examples | Files | Directories

package morebeam

import "github.com/bramp/morebeam"

Package morebeam provides additional functions useful for building Apache Beam pipelines.

Code:

package main

import (
    "context"
    "flag"
    "reflect"

    "bramp.net/morebeam"
    "bramp.net/morebeam/csvio"
    "github.com/apache/beam/sdks/go/pkg/beam"
    "github.com/apache/beam/sdks/go/pkg/beam/x/beamx"
    "github.com/apache/beam/sdks/go/pkg/beam/x/debug"
)

// Painting represents a single record in the csv file.
type Painting struct {
    Artist  string `csv:"artist"`
    Title   string `csv:"title"`
    Year    int    `csv:"year"`
    NotUsed string `csv:"-"` // Ignored field
}

func main() {
    flag.Parse()
    beam.Init()

    p, s := beam.NewPipelineWithRoot()

    // Read the CSV file as a PCollection<Painting>.
    paintings := csvio.Read(s, "paintings.csv", reflect.TypeOf(Painting{}))

    // Reshuffle the CSV output to improve parallelism.
    paintings = morebeam.Reshuffle(s, paintings)

    // Return a new PCollection<KV<string, Painting>> where the key is the artist.
    paintingsByArtist := morebeam.AddKey(s, func(painting Painting) string {
        return painting.Artist
    }, paintings)

    debug.Print(s, paintingsByArtist)

    beamx.Run(context.Background(), p)

}

Index

Examples

Package Files

addkey.go morebeam.go reshuffle.go

func AddKey Uses

func AddKey(s beam.Scope, keyFn interface{}, col beam.PCollection) beam.PCollection

AddKey takes a PCollection<V> and returns a PCollection<KV<K, V>> where the key is calculated from keyFn.

func Join Uses

func Join(elem ...string) string

Join is similar to path.Join but safe to use on URLs or filepaths.

func Reshuffle Uses

func Reshuffle(s beam.Scope, col beam.PCollection) beam.PCollection

Reshuffle takes a PCollection<A> and shuffles the data to help increase parallelism.

Directories

PathSynopsis
csvioPackage csvio contains Apache Beam transforms for reading CSV files with https://github.com/gocarina/gocsv.

Package morebeam imports 6 packages (graph). Updated 2018-12-16. Refresh now. Tools for package owners.