kubemr: github.com/turbobytes/kubemr/pkg/worker Index | Files

package worker

import "github.com/turbobytes/kubemr/pkg/worker"

Index

Package Files

init.go kvgroup.go runner.go utils.go worker.go

func Init Uses

func Init()

Init adds log filename hook

func KVGroup Uses

func KVGroup(input io.Reader, g chan *Group, sep string)

KVGroup groups continous items by keys in a pre-sorted reader

type Group Uses

type Group struct {
    Key  string
    Vals chan []byte
}

Group holds individual key/value-chan

type JobWorker Uses

type JobWorker interface {
    Map(id int, input string, utils *Utilities) (outputs map[int]string, err error)
    Reduce(id int, inputs []string, utils *Utilities) (output string, err error)
}

JobWorker is the user implimentation that performs the map/reduce tasks

type JobWorkerMerge Uses

type JobWorkerMerge interface {
    JobWorker
    ReduceMerge(inputs []string, args, secrets map[string]string, utils *Utilities) (output string, err error)
}

JobWorkerMerge is a JobWorker that can also do a ReduceMerge on results of Reduce to output single result TODO: Not actually being used - Future

type Runner Uses

type Runner struct {
    // contains filtered or unexported fields
}

Runner manages the lifecycle of a worker

func NewRunner Uses

func NewRunner() (*Runner, error)

NewRunner initializes things from enviornment and returns a NewRunner

func (*Runner) Run Uses

func (r *Runner) Run(w JobWorker) error

Run runs a worker

type Utilities Uses

type Utilities struct {
    // contains filtered or unexported fields
}

Utilities provide common useful methods that map/reduce functions may make use off.

func NewUtilities Uses

func NewUtilities(bucket *s3.Bucket, prefix string) *Utilities

NewUtilities creates new helper object

func (*Utilities) GetS3Object Uses

func (utils *Utilities) GetS3Object(src string) (io.ReadCloser, error)

GetS3Object gets object from s3, errors if src is not fully qualified uri matching our bucket

func (*Utilities) UploadFilename Uses

func (utils *Utilities) UploadFilename(key, src string) (string, error)

UploadFilename uploads file src into key in bucket

Package worker imports 11 packages (graph). Updated 2017-08-18. Refresh now. Tools for package owners.