worker

package
v0.0.0-...-31bcc22 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Feb 20, 2018 License: MIT Imports: 11 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Init

func Init()

Init adds log filename hook

func KVGroup

func KVGroup(input io.Reader, g chan *Group, sep string)

KVGroup groups continous items by keys in a pre-sorted reader

Types

type Group

type Group struct {
	Key  string
	Vals chan []byte
}

Group holds individual key/value-chan

type JobWorker

type JobWorker interface {
	Map(id int, input string, utils *Utilities) (outputs map[int]string, err error)
	Reduce(id int, inputs []string, utils *Utilities) (output string, err error)
}

JobWorker is the user implimentation that performs the map/reduce tasks

type JobWorkerMerge

type JobWorkerMerge interface {
	JobWorker
	ReduceMerge(inputs []string, args, secrets map[string]string, utils *Utilities) (output string, err error)
}

JobWorkerMerge is a JobWorker that can also do a ReduceMerge on results of Reduce to output single result TODO: Not actually being used - Future

type Runner

type Runner struct {
	// contains filtered or unexported fields
}

Runner manages the lifecycle of a worker

func NewRunner

func NewRunner() (*Runner, error)

NewRunner initializes things from enviornment and returns a NewRunner

func (*Runner) Run

func (r *Runner) Run(w JobWorker) error

Run runs a worker

type Utilities

type Utilities struct {
	// contains filtered or unexported fields
}

Utilities provide common useful methods that map/reduce functions may make use off.

func NewUtilities

func NewUtilities(bucket *s3.Bucket, prefix string) *Utilities

NewUtilities creates new helper object

func (*Utilities) GetS3Object

func (utils *Utilities) GetS3Object(src string) (io.ReadCloser, error)

GetS3Object gets object from s3, errors if src is not fully qualified uri matching our bucket

func (*Utilities) UploadFilename

func (utils *Utilities) UploadFilename(key, src string) (string, error)

UploadFilename uploads file src into key in bucket

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL