Documentation ¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type JobWorker ¶
type JobWorker interface { Map(id int, input string, utils *Utilities) (outputs map[int]string, err error) Reduce(id int, inputs []string, utils *Utilities) (output string, err error) }
JobWorker is the user implimentation that performs the map/reduce tasks
type JobWorkerMerge ¶
type JobWorkerMerge interface { JobWorker ReduceMerge(inputs []string, args, secrets map[string]string, utils *Utilities) (output string, err error) }
JobWorkerMerge is a JobWorker that can also do a ReduceMerge on results of Reduce to output single result TODO: Not actually being used - Future
type Runner ¶
type Runner struct {
// contains filtered or unexported fields
}
Runner manages the lifecycle of a worker
type Utilities ¶
type Utilities struct {
// contains filtered or unexported fields
}
Utilities provide common useful methods that map/reduce functions may make use off.
func NewUtilities ¶
NewUtilities creates new helper object
func (*Utilities) GetS3Object ¶
func (utils *Utilities) GetS3Object(src string) (io.ReadCloser, error)
GetS3Object gets object from s3, errors if src is not fully qualified uri matching our bucket
Click to show internal directories.
Click to hide internal directories.