mapreduce

package

v0.0.0-...-39572d5 Latest Latest Go to latest Published: Dec 13, 2016 License: MIT Imports: 23 Imported by: 0

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

github.com/mateuszdyminski/gomr

Links

Open Source Insights

Documentation ¶

Index ¶

Constants
type KeyValue
type Logger
- func NewLogger(prefix string, debug bool) *Logger
type MapReduce
type Master
- func NewMaster(host string, httpPort int, rpcPort int, workDir string, debug bool) (*Master, error)
type Wrk
- func NewWorker(id, host string, rpcPort, httpPort int, workDir string, debug bool) (*Wrk, error)

Constants ¶

View Source

const (
	// MasterServiceName name of the master service in Consul.
	MasterServiceName = "master"

	// WorkerServiceName name of the worker service in Consul.
	WorkerServiceName = "worker"

	// GrpcConsulTag tag set for the worker to notify the clients that grpc API should be used.
	GrpcConsulTag = "grpc"

	// Separator separates key and value in intermediate files.
	Separator = ":::"

	// RetriesLimit number of retries of.
	RetriesLimit = 3

	// MapPhase name of the map phase.
	MapPhase = "map"

	// ReducePhase name of the reduce phase.
	ReducePhase = "reduce"
)

Variables ¶

This section is empty.

Functions ¶

This section is empty.

Types ¶

type Logger ¶

type Logger struct {
	// contains filtered or unexported fields
}

Logger hold bacic info about logger configuration.

func (*Logger) Debug ¶

func (l *Logger) Debug(format string, v ...interface{})

Debug logs the formatted message with level DEBUG - only if debug flag is set to true.

func (*Logger) Error ¶

func (l *Logger) Error(format string, v ...interface{})

Error logs the formatted message with level ERROR.

func (*Logger) Fatal ¶

func (l *Logger) Fatal(format string, v ...interface{})

Fatal logs the formatted message with level FATAL.

func (*Logger) Info ¶

func (l *Logger) Info(format string, v ...interface{})

Info logs the formatted message with level INFO.

type MapReduce ¶

type MapReduce interface {
	Map(string, string) []KeyValue
	Reduce(string, []string) string
}

MapReduce it's interface for any job in gomr framework. Every gomr job should implement this interface.

type Master ¶

type Master struct {
	// contains filtered or unexported fields
}

Master holds information about the gomr Master service.

func NewMaster ¶

func NewMaster(host string, httpPort int, rpcPort int, workDir string, debug bool) (*Master, error)

NewMaster returns new instance of gomr Master service.

func (*Master) Start ¶

func (m *Master) Start() error

Start starts the gomr master service - creates the health check server, register master service in consul and tries to connect to available warokers. Func waits until it gets signal{Kill, Interrupt} or Stop func is called.

func (*Master) Stop ¶

func (m *Master) Stop() error

Stop stops the master service.

func (*Master) Submit ¶

func (m *Master) Submit(job *service.MrJob, stream service.Master_SubmitServer) error

Submit exposes the submit for the gRPC gomr client. It submits the job, prepares inputs for map phase, coordinates the map and reduce phase and merge the outputs from the reduce phase. It blocks until the job is done.

type Wrk ¶

type Wrk struct {
	// contains filtered or unexported fields
}

Wrk holds information about the gomr Worker service.

func NewWorker ¶

func NewWorker(id, host string, rpcPort, httpPort int, workDir string, debug bool) (*Wrk, error)

NewWorker returns new instance of gomr worker.

func (*Wrk) Map ¶

func (w *Wrk) Map(job *service.MrJob, server service.MapReduce_MapServer) error

Map starts map phase and returns the status of the tasks as long as its ongoing.

func (*Wrk) Reduce ¶

func (w *Wrk) Reduce(job *service.MrJob, server service.MapReduce_ReduceServer) error

Reduce starts reduce phase and returns the status of the tasks as long as its ongoing.

func (*Wrk) Start ¶

func (w *Wrk) Start() error

Start starts the gomr worker - creates the health check server and register workers service in consul. Func waits until it gets signal{Kill, Interrupt} or Stop func is called.

func (*Wrk) Stop ¶

func (w *Wrk) Stop() error

Stop stops the worker.

Source Files ¶

View all Source files

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL