Documentation
¶
Overview ¶
common_rpc.go defined all the parameters used in RPC between master and workers
Index ¶
- Constants
- func RemoveContents(dir string) error
- func RunMaster(task *Task, hostname string)
- func RunSequential(task *Task)
- func RunWorker(task *Task, hostname string, masterHostname string, nOps int)
- type KeyValue
- type MapFunc
- type Master
- type Operation
- type ReduceFunc
- type RegisterArgs
- type RegisterReply
- type RemoteWorker
- type RunArgs
- type ShuffleFunc
- type Task
- type Worker
Constants ¶
const ( REDUCE_PATH = "reduce/" RESULT_PATH = "result/" OPEN_FILE_MAX_RETRY = 3 )
const ( IDLE_WORKER_BUFFER = 100 RETRY_OPERATION_BUFFER = 100 )
const ( WORKER_IDLE workerStatus = "idle" WORKER_RUNNING workerStatus = "running" )
Variables ¶
This section is empty.
Functions ¶
func RemoveContents ¶
func RunMaster ¶
RunMaster will start a master node on the map reduce operations. In the distributed model, a Master should serve multiple workers and distribute the operations to be executed in order to complete the task.
- task: the Task object that contains the mapreduce operation.
- hostname: the tcp/ip address on which it will listen for connections.
func RunSequential ¶
func RunSequential(task *Task)
RunSequential will ensure that map and reduce function runs in a single-core linearly. The Task is passed from the calling package and should contains the definitions for all the required functions and parameters. Notice that this implementation will store data locally. In the distributed version of mapreduce it's common to store the data in the same worker that computed it and just pass a reference to reduce jobs so they can go grab it.
Types ¶
type Master ¶
type Master struct {
// contains filtered or unexported fields
}
func (*Master) Register ¶
func (master *Master) Register(args *RegisterArgs, reply *RegisterReply) error
RPC - Register Procedure that will be called by workers to register within this master.
type ReduceFunc ¶
type RegisterArgs ¶
type RegisterArgs struct {
WorkerHostname string
}
type RegisterReply ¶
type RemoteWorker ¶
type RemoteWorker struct {
// contains filtered or unexported fields
}
type ShuffleFunc ¶
type Task ¶
type Task struct {
// MapReduce functions
Map MapFunc
Shuffle ShuffleFunc
Reduce ReduceFunc
// Jobs
NumReduceJobs int
NumMapFiles int
// Channels for data
InputChan chan []byte
OutputChan chan []KeyValue
// Channels for filepaths
InputFilePathChan chan string
OutputFilePathChan chan string
}
Task is the exposed struct of the Framework that the calling code should initialize with the specific implementation of the operation.
Source Files