README
¶
parwork 
Parallel work processing package
Description
This package allows work to processed in parallel using a fork-join pattern. The implementation relies on goroutines, channels and wait groups.
The implementation allows the configuration of the processor by providing the degree of parallelism which defines how many goroutines will process work from the queues in parallel. By creating only a small number of goroutines, which defaults to the number of cores on the system, we avoid context switching instead of allowing a high number of goroutines to fight for processor resources.
This is the reason why this package makes more sense when used with work items that are CPU bound and do not switch context like waiting for IO.
In order to use the package the user has only to provide the implementation of the following:
Work interface
type Work interface {
Do()
GetError() error
Result() interface{}
}
The work interface defines a method Do() which contains all the processing logic of the work item. The GetError() error method can be used to flag the work item as failed and return a error. The Result() interface{} defines a method which returns the result of the work. Due to the lack of generics the data return has to be cast from interface{} to the actual result type in order to be usable in the WorkCollector.
WorkGenerator function
type WorkGenerator func() Work
The WorkGenerator function allows the user to provide a implementation that returns on each call a work item to be processed. If the generator returns nil the generation of work has finished.
WorkCollector function
type WorkCollector func(Work)
The WorkCollector function takes as a argument a completed Work item. It can check for a failure by calling the GetError or the Result method of the Work item and handle it appropriately.
Example
For a example implementation please take a look in the examples folder of the repository. The example implements a brute force method of trying to find the MD5 hash of a string. This is just a example implementation to demonstrate the usage of the package. And it should not be misused to break secrets.
Documentation
¶
Overview ¶
Package parwork - Parallel work processing package
Description ¶
This package allows work to processed in parallel using a fork-join pattern. The implementation relies on goroutines, channels and wait groups.
The implementation allows the configuration of the processor by providing the degree of parallelism which defines how many goroutines will process work from the queues in parallel. By creating only a small number of goroutines, which defaults to the number of cores on the system, we avoid context switching instead of allowing a high number of goroutines to fight for processor resources.
This is the reason why this package makes more sense when used with work items that are CPU bound and do not switch context like waiting for IO.
In order to use the package the user has only to provide the implementation of the following:
Work interface
type Work interface {
Do()
GetError() error
Result() interface{}
}
The work interface defines a method "Do()" which contains all the processing logic of the work item. The "GetError()" method can be used to flag the work item as failed and return a error. The "Result()" defines a method which returns the result of the work. Due to the lack of generics the data return has to be cast from "interface{}" to the actual result type in order to be usable in the WorkCollector.
WorkGenerator function
type WorkGenerator func() Work
The WorkGenerator function allows the user to provide a implementation that returns on each call a work item to be processed. If the generator returns "nil" the generation of work has finished.
WorkCollector function
type WorkCollector func(Work)
The WorkCollector function takes as a argument a completed Work item. It can check for a failure by calling the "GetError()" or the "Result()" method of the Work item and handle it appropriately.
Examples ¶
For a example implementation please take a look in the examples folder of the repository. The example implements a brute force method of trying to find the MD5 hash of a string. This is just a example implementation to demonstrate the usage of the package. And it should not be misused to break secrets.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Option ¶
Option defines a option for the processor
func Collector ¶
func Collector(reporter WorkCollector) Option
Collector defines a processor option for the work collector
type Processor ¶
type Processor struct {
// contains filtered or unexported fields
}
Processor handles the generation, distribution and reporting or work
func New ¶
func New(g WorkGenerator, options ...Option) (*Processor, error)
New returns a new work processor with default worker, queue length and reporter. Optional definitions are available through the processor options variadic arguments. In case of a Workers: Number of CPU Queue: Number of CPU * 100 Reporter: output to stdout
type Work ¶
type Work interface {
Do()
Err() error
Result() interface{}
}
Work define the interface that each work item has to implement in order to be processed. This implementation follows the command pattern.
type WorkCollector ¶
type WorkCollector func(Work)
WorkCollector defines a function that handles the collection of a completed work.
type WorkGenerator ¶
type WorkGenerator func() Work
WorkGenerator defines a function that generates work. Every time the function is called it will return work or nil which signals the end of the work generation.
Source Files
Directories