README
¶
Generic iterator library
This package standardizes iterator implementation using generic types. It is
designed for ease of use in a for loop, and includes several commonly used
iterator methods, such as converting a slice to an iterator and back, a
sequential map, etc. Of special note is ParallelMap, see below.
This package requires Go 1.18 or higher, as it uses generics.
Installation
go get github.com/stockparfait/iterator
Example usage
package main
import (
"fmt"
"github.com/stockparfait/iterator"
)
func main() {
it := iterator.FromSlice([]int{1, 2, 3, 4})
for v, ok := it.Next(); ok; v, ok = it.Next() {
fmt.Println(v)
}
}
ParallelMap - parallel execution with a specified number of workers
ParallelMap is similar to Map, except that it runs multiple function calls
f(In) ("jobs") in parallel on a given number of workers (unlimited when 0).
The order of the results is undefined, unless the number of workers is 1.
Canceling the supplied context immediately stops queuing new jobs, but the jobs that already started will finish and their results will be returned. Therefore, it is important to flush the iterator after canceling the context to release all the resources.
No job is started by this method itself. Jobs begin to run on the first Next()
call on the result iterator.
Note, that Next() is not go routine safe, and it doesn't require go routine
safety of the input iterator.
For simpler situations when the input sequence can be created ahead of time and
the results collected in a slice before further processing, use ParallelMapSlice:
package main
import (
"context"
"fmt"
"github.com/stockparfait/iterator"
)
func main() {
ctx := context.Background()
it := iterator.FromSlice([]int{5, 10, 15})
f := func(i int) int { return i + 1 }
pm := iterator.ParallelMap(ctx, 2, it, f)
defer pm.Close()
for r, ok := pm.Next(); ok; r, ok = pm.Next() {
fmt.Printf("result = %d\n", r)
// Early exit is safe, resources will be released.
}
}
Documentation
¶
Overview ¶
Package iterator implements generic iterator interface, including parallel processing.
Index ¶
- func Flush[T any](it Iterator[T])
- func ParallelMapSlice[In, Out any](ctx context.Context, workers int, in []In, f func(In) Out) []Out
- func Reduce[In, Out any](it Iterator[In], zero Out, f func(In, Out) Out) Out
- func TestSerialize(ctx context.Context) context.Context
- func ToSlice[T any](it Iterator[T]) []T
- type Iterator
- func Batch[T any](it Iterator[T], n int) Iterator[[]T]
- func Chain[T any](it Iterator[Iterator[T]]) Iterator[T]
- func FromSlice[T any](s []T) Iterator[T]
- func Map[In, Out any](it Iterator[In], f func(In) Out) Iterator[Out]
- func Repeat[T any](value T, n int) Iterator[T]
- func Unbatch[T any](it Iterator[[]T]) Iterator[T]
- type IteratorCloser
- func BatchCloser[T any](it IteratorCloser[T], n int) IteratorCloser[[]T]
- func BatchReduce[In, Out any](ctx context.Context, workers int, it Iterator[In], batchSize int, zero Out, ...) IteratorCloser[Out]
- func ChainCloser[T any](it IteratorCloser[IteratorCloser[T]]) IteratorCloser[T]
- func MapCloser[In, Out any](it IteratorCloser[In], f func(In) Out) IteratorCloser[Out]
- func OrderedParallelMap[In, Out any](ctx context.Context, workers, bufferSize int, it Iterator[In], f func(In) Out) IteratorCloser[Out]
- func ParallelMap[In, Out any](ctx context.Context, workers int, it Iterator[In], f func(In) Out) IteratorCloser[Out]
- func UnbatchCloser[T any](it IteratorCloser[[]T]) IteratorCloser[T]
- func WithClose[T any](it Iterator[T], close func()) IteratorCloser[T]
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Flush ¶ added in v0.1.3
Flush the remaining elements from the iterator. This can be useful for a custom IteratorCloser when the iterator needs to flush remaining elements to release resources.
func ParallelMapSlice ¶
ParallelMapSlice maps an input slice into the output slice using ParallelMap.
func Reduce ¶ added in v0.1.1
Reduce Iterator[In] into a single value Out by applying res[n+1] = f(x[n], res[n]), starting with res[0] = zero.
func TestSerialize ¶
TestSerialize forces the number of workers in ParallelMap to be 1, thereby running jobs serially and in the strict FIFO order. This helps make tests deterministic.
Types ¶
type Iterator ¶
Iterator is a generic interface for generating sequences of values of type T.
When the second Next()'s result is true it returns the next value. When it's false, the iterator is considered "empty", and subsequent calls to Next() are expected to return false.
Example use of an iterator "it":
for v, ok := it.Next(); ok; v, ok = it.Next() {
// use v
}
func Batch ¶ added in v0.1.2
Batch the input iterator values into n-sized slices and return them as a new iterator. Panics if n < 1.
type IteratorCloser ¶ added in v0.1.5
IteratorCloser is an iterator with an additional Close() method which empties the iterator (a subsequent Next() call returns ok=false) and releases all associated resources, such as active go-routines.
Example use of a closing iterator "it":
defer it.Close()
for v; ok := it.Next(); ok; v, ok = it.Next() {
// use v; can safely exit early
}
func BatchCloser ¶ added in v0.2.1
func BatchCloser[T any](it IteratorCloser[T], n int) IteratorCloser[[]T]
BatchCloser is like Batch but propagates Close() to the output iterator.
func BatchReduce ¶ added in v0.1.2
func BatchReduce[In, Out any](ctx context.Context, workers int, it Iterator[In], batchSize int, zero Out, f func(In, Out) Out) IteratorCloser[Out]
BatchReduce reduces the input iterator in parallel batches, returning an iterator of the results that can be further reduced by sequential Reduce, or another layer of BatchReduce. Panics if batchSize < 1.
Same as ParallelMap, canceling context stops queuing new jobs, but the iterator needs to Flush to release resources. See ParallelMap for an example.
func ChainCloser ¶ added in v0.2.1
func ChainCloser[T any](it IteratorCloser[IteratorCloser[T]]) IteratorCloser[T]
ChainCloser chains closing iterator of closing iterators into a single continuous closing iterator. Each iterator is closed when exhausted, or when the top-level Close() is called. The top-level Close() also closes the input iterator of the iterators.
func MapCloser ¶ added in v0.2.1
func MapCloser[In, Out any](it IteratorCloser[In], f func(In) Out) IteratorCloser[Out]
MapCloser is like Map but propagates Close() to the output iterator.
func OrderedParallelMap ¶ added in v0.2.0
func OrderedParallelMap[In, Out any](ctx context.Context, workers, bufferSize int, it Iterator[In], f func(In) Out) IteratorCloser[Out]
OrderedParallelMap is like ParallMap, only it preserves the output order. The bufferSize must be >= workers, otherwise it effectively reduces parallelism to bufferSize workers. Larger bufferSize may improve parallelization when some jobs take a lot longer than others.
func ParallelMap ¶
func ParallelMap[In, Out any](ctx context.Context, workers int, it Iterator[In], f func(In) Out) IteratorCloser[Out]
ParallelMap runs multiple function calls f(In) in parallel on a given number of workers (0=unlimited), collects their results and returns as an iterator. The order of the results is undefined, unless the number of workers is 1.
Canceling the supplied context immediately stops queuing new jobs, but the jobs that already started will finish and their results will be returned. Therefore, it is important to flush the iterator after canceling the context to release all the resources.
Similarly, any early exit from the iterator loop must ensure that the context is canceled and the iterator is flushed.
No job is started by this method itself. Jobs begin to run on the first Next() call on the result iterator, which is go routine safe.
Example usage:
m := ParallelMap(context.Background(), 2, it, f)
defer m.Close()
for v, ok := m.Next(); ok; v, ok = m.Next() {
// Process v.
// Exiting early is safe, m will be closed and resources released.
}
func UnbatchCloser ¶ added in v0.2.1
func UnbatchCloser[T any](it IteratorCloser[[]T]) IteratorCloser[T]
UnbatchCloser is like Unbatch but propagates Close() to the output iterator.
func WithClose ¶ added in v0.1.6
func WithClose[T any](it Iterator[T], close func()) IteratorCloser[T]
WithClose attaches a close function to an iterator.
Source Files