etl

package
v0.0.18-alpha Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 29, 2024 License: MIT Imports: 5 Imported by: 0

Documentation

Overview

Package etl provides a simple iterator interface.

Index

Constants

This section is empty.

Variables

View Source 
var EOI = io.EOF

EOI is returned when iterator doesn't have more data.

View Source 
var ErrCancelled = errors.New("iterator cancelled")

Functions

func Collect 

func Collect[T any](it Iter) ([]T, error)

func CollectContext 

func CollectContext[T any](ctx context.Context, it Iter) ([]T, error)

Collect collects all iterator values into a slice.

func Consume 

func Consume[T any](it Iter, fn func(T) error) error

Consume iterates over the given iterator and calls fn for each value.

func ConsumeBatch 

func ConsumeBatch[T any](it Iter, n int, fn func([]T) error) error

ConsumeBatch consumes the given iterator in batches of n values.

func ConsumeContext 

func ConsumeContext[T any](ctx context.Context, it Iter, fn func(T) error) error

func Count 

func Count(it Iter) (int, error)

Count consumes the iterator and return the number of iterations.

func Print 

func Print(it Iter) error

Print prints the values of the given iterator to stdout.

func Take 

func Take[T any](it Iter, n int) ([]T, error)

Take consumes the given iterator and return the first n values as a slice.

func WorkersConsume 

func WorkersConsume[Ti any](it Iter, workers int, fn func(context.Context, Ti) error) error

WorkersConsume creates a pool of workers that call fn for every iteration value. this will close the consumed Iter upon finish.

func WorkersConsumeContext 

func WorkersConsumeContext[Ti any](ctx context.Context, it Iter, workers int, fn func(context.Context, Ti) error) error

Types

type Custom 

type Custom[T any] struct {
	Next  func(context.Context) (T, error)
	Close func() error
}

type FilterFunc 

type FilterFunc[T any] func(T) bool

type Gen 

type Gen[T any] struct {
	Run   func(context.Context, Y[T]) error
	Close func() error
}

type Iter 

type Iter interface {
	Next(context.Context) (any, error)
	Close() error
}

Iter iterator interface.

func Cat 

func Cat(its ...Iter) Iter

func Chan 

func Chan[T any](ch <-chan T) Iter

func ErrIter 

func ErrIter(err error) Iter

ErrIter is an iterator that always returns an error.

func Filter 

func Filter[T any](it Iter, fn FilterFunc[T]) Iter

Filter returns an iterator that filters the values of the source given the func fn if fn returns true the value is passed through

func FlatMap 

func FlatMap[Ti, To any](it Iter, fn func(Ti) []To) Iter

func Limit 

func Limit(it Iter, n int) Iter

Limit returns an iterator that returns at most n values from the given iterator Closing the returned iterator will close the given iterator.

func MakeGen 

func MakeGen[T any](g Gen[T]) Iter

func MakeIter 

func MakeIter[T any](c Custom[T]) Iter

func Map 

func Map[Ti, To any](it Iter, fn func(Ti) To) Iter

Map returns an interator that transforms the values of the source iterator using the func fn.

func MapE 

func MapE[Ti, To any](it Iter, fn func(Ti) (To, error)) Iter

Map returns an interator that transforms the values of the source iterator using the func fn.

func NewIter 

func NewIter[IT, DT any](
	start func() (IT, error),
	next func(context.Context, IT) (DT, error),
	end func(IT) error,
) Iter

func Peek 

func Peek[T any](it Iter, fn func(v T)) Iter

Peek calls the func fn each time the Next from the iterator is called

func Seq 

func Seq(start, end, step int) Iter

Seq iterates over the sequence of integers from start to end.

func Values 

func Values[T any](vs ...T) Iter

Values returns an iterator that iterates over the variadic arguments.

func Workers 

func Workers[To any](it Iter, workers int, fn func(context.Context, W[To]) error) Iter

Workers iterates over a slice of values and calls a worker func for each value.

func WorkersValue 

func WorkersValue[Ti, To any](it Iter, workers int, fn func(context.Context, Ti, Y[To]) error) Iter

WorkersValue is a convinitent func that calls fn for every consumed value, it will yield any value by calling the yield func.

func Yield 

func Yield[Ti, To any](it Iter, fn func(Ti, Y[To]) error) Iter

Yield returns an iterator that for each consumed value calls a fn passing an yielder,

type W 

type W[To any] struct {
	Iter
	ID int
	// contains filtered or unexported fields
}

func (W[To]) Yield 

func (w W[To]) Yield(v To) error

type Y 

type Y[T any] func(T) error

Y yield parameter.

Source Files

View all Source files

Directories

Path Synopsis
Package etlcsv contains iterators that handle csv data.
 Package etlcsv contains iterators that handle csv data.
Package etldrow contains functions to manipulate iter.Iter of drow.Row.
 Package etldrow contains functions to manipulate iter.Iter of drow.Row.
Package iterfs provides file system iterators.
 Package iterfs provides file system iterators.
Package etlhttp contains iterators that handle http.
 Package etlhttp contains iterators that handle http.
Package etlio contains logic do handle io stuff.
 Package etlio contains logic do handle io stuff.
Package etljson provides iterators to handle json.
 Package etljson provides iterators to handle json.
Package itermetrics contains utils to debug and measure iterators.
 Package itermetrics contains utils to debug and measure iterators.
mysql
psql
Package etlutil contains basic iter utils.
 Package etlutil contains basic iter utils.
Package etlzip iterates over zip files data.
 Package etlzip iterates over zip files data.
x
etlcloud
Package etlcloud provides etl iters based on gocloud.dev
 Package etlcloud provides etl iters based on gocloud.dev
etlgzip
etlparquet

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.