Documentation ¶
Overview ¶
Package strit introduces a string iterator of "push" type, as well as a number of iterator constructors and wrappers.
Typically, iteration over some source of data involves creating an object of some reader type, then repeatedly calling a method on the object to get the data elements, and finally calling some method to release the resources allocated inside the reader object. And in the languages without exceptions (like Go) there must be an error check on each step. In other words, there is an interface and some protocol all the clients must follow, which usually results in certain amount of boilerplate code to write.
The above example describes the so called "pull" model of iteration where the client has to call a method on the reader to "pull" the next data element out of the reader object. Alternative to that is the "push" model where client gives the iterator a callback function to be invoked once per each element of input data.
Another aspect of the iteration model is about dealing with errors. In the absence of exceptions, like in Go language, the "pull" model implies that the result of each step of the iteration is either a data item, or an error. The same is true for pure "push" model where each invocation of the callback has to receive two parameters: a data element and an error. Or there maybe two callbacks, one for data and another for errors, but this is conceptually the same.
This library is an experiment in combining both "push" and "pull" models: it utilises "push" model for data elements, and "pull" model for error propagation. The data elements here are strings only, as currently the Go language does not offer any generic programming mechanism. The iterator type itself is a function that invokes the supplied callback once per each input string, and it returns either an error or nil. This arrangement reduces the amount of boilerplate code, at least in the most common scenarios, plus it allows for easy iterator combination, especially useful where dynamic creation of string processing pipelines is required.
The string iterator type used throughout this library is simply a function of the type func(Func) error, where Func is the type of the callback, func([]byte) error. Strings are represented as byte slices for performance reasons, to allow for in-place modifications where possible, like in bytes.TrimSpace() function. Iteration is simply an invocation of the iterator function with a callback to receive strings. The iterator function returns whatever error may have occurred during the iteration. The callback function itself may also return an error which will stop the iteration and will be returned from the iterator function. The special error value of io.EOF is treated as a request to stop the iteration and will not be propagated up to the caller.
The library offers a number of iterator constructors making iterators from different sources like strings, byte slices, byte readers, files and shell command outputs. There is also a number of mapping and filtering functions, as well as writers for strings, byte slices and files. All the above is composable using fluent API. For example, reading a file line by line, removing leading and trailing whitespace from each line, selecting only non-empty lines that also do not start from the character '#', and storing the result in a slice of strings, may look like this:
lines, err := strit.FromFile("some-file"). Map(bytes.TrimSpace). Filter(strit.Not(strit.Empty).OrNot(strit.StartsWith("#"))). Strings()
Here strit.FromFile is an iterator constructor, Map and Filter operations are iterator wrappers each creating a new iterator from the existing one, and Strings() is the iterator invocation. Also notice the use of the provided predicate combinators in the parameter to the Filter() function.
Given that each iterator is itself a function, other iterators can be derived from the existing ones. For example, the following function takes an iterator and creates a new iterator that prepends the line number to each line from the original iterator:
func Numbered(iter Iter) Iter { return func(fn Func) error { // this is the new iterator function i := 0 // this is the invocation of the original iterator function return iter(func(line []byte) error { i++ line = []byte(fmt.Sprintf("%d %s", i, string(line))) return fn(line) // invocation of the callback }) } }
Iterators in this library are lazy, meaning that the actual iteration happens only when the iterator function is called, not when it is created. Some of the provided iterators are reusable, meaning that such iterator may be called more than once, but in general this property cannot be guaranteed for those sources that cannot be read multiple times (like io.Reader), so in general it is advised that in the absence of any specific knowledge every iterator should be treated as not reusable. All the iterator implementations have O(n) complexity in time and O(1) in space.
One disadvantage of the suggested model is the complex implementation of parallel composition of iterators, see the comment to the Merge() function for more details. Also, there is a certain cost of abstraction incurred, in other words, a straight-forward implementation of the above example could be somewhat more efficient in terms of CPU cycles and memory consumption, though the difference would mostly come from the extra function calls and as such would probably be not so substantial, especially for i/o-bound sources. The actual benchmaking of some simple processing scenarios shows that the performance difference is truly minor.
Index ¶
- Variables
- func Empty(line []byte) bool
- func ScanNullTerminatedLines(data []byte, atEOF bool) (advance int, token []byte, err error)
- type ExitError
- type Func
- type Iter
- func Chain(its ...Iter) Iter
- func FromBytes(src []byte) Iter
- func FromBytesSF(sf bufio.SplitFunc, src []byte) Iter
- func FromCommand(cmd *exec.Cmd) Iter
- func FromCommandSF(cmd *exec.Cmd, sf bufio.SplitFunc) Iter
- func FromDir(name string, pred func(os.FileInfo) bool) Iter
- func FromDirWalk(root string, wf filepath.WalkFunc) Iter
- func FromFile(name string) Iter
- func FromFileSF(sf bufio.SplitFunc, name string) Iter
- func FromReadCloser(input io.ReadCloser) Iter
- func FromReadCloserSF(sf bufio.SplitFunc, input io.ReadCloser) Iter
- func FromReader(input io.Reader) Iter
- func FromReaderSF(sf bufio.SplitFunc, input io.Reader) Iter
- func FromString(src string) Iter
- func FromStringSF(sf bufio.SplitFunc, src string) Iter
- func FromStrings(src []string) Iter
- func Merge(sep string, its ...Iter) Iter
- func (iter Iter) Bytes() (res []byte, err error)
- func (iter Iter) Filter(pred Pred) Iter
- func (iter Iter) FirstNonEmpty(mapper func([]byte) []byte) Iter
- func (iter Iter) GenMap(mapper func([]byte) ([]byte, error)) Iter
- func (iter Iter) Join(sep string) (res string, err error)
- func (iter Iter) JoinBytes(sep string) (res []byte, err error)
- func (iter Iter) Map(mapper func([]byte) []byte) Iter
- func (iter Iter) Parse(p Parser) error
- func (iter Iter) Pipe(cmd *exec.Cmd) Iter
- func (iter Iter) PipeSF(cmd *exec.Cmd, sf bufio.SplitFunc) Iter
- func (iter Iter) Skip(numLines uint64) Iter
- func (iter Iter) SkipWhile(pred Pred) Iter
- func (iter Iter) String() (res string, err error)
- func (iter Iter) Strings() (res []string, err error)
- func (iter Iter) Take(numLines uint64) Iter
- func (iter Iter) TakeWhile(pred Pred) Iter
- func (iter Iter) WriteSepTo(dest io.Writer, sep string) (n int64, err error)
- func (iter Iter) WriteSepToFile(name, sep string) (n int64, err error)
- func (iter Iter) WriteTo(dest io.Writer) (int64, error)
- func (iter Iter) WriteToFile(name string) (int64, error)
- type Parser
- type ParserFunc
- type Pred
Constants ¶
This section is empty.
Variables ¶
var ErrSkip = errors.New("Item skipped")
ErrSkip is a special error type to skip an item in GenMap() function.
Functions ¶
func ScanNullTerminatedLines ¶
ScanNullTerminatedLines is a split function that splits input on null bytes. Useful mostly with FromCommandSF function, in cases where the invoked command generates null-terminated strings, like 'find ... -print0'.
Types ¶
type ExitError ¶
ExitError is the error type used for delivering command exit code and 'stderr' output.
type Iter ¶
Iter is the iterator type.
func Chain ¶
Chain implements sequential composition of its input iterators. The returned iterator invokes all the input iterators one after another, from left to the right.
func FromBytes ¶
FromBytes constructs a new iterator that reads the specified byte slice and breaks it into lines with line termination stripped.
func FromBytesSF ¶
FromBytesSF constructs a new iterator that reads the specified byte slice and breaks it into tokens using the supplied split function. If the function is set to nil then the iterator breaks the input into lines with line termination stripped. Internally the iterator is implemented using bufio.Scanner, please refer to its documentation for more details on split functions.
func FromCommand ¶
FromCommand constructs a new iterator that invokes the specified command, reads the command output (stdout) and breaks it into lines with line termination stripped.
func FromCommandSF ¶
FromCommandSF constructs a new iterator that invokes the specified command, reads the command output (stdout) and breaks it into tokens using the supplied split function. If the split function is set to nil then the iterator breaks the command output into lines with line termination stripped. Internally the iterator is implemented using bufio.Scanner, please refer to its documentation for more details on split functions.
func FromDir ¶
FromDir creates a new iterator that reads all entries from the specified directory and produces the names of those entries for which the supplied predicate returns 'true'. If the predicate is set to nil then the iterator produces only regular files, directories and symlinks to files.
func FromDirWalk ¶
FromDirWalk creates a new iterator that wraps filepath.Walk recursive directory traversal function. The iterator produces names of filesystem entries in accordance with the supplied WalkFunc, please refer to the filepath.Walk documentation for more details. If the WalkFunc is set to nil then the default function will be used which accepts all the filesystem entries.
func FromFile ¶
FromFile constructs a new iterator that reads its input byte stream from the specified file and breaks the stream into lines with line termination stripped.
func FromFileSF ¶
FromFileSF constructs a new iterator that reads its input byte stream from the specified file and breaks the stream into tokens using the supplied split function. If the function is set to nil then the iterator breaks the input into lines with line termination stripped. Internally the iterator is implemented using bufio.Scanner, please refer to its documentation for more details on split functions.
func FromReadCloser ¶
func FromReadCloser(input io.ReadCloser) Iter
FromReadCloser constructs a new iterator that reads its input byte stream from the specified Reader and breaks the input into lines with line termination stripped. The input Reader gets closed at the end of the iteration.
func FromReadCloserSF ¶
func FromReadCloserSF(sf bufio.SplitFunc, input io.ReadCloser) Iter
FromReadCloserSF is a wrapper around FromReaderSF with exactly the same functionality that also closes the input Reader at the end of the iteration.
func FromReader ¶
FromReader constructs a new iterator that reads its input byte stream from the specified Reader and breaks the input into lines with line termination stripped.
func FromReaderSF ¶
FromReaderSF constructs a new iterator that reads its input byte stream from the specified Reader and breaks the stream into tokens using the supplied split function. If the function is set to nil then the iterator breaks the input into lines with line terminators stripped. Internally the iterator is implemented using bufio.Scanner, please refer to its documentation for more details on split functions.
func FromString ¶
FromString constructs a new iterator that reads the specified string and breaks it into lines with line termination stripped.
func FromStringSF ¶
FromStringSF constructs a new iterator that reads the specified string and breaks it into tokens using the supplied split function. If the function is set to nil then the iterator breaks the input into lines with line termination stripped. Internally the iterator is implemented using bufio.Scanner, please refer to its documentation for more details on split functions.
func FromStrings ¶
FromStrings constructs a new iterator that reads the specified slice of strings, one string at a time.
func Merge ¶
Merge implements parallel composition of its input iterators. The returned iterator on each step invokes all its inputs in parallel, taking one string from each input iterator, and then joins those strings around the specified separator to produce one output string. The iteration stops when any input iterator gets exhausted. It should be noted that the iterator type used in this library provides for easy sequential composition of iterators, but the parallel composition is substantially more complex. In the current implementation each input iterator except the first one runs in a dedicated goroutine which pipes its output back through a channel. That complexity is usually not a problem if at least one of the input iterators is i/o-bound, but it is likely to become a performance bottleneck if all the iterators read from the memory. Please use this function responsibly.
func (Iter) Filter ¶
Filter makes a new iterator that produces only the strings for which the supplied predicate returns 'true'.
func (Iter) FirstNonEmpty ¶
FirstNonEmpty makes a new iterator that skips everything until the supplied mapper function returns a non-empty slice, which gets passed down the pipeline, and then the iteration stops. This is essentially a search for the first non-empty string returned from the mapper.
func (Iter) GenMap ¶
GenMap makes a new iterator that applies the supplied function to every input string. The function may return a non-nil error, in which case the iteration will stop and the error will be propagated back to the source. A special error value of ErrSkip instructs the iterator to simply skip the current string.
func (Iter) Join ¶
Join invokes the iterator and collects its output into one string, delimited by the specified separator.
func (Iter) JoinBytes ¶
JoinBytes invokes the iterator and collects its output into one byte slice, delimited by the specified separator.
func (Iter) Map ¶
Map makes a new iterator that applies the specified function to every input string.
func (Iter) Pipe ¶
Pipe makes an iterator that pumps the data from its parent through the specified command and iterates over the command's stdout.
func (Iter) PipeSF ¶
PipeSF makes an iterator that pumps the data from its parent through the specified command and iterates over the command's stdout, using the given splitter to separate strings.
func (Iter) Skip ¶
Skip makes a new iterator that skips the specified number of input strings and passes all the remaining strings to the callback function.
func (Iter) SkipWhile ¶
SkipWhile makes a new iterator that skips input strings until the supplied predicate returns 'false' for the first time and passes the remaining strings to the callback function.
func (Iter) Take ¶
Take makes a new iterator that produces no more than the specified number of output strings.
func (Iter) TakeWhile ¶
TakeWhile makes a new iterator that produces its output while the specified predicate returns 'true'.
func (Iter) WriteSepTo ¶
WriteSepTo invokes the iterator and writes its output strings, delimited by the specified separator, into the specified Writer. The method returns the total number of bytes written, or an error.
func (Iter) WriteSepToFile ¶
WriteSepToFile creates a file with the specified name, and then invokes the iterator and writes its output strings, delimited by the specified separator, into the file. The method returns the total number of bytes written, or an error, in which case the resulting file gets deleted.
func (Iter) WriteTo ¶
WriteTo invokes the iterator and writes its output strings, delimited by new line character, into the specified Writer. The method returns the total number of bytes written, or an error.
func (Iter) WriteToFile ¶
WriteToFile creates a file with the specified name, and then invokes the iterator and writes its output strings, delimited by new line character, into the file. The method returns the total number of bytes written, or an error, in which case the resulting file gets deleted.
type Parser ¶
type Parser interface { // Enter is the entry point of the parser and gets called on the first iteration. // The returned function usually refers to another method of the same class, thus implementing // the parser state machine. Any non-nil error stops the iteration. Enter([]byte) (ParserFunc, error) // Done is called after the iteration is complete or an error has occurred. The error // it returns (if any) becomes the return value of the iterator. Done(error) error }
Parser is the interface of a line parser.
type ParserFunc ¶
type ParserFunc func([]byte) (ParserFunc, error)
ParserFunc represents class of functions implementing parser state machines.
type Pred ¶
Pred is the type of string predicate. The type has a number of combining methods allowing for convenient composition of predicate functions, for example:
strit.Not(strit.Empty).AndNot(strit.StartsWith("#"))
or
strit.StartsWith("xyz").Or(strit.StartsWith("abc"))
func EndsWith ¶
EndsWith is a predicate that returns 'true' if the input string has the specified suffix.
func Not ¶
Not is a predicate wrapper that returns a new predicate negating the result of the supplied predicate.
func StartsWith ¶
StartsWith is a predicate that returns 'true' if the input string has the specified prefix.
func (Pred) And ¶
And is a predicate combinator. It creates a new predicate that applies logical 'and' to the original and the supplied predicates.
func (Pred) AndNot ¶
AndNot is a predicate combinator. It creates a new predicate that returns 'true' only if the original returns 'true' and the other predicate returns 'false'.