csvplus: github.com/maxim2266/csvplus Index | Files

package csvplus

import "github.com/maxim2266/csvplus"

Package csvplus extends the standard Go encoding/csv package with fluent interface, lazy stream processing operations, indices and joins.

Index

Package Files

csvplus.go

func All Uses

func All(funcs ...func(Row) bool) func(Row) bool

All is a predicate combinator that takes any number of other predicates and produces a new predicate which returns 'true' only if all the specified predicates return 'true' for the same input Row.

func Any Uses

func Any(funcs ...func(Row) bool) func(Row) bool

Any is a predicate combinator that takes any number of other predicates and produces a new predicate which returns 'true' if any the specified predicates returns 'true' for the same input Row.

func Like Uses

func Like(match Row) func(Row) bool

Like produces a predicate that returns 'true' if its input Row matches all the corresponding values from the specified 'match' Row.

func Not Uses

func Not(pred func(Row) bool) func(Row) bool

Not produces a new predicate that reverts the return value from the given predicate.

type DataSource Uses

type DataSource func(RowFunc) error

DataSource is the iterator type used throughout this library. The iterator calls the given RowFunc once per each row. The iteration continues until either the data source is exhausted or the supplied RowFunc returns a non-nil error, in which case the error is returned back to the caller of the iterator. A special case of io.EOF simply stops the iteration and the iterator function returns nil error.

func Take Uses

func Take(src interface {
    Iterate(fn RowFunc) error
}) DataSource

Take converts anything with Iterate() method to a DataSource.

func TakeRows Uses

func TakeRows(rows []Row) DataSource

TakeRows converts a slice of Rows to a DataSource.

func (DataSource) Drop Uses

func (src DataSource) Drop(n uint64) DataSource

Drop specifies the number of Rows to ignore before passing the remaining rows down the pipeline.

func (DataSource) DropColumns Uses

func (src DataSource) DropColumns(columns ...string) DataSource

DropColumns removes the specifies columns from each row.

func (DataSource) DropWhile Uses

func (src DataSource) DropWhile(pred func(Row) bool) DataSource

DropWhile ignores all the Rows for as long as the specified predicate is true; afterwards all the remaining Rows are passed down the pipeline.

func (DataSource) Except Uses

func (src DataSource) Except(index *Index, columns ...string) DataSource

Except returns a table containing all the rows not in the specified Index, unchanged. The specified columns are matched against those from the index, in the order of specification. If no columns are specified then the columns list is taken from the index.

func (DataSource) Filter Uses

func (src DataSource) Filter(pred func(Row) bool) DataSource

Filter takes a predicate which, when applied to a Row, decides if that Row should be passed down for further processing. The predicate should return 'true' to pass the Row.

func (DataSource) IndexOn Uses

func (src DataSource) IndexOn(columns ...string) (*Index, error)

IndexOn iterates the input source, building index on the specified columns. Columns are taken from the specified list from left to the right.

func (DataSource) Join Uses

func (src DataSource) Join(index *Index, columns ...string) DataSource

Join returns a DataSource which is a join between the current DataSource and the specified Index. The specified columns are matched against those from the index, in the order of specification. Empty 'columns' list yields a join on the columns from the Index (aka "natural join") which all must exist in the current DataSource. Each row in the resulting table contains all the columns from both the current table and the index. This is a lazy operation, the actual join is performed only when the resulting table is iterated over.

func (DataSource) Map Uses

func (src DataSource) Map(mf func(Row) Row) DataSource

Map takes a function which gets applied to each Row when the source is iterated over. The function may return a modified input Row, or an entirely new Row.

func (DataSource) SelectColumns Uses

func (src DataSource) SelectColumns(columns ...string) DataSource

SelectColumns leaves only the specified columns on each row. It is an error if any such column does not exist.

func (DataSource) TakeWhile Uses

func (src DataSource) TakeWhile(pred func(Row) bool) DataSource

TakeWhile takes a predicate which gets applied to each Row upon iteration. The iteration stops when the predicate returns 'false'.

func (DataSource) ToCsv Uses

func (src DataSource) ToCsv(out io.Writer, columns ...string) (err error)

ToCsv iterates the data source and writes the selected columns in .csv format to the given io.Writer. The data are written in the "canonical" form with the header on the first line and with all the lines having the same number of fields, using default settings for the underlying csv.Writer.

func (DataSource) ToCsvFile Uses

func (src DataSource) ToCsvFile(name string, columns ...string) error

ToCsvFile iterates the data source and writes the selected columns in .csv format to the given file. The data are written in the "canonical" form with the header on the first line and with all the lines having the same number of fields, using default settings for the underlying csv.Writer.

func (DataSource) ToJSON Uses

func (src DataSource) ToJSON(out io.Writer) (err error)

ToJSON iterates over the data source and writes all Rows to the given io.Writer in JSON format.

func (DataSource) ToJSONFile Uses

func (src DataSource) ToJSONFile(name string) error

ToJSONFile iterates over the data source and writes all Rows to the given file in JSON format.

func (DataSource) ToRows Uses

func (src DataSource) ToRows() (rows []Row, err error)

ToRows iterates the DataSource storing the result in a slice of Rows.

func (DataSource) Top Uses

func (src DataSource) Top(n uint64) DataSource

Top specifies the number of Rows to pass down the pipeline before stopping the iteration.

func (DataSource) Transform Uses

func (src DataSource) Transform(trans func(Row) (Row, error)) DataSource

Transform is the most generic operation on a Row. It takes a function that maps a Row to another Row and an error. Any error returned from that function stops the iteration, otherwise the returned Row, if not empty, gets passed down to the next stage of the processing pipeline.

func (DataSource) UniqueIndexOn Uses

func (src DataSource) UniqueIndexOn(columns ...string) (*Index, error)

UniqueIndexOn iterates the input source, building unique index on the specified columns. Columns are taken from the specified list from left to the right.

func (DataSource) Validate Uses

func (src DataSource) Validate(vf func(Row) error) DataSource

Validate takes a function which checks every Row upon iteration and returns an error if the validation fails. The iteration stops at the first error encountered.

type DataSourceError Uses

type DataSourceError struct {
    Line uint64 // counting from 1
    Err  error
}

DataSourceError is the type of the error returned from Reader.Iterate method.

func (*DataSourceError) Error Uses

func (e *DataSourceError) Error() string

Error returns a human-readable error message string.

type Index Uses

type Index struct {
    // contains filtered or unexported fields
}

Index is a sorted collection of Rows with O(log(n)) complexity of search on the indexed columns. Iteration over the Index yields a sequence of Rows sorted on the index.

func LoadIndex Uses

func LoadIndex(fileName string) (*Index, error)

LoadIndex reads the index from the specified file.

func (*Index) Find Uses

func (index *Index) Find(values ...string) DataSource

Find returns a DataSource of all Rows from the Index that match the specified values in the indexed columns, left to the right. The number of specified values may be less than the number of the indexed columns.

func (*Index) Iterate Uses

func (index *Index) Iterate(fn RowFunc) error

Iterate iterates over all rows of the index. The rows are sorted by the values of the columns specified when the Index was created.

func (*Index) ResolveDuplicates Uses

func (index *Index) ResolveDuplicates(resolve func(rows []Row) (Row, error)) error

ResolveDuplicates calls the specified function once per each pack of duplicates with the same key. The specified function must not modify its parameter and is expected to do one of the following:

- Select and return one row from the input list. The row will be used as the only row with its key;

- Return an empty row. The entire set of rows will be ignored;

- Return an error which will be passed back to the caller of ResolveDuplicates().

func (*Index) SubIndex Uses

func (index *Index) SubIndex(values ...string) *Index

SubIndex returns an Index containing only the rows where the values of the indexed columns match the supplied values, left to the right. The number of specified values must be less than the number of indexed columns.

func (*Index) WriteTo Uses

func (index *Index) WriteTo(fileName string) (err error)

WriteTo writes the index to the specified file.

type Reader Uses

type Reader struct {
    // contains filtered or unexported fields
}

Reader is iterable csv reader. The iteration is performed in its Iterate() method, which may only be invoked once per each instance of the Reader.

func FromFile Uses

func FromFile(name string) *Reader

FromFile constructs a new csv reader bound to the specified file, with default settings.

func FromReadCloser Uses

func FromReadCloser(input io.ReadCloser) *Reader

FromReadCloser constructs a new csv reader from the given io.ReadCloser, with default settings.

func FromReader Uses

func FromReader(input io.Reader) *Reader

FromReader constructs a new csv reader from the given io.Reader, with default settings.

func (*Reader) AssumeHeader Uses

func (r *Reader) AssumeHeader(spec map[string]int) *Reader

AssumeHeader sets the header for those input sources that do not have their column names specified on the first row. The header specification is a map from the assigned column names to their corresponding column indices.

func (*Reader) CommentChar Uses

func (r *Reader) CommentChar(c rune) *Reader

CommentChar sets the symbol that starts a comment.

func (*Reader) Delimiter Uses

func (r *Reader) Delimiter(c rune) *Reader

Delimiter sets the symbol to be used as a field delimiter.

func (*Reader) ExpectHeader Uses

func (r *Reader) ExpectHeader(spec map[string]int) *Reader

ExpectHeader sets the header for input sources that have their column names specified on the first row. The row gets verified against this specification when the reading starts. The header specification is a map from the expected column names to their corresponding column indices. A negative value for an index means that the real value of the index will be found by searching the first row for the specified column name.

func (*Reader) Iterate Uses

func (r *Reader) Iterate(fn RowFunc) error

Iterate reads the input row by row, converts each line to the Row type, and calls the supplied RowFunc.

func (*Reader) LazyQuotes Uses

func (r *Reader) LazyQuotes() *Reader

LazyQuotes specifies that a quote may appear in an unquoted field and a non-doubled quote may appear in a quoted field of the input.

func (*Reader) NumFields Uses

func (r *Reader) NumFields(n int) *Reader

NumFields sets the expected number of fields on each row of the input. It is an error if any row does not have this exact number of fields.

func (*Reader) NumFieldsAny Uses

func (r *Reader) NumFieldsAny() *Reader

NumFieldsAny specifies that each row of the input may have different number of fields. Rows shorter than the maximum column index in the header specification will be padded with empty fields.

func (*Reader) NumFieldsAuto Uses

func (r *Reader) NumFieldsAuto() *Reader

NumFieldsAuto specifies that the number of fields on each row must match that of the first row of the input.

func (*Reader) SelectColumns Uses

func (r *Reader) SelectColumns(names ...string) *Reader

SelectColumns specifies the names of the columns to read from the input source. The header specification is built by searching the first row of the input for the names specified and recording the indices of those columns. It is an error if any column name is not found.

func (*Reader) TrimLeadingSpace Uses

func (r *Reader) TrimLeadingSpace() *Reader

TrimLeadingSpace specifies that the leading white space in a field should be ignored.

type Row Uses

type Row map[string]string

Row represents one line from a data source like a .csv file.

Each Row is a map from column names to the string values under that columns on the current line. It is assumed that each column has a unique name. In a .csv file, the column names may either come from the first line of the file ("expected header"), or they can be set-up via configuration of the reader object ("assumed header").

Using meaningful column names instead of indices is usually more convenient when the columns get rearranged during the execution of the processing pipeline.

func (Row) Clone Uses

func (row Row) Clone() Row

Clone returns a copy of the current Row.

func (Row) HasColumn Uses

func (row Row) HasColumn(col string) (found bool)

HasColumn is a predicate returning 'true' when the specified column is present.

func (Row) Header Uses

func (row Row) Header() []string

Header returns a slice of all column names, sorted via sort.Strings.

func (Row) SafeGetValue Uses

func (row Row) SafeGetValue(col, subst string) string

SafeGetValue returns the value under the specified column, if present, otherwise it returns the substitution value.

func (Row) Select Uses

func (row Row) Select(cols ...string) (Row, error)

Select takes a list of column names and returns a new Row containing only the specified columns, or an error if any column is not present.

func (Row) SelectExisting Uses

func (row Row) SelectExisting(cols ...string) Row

SelectExisting takes a list of column names and returns a new Row containing only those columns from the list that are present in the current Row.

func (Row) SelectValues Uses

func (row Row) SelectValues(cols ...string) ([]string, error)

SelectValues takes a list of column names and returns a slice of their corresponding values, or an error if any column is not present.

func (Row) String Uses

func (row Row) String() string

String returns a string representation of the Row.

func (Row) ValueAsFloat64 Uses

func (row Row) ValueAsFloat64(column string) (res float64, err error)

ValueAsFloat64 returns the value of the given column converted to floating point type, or an error. The column must be present on the row.

func (Row) ValueAsInt Uses

func (row Row) ValueAsInt(column string) (res int, err error)

ValueAsInt returns the value of the given column converted to integer type, or an error. The column must be present on the row.

type RowFunc Uses

type RowFunc func(Row) error

RowFunc is the function type used when iterating Rows.

Package csvplus imports 12 packages (graph) and is imported by 1 packages. Updated 2019-09-28. Refresh now. Tools for package owners.