etable

package
v2.0.0-dev0.0.19 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 15, 2024 License: BSD-3-Clause Imports: 19 Imported by: 44

README

etable

Docs: GoDoc

etable provides the etable.Table structure which provides a DataTable or DataFrame data representation, which is a collection of columnar data all having the same number of rows.

Each column is an etensor.Tensor, so it can represent scalar or higher dimensional data per each cell (row x column location) in the Table. Thus, scalar data is represented using a 1D Tensor where the 1 dimension is the rows of the table, and likewise higher dimensional data always has the outer-most dimension as the row.

All tensors MUST have RowMajor stride layout for consistency, with the outer-most dimension as the row dimension, which is enforced to be the same across all columns.

The tensor columns can be individually converted to / from arrow.Tensors and conversion between arrow.Table is planned, along with inter-conversion with relevant gonum structures including the planned dframe.Frame.

Native support is provided for basic CSV, TSV I/O, including the C++ emergent standard TSV format with full type information in the first row column headers.

The etable.IndexView is an indexed view into a Table, which is used for all data-processing operations such as Sort, Filter, Split (group), and for aggregating data as in a pivot-table.

See agg package for aggregation functions that operate on the IndexView to perform standard aggregation operations such as Sum, Mean, etc, and split for pivot table support.

Other relevant examples of DataTable-like structures:

Documentation

Overview

Package etable provides the etable.Table structure which provides a DataTable or DataFrame data representation, which is a collection of columnar data all having the same number of rows.

Each column is an etensor.Tensor, so it can represent scalar or higher dimensional data per each cell (row x column location) in the Table. Thus, scalar data is represented using a 1D Tensor where the 1 dimension is the rows of the table, and likewise higher dimensional data always has the outer-most dimension as the row.

All tensors MUST have RowMajor stride layout for consistency, with the outer-most dimension as the row dimension, which is enforced to be the same across all columns.

The tensor columns can be individually converted to / from arrow.Tensors and conversion between arrow.Table is planned, along with inter-conversion with relevant gonum structures including the planned dframe.Frame.

Native support is provided for basic CSV, TSV I/O, including the C++ emergent standard TSV format with full type information in the first row column headers.

The etable.IndexView is an indexed view into a Table, which is used for all data-processing operations such as Sort, Filter, Split (groupby), and for aggregating data as in a pivot-table.

See etable/agg package for aggregation functions that operate on the IndexView to perform standard aggregation operations such as Sum, Mean, etc.

Other relevant examples of DataTable-like structures: * https://github.com/apache/arrow/tree/master/go/arrow Table * http://xarray.pydata.org/en/stable/index.html * https://pandas.pydata.org/pandas-docs/stable/reference/frame.html * https://www.rdocumentation.org/packages/base/versions/3.4.3/topics/data.frame * https://github.com/tobgu/qframe * https://github.com/kniren/gota

Index

Constants

View Source 
const (
	// Contains means the string only needs to contain the target string (see Equals)
	Contains bool = true
	// Equals means the string must equal the target string (see Contains)
	Equals = false
	// IgnoreCase means that differences in case are ignored in comparing strings
	IgnoreCase = true
	// UseCase means that case matters when comparing strings
	UseCase = false
)

Named arg values for Contains, IgnoreCase

View Source 
const (
	// Ascending specifies an ascending sort direction for etable Sort routines
	Ascending = true

	// Descending specifies a descending sort direction for etable Sort routines
	Descending = false
)
View Source 
const (
	//	Headers is passed to CSV methods for the headers arg, to use headers
	Headers = true

	// NoHeaders is passed to CSV methods for the headers arg, to not use headers
	NoHeaders = false
)
View Source 
const (
	// ColNameOnly means resulting agg table just has the original column name, no aggregation name
	ColNameOnly bool = true
	// AddAggName means resulting agg table columns have aggregation name appended
	AddAggName = false
)

use these for arg to ArgsToTable*

Variables

View Source 
var EmerHdrCharToType = map[byte]etensor.Type{
	'$': etensor.STRING,
	'%': etensor.FLOAT32,
	'#': etensor.FLOAT64,
	'|': etensor.INT64,
	'@': etensor.UINT8,
	'&': etensor.STRING,
	'^': etensor.BOOL,
}
View Source 
var EmerHdrTypeToChar map[etensor.Type]byte

Functions

func DetectEmerHeaders 

func DetectEmerHeaders(hdrs []string) bool

DetectEmerHeaders looks for emergent header special characters -- returns true if found

func EmerColType 

func EmerColType(nm string) (etensor.Type, string)

EmerColType parses the column header for type information using the emergent naming convention

func FilterNull 

func FilterNull(et *Table, row int) bool

FilterNull is a FilterFunc that filters out all rows that have a Null value in a 1D (scalar) column, according to the IsNull flag

func InferDataType 

func InferDataType(str string) etensor.Type

InferDataType returns the inferred data type for the given string only deals with float64, int, and string types

func ShapeFromString 

func ShapeFromString(dims string) []int

ShapeFromString parses string representation of shape as N:d,d,..

Types

type Column 

type Column struct {

	// name of column -- must be unique for a table
	Name string

	// data type, using etensor types which are isomorphic with arrow.Type
	Type etensor.Type

	// shape of a single cell in the column (i.e., without the row dimension) -- for scalars this is nil -- tensor column will add the outer row dimension to this shape
	CellShape []int

	// names of the dimensions within the CellShape -- 'Row' will be added to outer dimension
	DimNames []string
}

Column specifies everything about a column -- can be used for constructing tables

type Delims 

type Delims int32 //enums:enum

Delim are standard CSV delimiter options (Tab, Comma, Space) 

const (
	// Tab is the tab rune delimiter, for TSV tab separated values
	Tab Delims = iota

	// Comma is the comma rune delimiter, for CSV comma separated values
	Comma

	// Space is the space rune delimiter, for SSV space separated value
	Space

	// Detect is used during reading a file -- reads the first line and detects tabs or commas
	Detect
)
const DelimsN Delims = 4

DelimsN is the highest valid value for type Delims, plus one.

func DelimsValues 

func DelimsValues() []Delims

DelimsValues returns all possible values for the type Delims.

func (Delims) Desc 

func (i Delims) Desc() string

Desc returns the description of the Delims value.

func (Delims) Int64 

func (i Delims) Int64() int64

Int64 returns the Delims value as an int64.

func (Delims) MarshalText 

func (i Delims) MarshalText() ([]byte, error)

MarshalText implements the encoding.TextMarshaler interface.

func (Delims) Rune 

func (dl Delims) Rune() rune

func (*Delims) SetInt64 

func (i *Delims) SetInt64(in int64)

SetInt64 sets the Delims value from an int64.

func (*Delims) SetString 

func (i *Delims) SetString(s string) error

SetString sets the Delims value from its string representation, and returns an error if the string is invalid.

func (Delims) String 

func (i Delims) String() string

String returns the string representation of this Delims value.

func (*Delims) UnmarshalText 

func (i *Delims) UnmarshalText(text []byte) error

UnmarshalText implements the encoding.TextUnmarshaler interface.

func (Delims) Values 

func (i Delims) Values() []enums.Enum

Values returns all possible values for the type Delims.

type FilterFunc 

type FilterFunc func(et *Table, row int) bool

FilterFunc is a function used for filtering that returns true if Table row should be included in the current filtered view of the table, and false if it should be removed.

type IndexView 

type IndexView struct {

	// Table that we are an indexed view onto
	Table *Table

	// current indexes into Table
	Indexes []int
	// contains filtered or unexported fields
}

IndexView is an indexed wrapper around an etable.Table that provides a specific view onto the Table defined by the set of indexes. This provides an efficient way of sorting and filtering a table by only updating the indexes while doing nothing to the Table itself. To produce a table that has data actually organized according to the indexed order, call the NewTable method. IndexView views on a table can also be organized together as Splits of the table rows, e.g., by grouping values along a given column.

func NewIndexView 

func NewIndexView(et *Table) *IndexView

NewIndexView returns a new IndexView based on given table, initialized with sequential idxes

func (*IndexView) AddIndex 

func (ix *IndexView) AddIndex(idx int)

AddIndex adds a new index to the list

func (*IndexView) AddRows 

func (ix *IndexView) AddRows(n int)

AddRows adds n rows to end of underlying Table, and to the indexes in this view

func (*IndexView) AggCol 

func (ix *IndexView) AggCol(colIndex int, ini float64, fun etensor.AggFunc) []float64

AggCol applies given aggregation function to each element in the given column, using float64 conversions of the values. init is the initial value for the agg variable. Operates independently over each cell on n-dimensional columns and returns the result as a slice of values per cell.

func (*IndexView) Clone 

func (ix *IndexView) Clone() *IndexView

Clone returns a copy of the current index view with its own index memory

func (*IndexView) CopyFrom 

func (ix *IndexView) CopyFrom(oix *IndexView)

CopyFrom copies from given other IndexView (we have our own unique copy of indexes)

func (*IndexView) DeleteInvalid 

func (ix *IndexView) DeleteInvalid()

DeleteInvalid deletes all invalid indexes from the list. Call this if rows (could) have been deleted from table.

func (*IndexView) DeleteRows 

func (ix *IndexView) DeleteRows(at, n int)

DeleteRows deletes n rows of indexes starting at given index in the list of indexes

func (*IndexView) Filter 

func (ix *IndexView) Filter(filterFunc func(et *Table, row int) bool)

Filter filters the indexes into our Table using given Filter function. The Filter function operates directly on row numbers into the Table as these row numbers have already been projected through the indexes.

func (*IndexView) FilterCol 

func (ix *IndexView) FilterCol(colIndex int, str string, exclude, contains, ignoreCase bool)

FilterCol sorts the indexes into our Table according to values in given column index, using string representation of column values. Includes rows with matching values unless exclude is set. If contains, only checks if row contains string; if ignoreCase, ignores case. Use named args for greater clarity. Only valid for 1-dimensional columns.

func (*IndexView) FilterColName 

func (ix *IndexView) FilterColName(colNm string, str string, exclude, contains, ignoreCase bool) error

FilterColName filters the indexes into our Table according to values in given column name, using string representation of column values. Includes rows with matching values unless exclude is set. If contains, only checks if row contains string; if ignoreCase, ignores case. Use named args for greater clarity. Only valid for 1-dimensional columns. Returns error if column name not found.

func (*IndexView) InsertRows 

func (ix *IndexView) InsertRows(at, n int)

InsertRows adds n rows to end of underlying Table, and to the indexes starting at given index in this view

func (*IndexView) Len 

func (ix *IndexView) Len() int

Len returns the length of the index list

func (*IndexView) Less 

func (ix *IndexView) Less(i, j int) bool

Less calls the LessFunc for sorting

func (*IndexView) NewTable 

func (ix *IndexView) NewTable() *Table

NewTable returns a new table with column data organized according to the indexes

func (*IndexView) OpenCSV 

func (ix *IndexView) OpenCSV(filename core.Filename, delim Delims) error

OpenCSV reads a table idx view from a comma-separated-values (CSV) file (where comma = any delimiter, specified in the delim arg), using the Go standard encoding/csv reader conforming to the official CSV standard. If the table does not currently have any columns, the first row of the file is assumed to be headers, and columns are constructed therefrom. The C++ emergent column headers are parsed -- these have full configuration information for tensor dimensionality. If the table DOES have existing columns, then those are used robustly for whatever information fits from each row of the file.

func (*IndexView) OpenFS 

func (ix *IndexView) OpenFS(fsys fs.FS, filename string, delim Delims) error

OpenFS is the version of IndexView.OpenCSV that uses an fs.FS filesystem.

func (*IndexView) Permuted 

func (ix *IndexView) Permuted()

Permuted sets indexes to a permuted order -- if indexes already exist then existing list of indexes is permuted, otherwise a new set of permuted indexes are generated

func (*IndexView) RowsByString 

func (ix *IndexView) RowsByString(colNm string, str string, contains, ignoreCase bool) []int

RowsByString returns the list of *our indexes* whose row in the table has given string value in given column name (de-reference our indexes to get actual row). if contains, only checks if row contains string; if ignoreCase, ignores case. returns nil if name invalid -- see also Try. Use named args for greater clarity.

func (*IndexView) RowsByStringIndex 

func (ix *IndexView) RowsByStringIndex(colIndex int, str string, contains, ignoreCase bool) []int

RowsByStringIndex returns the list of *our indexes* whose row in the table has given string value in given column index (de-reference our indexes to get actual row). if contains, only checks if row contains string; if ignoreCase, ignores case. Use named args for greater clarity.

func (*IndexView) RowsByStringTry 

func (ix *IndexView) RowsByStringTry(colNm string, str string, contains, ignoreCase bool) ([]int, error)

RowsByStringTry returns the list of *our indexes* whose row in the table has given string value in given column name (de-reference our indexes to get actual row). if contains, only checks if row contains string; if ignoreCase, ignores case. returns error message for invalid column name. Use named args for greater clarity.

func (*IndexView) SaveCSV 

func (ix *IndexView) SaveCSV(filename core.Filename, delim Delims, headers bool) error

SaveCSV writes a table idx view to a comma-separated-values (CSV) file (where comma = any delimiter, specified in the delim arg). If headers = true then generate C++ emergent-tyle column headers. These headers have full configuration information for the tensor columns. Otherwise, only the data is written.

func (*IndexView) Sequential 

func (ix *IndexView) Sequential()

Sequential sets indexes to sequential row-wise indexes into table

func (*IndexView) SetTable 

func (ix *IndexView) SetTable(et *Table)

SetTable sets as indexes into given table with sequential initial indexes

func (*IndexView) Sort 

func (ix *IndexView) Sort(lessFunc func(et *Table, i, j int) bool)

Sort sorts the indexes into our Table using given Less function. The Less function operates directly on row numbers into the Table as these row numbers have already been projected through the indexes.

func (*IndexView) SortCol 

func (ix *IndexView) SortCol(colIndex int, ascending bool)

SortCol sorts the indexes into our Table according to values in given column index, using either ascending or descending order. Only valid for 1-dimensional columns.

func (*IndexView) SortColName 

func (ix *IndexView) SortColName(colNm string, ascending bool) error

SortColName sorts the indexes into our Table according to values in given column name, using either ascending or descending order. Only valid for 1-dimensional columns. Returns error if column name not found.

func (*IndexView) SortColNames 

func (ix *IndexView) SortColNames(colNms []string, ascending bool) error

SortColNames sorts the indexes into our Table according to values in given column names, using either ascending or descending order. Only valid for 1-dimensional columns. Returns error if column name not found.

func (*IndexView) SortCols 

func (ix *IndexView) SortCols(colIndexes []int, ascending bool)

SortCols sorts the indexes into our Table according to values in given list of column indexes, using either ascending or descending order for all of the columns. Only valid for 1-dimensional columns.

func (*IndexView) SortIndexes 

func (ix *IndexView) SortIndexes()

SortIndexes sorts the indexes into our Table directly in numerical order, producing the native ordering, while preserving any filtering that might have occurred.

func (*IndexView) SortStable 

func (ix *IndexView) SortStable(lessFunc func(et *Table, i, j int) bool)

SortStable stably sorts the indexes into our Table using given Less function. The Less function operates directly on row numbers into the Table as these row numbers have already been projected through the indexes. It is *essential* that it always returns false when the two are equal for the stable function to actually work.

func (*IndexView) SortStableCol 

func (ix *IndexView) SortStableCol(colIndex int, ascending bool)

SortStableCol sorts the indexes into our Table according to values in given column index, using either ascending or descending order. Only valid for 1-dimensional columns.

func (*IndexView) SortStableColName 

func (ix *IndexView) SortStableColName(colNm string, ascending bool) error

SortStableColName sorts the indexes into our Table according to values in given column name, using either ascending or descending order. Only valid for 1-dimensional columns. Returns error if column name not found.

func (*IndexView) SortStableColNames 

func (ix *IndexView) SortStableColNames(colNms []string, ascending bool) error

SortStableColNames sorts the indexes into our Table according to values in given column names, using either ascending or descending order. Only valid for 1-dimensional columns. Returns error if column name not found.

func (*IndexView) SortStableCols 

func (ix *IndexView) SortStableCols(colIndexes []int, ascending bool)

SortStableCols sorts the indexes into our Table according to values in given list of column indexes, using either ascending or descending order for all of the columns. Only valid for 1-dimensional columns.

func (*IndexView) Swap 

func (ix *IndexView) Swap(i, j int)

Swap switches the indexes for i and j

func (*IndexView) WriteCSV 

func (ix *IndexView) WriteCSV(w io.Writer, delim Delims, headers bool) error

WriteCSV writes only rows in table idx view to a comma-separated-values (CSV) file (where comma = any delimiter, specified in the delim arg). If headers = true then generate C++ emergent-style column headers. These headers have full configuration information for the tensor columns. Otherwise, only the data is written.

type LessFunc 

type LessFunc func(et *Table, i, j int) bool

LessFunc is a function used for sort comparisons that returns true if Table row i is less than Table row j -- these are the raw row numbers, which have already been projected through indexes when used for sorting via Indexes.

type Schema 

type Schema []Column

Schema specifies all of the columns of a table, sufficient to create the table. It is just a slice list of Columns

func SchemaFromEmerHeaders 

func SchemaFromEmerHeaders(hdrs []string) (Schema, error)

SchemaFromEmerHeaders attempts to configure a Table Schema based on emergent DataTable headers

func SchemaFromHeaders 

func SchemaFromHeaders(hdrs []string, rec [][]string) (Schema, error)

SchemaFromHeaders attempts to configure a Table Schema based on the headers for non-Emergent headers, data is examined to

func SchemaFromPlainHeaders 

func SchemaFromPlainHeaders(hdrs []string, rec [][]string) (Schema, error)

SchemaFromPlainHeaders configures a Table Schema based on plain headers. All columns are of type String and must be converted later to numerical types as appropriate.

type SplitAgg 

type SplitAgg struct {

	// the name of the aggregation operation performed, e.g., Sum, Mean, etc
	Name string

	// column index on which the aggregation was performed -- results will have same shape as cells in this column
	ColIndex int

	// aggregation results -- outer index is length of splits, inner is the length of the cell shape for the column
	Aggs [][]float64
}

SplitAgg contains aggregation results for splits

func (*SplitAgg) Clone 

func (sa *SplitAgg) Clone() *SplitAgg

Clone returns a cloned copy of our SplitAgg

func (*SplitAgg) CopyFrom 

func (sa *SplitAgg) CopyFrom(osa *SplitAgg)

CopyFrom copies from other SplitAgg -- we get our own unique copy of everything

type Splits 

type Splits struct {

	// the list of index views for each split
	Splits []*IndexView

	// levels of indexes used to organize the splits -- each split contains the full outer product across these index levels.  for example, if the split was generated by grouping over column values, then these are the column names in order of grouping.  the splits are not automatically sorted hierarchically by these levels but e.g., the GroupBy method produces that result -- use the Sort methods to explicitly sort.
	Levels []string

	// the values of the index levels associated with each split.  The outer dimension is the same length as Splits, and the inner dimension is the levels.
	Values [][]string

	// aggregate results, one for each aggregation operation performed -- split-level data is contained within each SplitAgg struct -- deleting a split removes these aggs but adding new splits just invalidates all existing aggs (they are automatically deleted).
	Aggs []*SplitAgg
	// contains filtered or unexported fields
}

Splits is a list of indexed views into a given Table, that represent a particular way of splitting up the data, e.g., whenever a given column value changes.

It is functionally equivalent to the MultiIndex in python's pandas: it has multiple levels of indexes as listed in the Levels field, which then have corresponding Values for each split. These index levels can be re-ordered, and new Splits or IndexViews's can be created from subsets of the existing levels. The Values are stored simply as string values, as this is the most general type and often index values are labels etc.

For Splits created by the splits.GroupBy function for example, each index Level is the column name that the data was grouped by, and the Values for each split are then the values of those columns. However, any arbitrary set of levels and values can be used, e.g., as in the splits.GroupByFunc function.

Conceptually, a given Split always contains the full "outer product" of all the index levels -- there is one split for each unique combination of values along each index level. Thus, removing one level collapses across those values and moves the corresponding indexes into the remaining split indexes.

You can Sort and Filter based on the index values directly, to reorganize the splits and drop particular index values, etc.

Splits also maintains Aggs aggregate values for each split, which can be computed using standard aggregation methods over data columns, using the split.Agg* functions.

The etable code contains the structural methods for managing the Splits data. See split package for end-user methods to generate different kinds of splits, and perform aggregations, etc.

func (*Splits) AddAgg 

func (spl *Splits) AddAgg(name string, colIndex int) *SplitAgg

AddAgg adds a new set of aggregation results for the Splits

func (*Splits) AggByColName 

func (spl *Splits) AggByColName(name string) *SplitAgg

AggByColName returns Agg results for given column name, optionally including :Name agg name appended, where Name is the name given to the Agg result (e.g., Mean for a standard Mean agg). Returns nil if not found. See also Try version for error message.

func (*Splits) AggByColNameTry 

func (spl *Splits) AggByColNameTry(name string) (*SplitAgg, error)

AggByColNameTry returns Agg results for given column name, optionally including :Name agg name appended, where Name is the name given to the Agg result (e.g., Mean for a standard Mean agg). Returns error message if not found.

func (*Splits) AggByName 

func (spl *Splits) AggByName(name string) *SplitAgg

AggByName returns Agg results for given name, which does NOT include the column name, just the name given to the Agg result (e.g., Mean for a standard Mean agg). See also AggByColName. Returns nil if not found. See also Try version for error message.

func (*Splits) AggByNameTry 

func (spl *Splits) AggByNameTry(name string) (*SplitAgg, error)

AggByNameTry returns Agg results for given name, which does NOT include the column name, just the name given to the Agg result (e.g., Mean for a standard Mean agg). See also AggByColName. Returns error message if not found.

func (*Splits) AggsToTable 

func (spl *Splits) AggsToTable(colName bool) *Table

AggsToTable returns a Table containing this Splits' aggregate data. Must have Levels and Aggs all created as in the split.Agg* methods. if colName == ColNameOnly, then the name of the columns for the Table is just the corresponding agg column name -- otherwise it also includes the name of the aggregation function with a : divider (e.g., Name:Mean)

func (*Splits) AggsToTableCopy 

func (spl *Splits) AggsToTableCopy(colName bool) *Table

AggsToTableCopy returns a Table containing this Splits' aggregate data and a copy of the first row of data for each split for all non-agg cols, which is useful for recording other data that goes along with aggregated values. Must have Levels and Aggs all created as in the split.Agg* methods. if colName == ColNameOnly, then the name of the columns for the Table is just the corresponding agg column name -- otherwise it also includes the name of the aggregation function with a : divider (e.g., Name:Mean)

func (*Splits) ByValue 

func (spl *Splits) ByValue(values []string) []int

ByValue finds split indexes by matching to split values, returns nil if not found. values are used in order as far as they go and any remaining values are assumed to match, and any empty values will match anything. Can use this to access different subgroups within overall set of splits.

func (*Splits) Clone 

func (spl *Splits) Clone() *Splits

Clone returns a cloned copy of our splits

func (*Splits) CopyFrom 

func (spl *Splits) CopyFrom(osp *Splits)

CopyFrom copies from other Splits -- we get our own unique copy of everything

func (*Splits) Delete 

func (spl *Splits) Delete(idx int)

Delete deletes split at given index -- use this to coordinate deletion of Splits, Values, and Aggs values for given split

func (*Splits) DeleteAggs 

func (spl *Splits) DeleteAggs()

DeleteAggs deletes all existing aggregation data

func (*Splits) ExtractLevels 

func (spl *Splits) ExtractLevels(levels []int) (*Splits, error)

ExtractLevels returns a new Splits that only has the given levels of indexes, in their given order, with the other levels removed and their corresponding indexes merged into the appropriate remaining levels. Any existing aggregation data is not retained in the new splits.

func (*Splits) Filter 

func (spl *Splits) Filter(fun func(idx int) bool)

Filter removes any split for which given function returns false

func (*Splits) Len 

func (spl *Splits) Len() int

Len returns number of splits

func (*Splits) Less 

func (spl *Splits) Less(i, j int) bool

Less calls the LessFunc for sorting

func (*Splits) New 

func (spl *Splits) New(dt *Table, values []string, rows ...int) *IndexView

New adds a new split to the list for given table, and with associated values, which are copied before saving into Values list, and any number of rows from the table associated with this split (also copied). Any existing Aggs are deleted by this.

func (*Splits) ReorderLevels 

func (spl *Splits) ReorderLevels(order []int) error

ReorderLevels re-orders the index levels according to the given new ordering indexes e.g., []int{1,0} will move the current level 0 to level 1, and 1 to level 0 no checking is done to ensure these are sensible beyond basic length test -- behavior undefined if so. Typically you want to call SortLevels after this.

func (*Splits) SetLevels 

func (spl *Splits) SetLevels(levels ...string)

SetLevels sets the Levels index names -- must match actual index dimensionality of the Values. This is automatically done by e.g., GroupBy, but must be done manually if creating custom indexes.

func (*Splits) Sort 

func (spl *Splits) Sort(lessFunc func(spl *Splits, i, j int) bool)

Sort sorts the splits according to the given Less function.

func (*Splits) SortLevels 

func (spl *Splits) SortLevels()

SortLevels sorts the splits according to the current index level ordering of values i.e., first index level is outer sort dimension, then within that is the next, etc

func (*Splits) SortOrder 

func (spl *Splits) SortOrder(order []int) error

SortOrder sorts the splits according to the given ordering of index levels which can be a subset as well

func (*Splits) Swap 

func (spl *Splits) Swap(i, j int)

Swap switches the indexes for i and j

func (*Splits) Table 

func (spl *Splits) Table() *Table

Table returns the table from the first split (should be same for all) returns nil if no splits yet

type SplitsLessFunc 

type SplitsLessFunc func(spl *Splits, i, j int) bool

SplitsLessFunc is a function used for sort comparisons that returns true if split i is less than split j

type Table 

type Table struct {

	// columns of data, as etensor.Tensor tensors
	Cols []etensor.Tensor `view:"no-inline"`

	// the names of the columns
	ColNames []string

	// number of rows, which is enforced to be the size of the outer-most dimension of the column tensors
	Rows int `edit:"-"`

	// the map of column names to column numbers
	ColNameMap map[string]int `view:"-"`

	// misc meta data for the table.  We use lower-case key names following the struct tag convention:  name = name of table; desc = description; read-only = gui is read-only; precision = n for precision to write out floats in csv.  For Column-specific data, we look for ColName: prefix, specifically ColName:desc = description of the column contents, which is shown as tooltip in the etview.TableView, and :width for width of a column
	MetaData map[string]string
}

etable.Table is the emer DataTable structure, containing columns of etensor tensors. All tensors MUST have RowMajor stride layout!

func New 

func New(sc Schema, rows int) *Table

New returns a new Table constructed from given Schema. The actual tensor number of rows is enforced to be > 0, because we cannot have a null dimension in tensor shape

func NewTable 

func NewTable(name string) *Table

func (*Table) AddCol 

func (dt *Table) AddCol(tsr etensor.Tensor, name string) error

AddCol adds the given tensor as a column to the table. returns error if it is not a RowMajor organized tensor, and automatically adjusts the shape to fit the current number of rows.

func (*Table) AddRows 

func (dt *Table) AddRows(n int)

AddRows adds n rows to each of the columns

func (*Table) AppendRows 

func (dt *Table) AppendRows(dt2 *Table)

AppendRows appends shared columns in both tables with input table rows

func (*Table) CellFloat 

func (dt *Table) CellFloat(colNm string, row int) float64

CellFloat returns the float64 value of cell at given column (by name), row index for columns that have 1-dimensional tensors. Returns NaN if column is not a 1-dimensional tensor or col name not found, or row not valid.

func (*Table) CellFloatIndex 

func (dt *Table) CellFloatIndex(col, row int) float64

CellFloatIndex returns the float64 value of cell at given column, row index for columns that have 1-dimensional tensors. Returns NaN if column is not a 1-dimensional tensor or row not valid.

func (*Table) CellFloatTry 

func (dt *Table) CellFloatTry(colNm string, row int) (float64, error)

CellFloatTry returns the float64 value of cell at given column (by name), row index for columns that have 1-dimensional tensors. Returns an error if column not found, or column is not a 1-dimensional tensor, or row not valid.

func (*Table) CellString 

func (dt *Table) CellString(colNm string, row int) string

CellString returns the string value of cell at given column (by name), row index for columns that have 1-dimensional tensors. Returns "" if column is not a 1-dimensional tensor or row not valid.

func (*Table) CellStringIndex 

func (dt *Table) CellStringIndex(col, row int) string

CellStringIndex returns the string value of cell at given column, row index for columns that have 1-dimensional tensors. Returns "" if column is not a 1-dimensional tensor or row not valid.

func (*Table) CellStringTry 

func (dt *Table) CellStringTry(colNm string, row int) (string, error)

CellStringTry returns the string value of cell at given column (by name), row index for columns that have 1-dimensional tensors. Returns an error if column not found, or column is not a 1-dimensional tensor, or row not valid.

func (*Table) CellTensor 

func (dt *Table) CellTensor(colNm string, row int) etensor.Tensor

CellTensor returns the tensor SubSpace for given column (by name), row index for columns that have higher-dimensional tensors so each row is represented by an n-1 dimensional tensor, with the outer dimension being the row number. Returns nil on any error -- see Try version for error returns.

func (*Table) CellTensorFloat1D 

func (dt *Table) CellTensorFloat1D(colNm string, row int, idx int) float64

CellTensorFloat1D returns the float value of a Tensor cell's cell at given 1D offset within cell, for given column (by name), row index for columns that have higher-dimensional tensors so each row is represented by an n-1 dimensional tensor, with the outer dimension being the row number. Returns 0 on any error -- see Try version for error returns.

func (*Table) CellTensorFloat1DTry 

func (dt *Table) CellTensorFloat1DTry(colNm string, row int, idx int) (float64, error)

CellTensorFloat1DTry returns the float value of a Tensor cell's cell at given 1D offset within cell, for given column (by name), row index for columns that have higher-dimensional tensors so each row is represented by an n-1 dimensional tensor, with the outer dimension being the row number. Returns any error.

func (*Table) CellTensorIndex 

func (dt *Table) CellTensorIndex(col, row int) etensor.Tensor

CellTensorIndex returns the tensor SubSpace for given column, row index for columns that have higher-dimensional tensors so each row is represented by an n-1 dimensional tensor, with the outer dimension being the row number. Returns nil if column is a 1-dimensional tensor or there is any error from the etensor.Tensor.SubSpace call.

func (*Table) CellTensorTry 

func (dt *Table) CellTensorTry(colNm string, row int) (etensor.Tensor, error)

CellTensorTry returns the tensor SubSpace for given column (by name), row index for columns that have higher-dimensional tensors so each row is represented by an n-1 dimensional tensor, with the outer dimension being the row number. Returns an error if column is a 1-dimensional tensor or any error from the etensor.Tensor.SubSpace call.

func (*Table) Clone 

func (dt *Table) Clone() *Table

Clone returns a complete copy of this table

func (*Table) Col 

func (dt *Table) Col(i int) etensor.Tensor

Col returns the tensor at given column index

func (*Table) ColByName 

func (dt *Table) ColByName(name string) etensor.Tensor

ColByName returns the tensor at given column name without any error messages -- just returns nil if not found

func (*Table) ColByNameTry 

func (dt *Table) ColByNameTry(name string) (etensor.Tensor, error)

ColByNameTry returns the tensor at given column name, if not found, returns error

func (*Table) ColIndex 

func (dt *Table) ColIndex(name string) int

ColIndex returns the index of the given column name. returns -1 if name not found -- see Try version for error message.

func (*Table) ColIndexTry 

func (dt *Table) ColIndexTry(name string) (int, error)

ColIndexTry returns the index of the given column name, along with an error if not found.

func (*Table) ColIndexesByNames 

func (dt *Table) ColIndexesByNames(names []string) []int

ColIndexesByNames returns the indexes of the given column names. idxs have -1 if name not found -- see Try version for error message.

func (*Table) ColIndexesByNamesTry 

func (dt *Table) ColIndexesByNamesTry(names []string) ([]int, error)

ColsIndexesByNamesTry returns the indexes of the given column names, along with an error if any not found.

func (*Table) ColName 

func (dt *Table) ColName(i int) string

ColName returns the name of given column

func (*Table) CopyCell 

func (dt *Table) CopyCell(colNm string, row int, cpt *Table, cpColNm string, cpRow int) error

CopyCell copies into cell at given col, row from cell in other table. It is robust to differences in type -- uses destination cell type. Returns error if column names are invalid.

func (*Table) CopyMetaDataFrom 

func (dt *Table) CopyMetaDataFrom(cp *Table)

CopyMetaDataFrom copies meta data from other table

func (*Table) DeleteAll 

func (dt *Table) DeleteAll()

DeleteAll deletes all columns -- full reset

func (*Table) DeleteColIndex 

func (dt *Table) DeleteColIndex(idx int)

DeleteColIndex deletes column of given index

func (*Table) DeleteColName 

func (dt *Table) DeleteColName(name string) error

DeleteColName deletes column of given name.

func (*Table) EmerHeaders 

func (dt *Table) EmerHeaders() []string

EmerHeaders generates emergent DataTable header strings from the table. These have full information about type and tensor cell dimensionality.

func (*Table) IsValidRow 

func (dt *Table) IsValidRow(row int) bool

IsValidRow returns true if the row is valid

func (*Table) IsValidRowTry 

func (dt *Table) IsValidRowTry(row int) error

IsValidRowTry returns an error message if the row is not valid.

func (*Table) NumCols 

func (dt *Table) NumCols() int

NumCols returns the number of columns (arrow / dframe api)

func (*Table) NumRows 

func (dt *Table) NumRows() int

NumRows returns the number of rows (arrow / dframe api)

func (*Table) OpenCSV 

func (dt *Table) OpenCSV(filename core.Filename, delim Delims) error

OpenCSV reads a table from a comma-separated-values (CSV) file (where comma = any delimiter, specified in the delim arg), using the Go standard encoding/csv reader conforming to the official CSV standard. If the table does not currently have any columns, the first row of the file is assumed to be headers, and columns are constructed therefrom. The C++ emergent column headers are parsed -- these have full configuration information for tensor dimensionality. If the table DOES have existing columns, then those are used robustly for whatever information fits from each row of the file.

func (*Table) OpenFS 

func (dt *Table) OpenFS(fsys fs.FS, filename string, delim Delims) error

OpenFS is the version of Table.OpenCSV that uses an fs.FS filesystem.

func (*Table) ReadCSV 

func (dt *Table) ReadCSV(r io.Reader, delim Delims) error

ReadCSV reads a table from a comma-separated-values (CSV) file (where comma = any delimiter, specified in the delim arg), using the Go standard encoding/csv reader conforming to the official CSV standard. If the table does not currently have any columns, the first row of the file is assumed to be headers, and columns are constructed therefrom. The C++ emergent column headers are parsed -- these have full configuration information for tensor dimensionality. If the table DOES have existing columns, then those are used robustly for whatever information fits from each row of the file.

func (*Table) ReadCSVRow 

func (dt *Table) ReadCSVRow(rec []string, row int)

ReadCSVRow reads a record of CSV data into given row in table

func (*Table) RowsByString 

func (dt *Table) RowsByString(colNm string, str string, contains, ignoreCase bool) []int

RowsByString returns the list of rows that have given string value in given column name. returns nil if name invalid -- see also Try. if contains, only checks if row contains string; if ignoreCase, ignores case. Use named args for greater clarity.

func (*Table) RowsByStringIndex 

func (dt *Table) RowsByStringIndex(colIndex int, str string, contains, ignoreCase bool) []int

RowsByStringIndex returns the list of rows that have given string value in given column index. if contains, only checks if row contains string; if ignoreCase, ignores case. Use named args for greater clarity.

func (*Table) RowsByStringTry 

func (dt *Table) RowsByStringTry(colNm string, str string, contains, ignoreCase bool) ([]int, error)

RowsByStringTry returns the list of rows that have given string value in given column name. returns error message for invalid column name. if contains, only checks if row contains string; if ignoreCase, ignores case. Use named args for greater clarity.

func (*Table) SaveCSV 

func (dt *Table) SaveCSV(filename core.Filename, delim Delims, headers bool) error

SaveCSV writes a table to a comma-separated-values (CSV) file (where comma = any delimiter, specified in the delim arg). If headers = true then generate C++ emergent-tyle column headers. These headers have full configuration information for the tensor columns. Otherwise, only the data is written.

func (*Table) Schema 

func (dt *Table) Schema() Schema

Schema returns the Schema (column properties) for this table

func (*Table) SetCellFloat 

func (dt *Table) SetCellFloat(colNm string, row int, val float64) bool

SetCellFloat sets the float64 value of cell at given column (by name), row index for columns that have 1-dimensional tensors.

func (*Table) SetCellFloatIndex 

func (dt *Table) SetCellFloatIndex(col, row int, val float64) bool

SetCellFloatIndex sets the float64 value of cell at given column, row index for columns that have 1-dimensional tensors. Returns true if set.

func (*Table) SetCellFloatTry 

func (dt *Table) SetCellFloatTry(colNm string, row int, val float64) error

SetCellFloatTry sets the float64 value of cell at given column (by name), row index for columns that have 1-dimensional tensors. Returns an error if column not found, or column is not a 1-dimensional tensor.

func (*Table) SetCellString 

func (dt *Table) SetCellString(colNm string, row int, val string) bool

SetCellString sets the string value of cell at given column (by name), row index for columns that have 1-dimensional tensors. Returns true if set.

func (*Table) SetCellStringIndex 

func (dt *Table) SetCellStringIndex(col, row int, val string) bool

SetCellStringIndex sets the string value of cell at given column, row index for columns that have 1-dimensional tensors. Returns true if set.

func (*Table) SetCellStringTry 

func (dt *Table) SetCellStringTry(colNm string, row int, val string) error

SetCellStringTry sets the string value of cell at given column (by name), row index for columns that have 1-dimensional tensors. Returns an error if column not found, or column is not a 1-dimensional tensor.

func (*Table) SetCellTensor 

func (dt *Table) SetCellTensor(colNm string, row int, val etensor.Tensor) bool

SetCellTensor sets the tensor value of cell at given column (by name), row index for columns that have n-dimensional tensors. Returns true if set.

func (*Table) SetCellTensorFloat1D 

func (dt *Table) SetCellTensorFloat1D(colNm string, row int, idx int, val float64) bool

SetCellTensorFloat1D sets the tensor cell's float cell value at given 1D index within cell, at given column (by name), row index for columns that have n-dimensional tensors. Returns true if set.

func (*Table) SetCellTensorFloat1DTry 

func (dt *Table) SetCellTensorFloat1DTry(colNm string, row int, idx int, val float64) error

SetCellTensorFloat1DTry sets the string value of cell at given column (by name), row index for columns that have 1-dimensional tensors. Returns an error if column not found, or column is not a 1-dimensional tensor.

func (*Table) SetCellTensorIndex 

func (dt *Table) SetCellTensorIndex(col, row int, val etensor.Tensor) bool

SetCellTensorIndex sets the tensor value of cell at given column, row index for columns that have n-dimensional tensors. Returns true if set.

func (*Table) SetCellTensorTry 

func (dt *Table) SetCellTensorTry(colNm string, row int, val etensor.Tensor) error

SetCellTensorTry sets the string value of cell at given column (by name), row index for columns that have 1-dimensional tensors. Returns an error if column not found, or column is not a 1-dimensional tensor.

func (*Table) SetFromSchema 

func (dt *Table) SetFromSchema(sc Schema, rows int)

SetFromSchema configures table from given Schema. The actual tensor number of rows is enforced to be > 0, because we cannot have a null dimension in tensor shape. does not preserve any existing columns / data.

func (*Table) SetMetaData 

func (dt *Table) SetMetaData(key, val string)

SetMetaData sets given meta-data key to given value, safely creating the map if not yet initialized. Standard Keys are: * name -- name of table * desc -- description of table * read-only -- makes gui read-only (inactive edits) for etview.TableView * ColName:* -- prefix for all column-specific meta-data

  • desc -- description of column

func (*Table) SetNumRows 

func (dt *Table) SetNumRows(rows int)

SetNumRows sets the number of rows in the table, across all columns if rows = 0 then effective number of rows in tensors is 1, as this dim cannot be 0

func (*Table) UpdateColNameMap 

func (dt *Table) UpdateColNameMap()

UpdateColNameMap updates the column name map

func (*Table) WriteCSV 

func (dt *Table) WriteCSV(w io.Writer, delim Delims, headers bool) error

WriteCSV writes a table to a comma-separated-values (CSV) file (where comma = any delimiter, specified in the delim arg). If headers = true then generate C++ emergent-style column headers. These headers have full configuration information for the tensor columns. Otherwise, only the data is written.

func (*Table) WriteCSVHeaders 

func (dt *Table) WriteCSVHeaders(w io.Writer, delim Delims) (int, error)

WriteCSVHeaders writes headers to a comma-separated-values (CSV) file (where comma = any delimiter, specified in the delim arg). Returns number of columns in header

func (*Table) WriteCSVRow 

func (dt *Table) WriteCSVRow(w io.Writer, row int, delim Delims) error

WriteCSVRow writes given row to a comma-separated-values (CSV) file (where comma = any delimiter, specified in the delim arg)

func (*Table) WriteCSVRowWriter 

func (dt *Table) WriteCSVRowWriter(cw *csv.Writer, row int, ncol int) error

WriteCSVRowWriter uses csv.Writer to write one row

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.