feature

type CountVectorizer ¶

type CountVectorizer struct {
	Mapping   map[string]uint // word to index
	Separator string          // default space
}

CountVectorizer performs bag of words encoding of text.

Separator should not be a part of any word. Responsibility to ensure this is on caller. Words that have separator as its substring will be ommited.

Mapping should contain all values from 0 to N where N is len(Mapping). Responsibility to ensure this is on caller. If some index is higher than N or lower than 0, then code will panic. If some index is not set, then that index will be skipped. If some index is set twice, then index will have sum of words.

func (t *CountVectorizer) FeatureNames() []string

func (t *CountVectorizer) Fit(vals []string)

func (t *CountVectorizer) NumFeatures() int

func (t *CountVectorizer) Transform(v string) []float64

func (t *CountVectorizer) TransformInplace(dest []float64, v string)

type Identity ¶

type Identity struct{}

Identity is a transformer that returns unmodified input value

func (t *Identity) Fit(_ []float64)

func (t *Identity) Transform(v float64) float64

type KBinsDiscretizer ¶

type KBinsDiscretizer struct {
	QuantileScaler
}

KBinsDiscretizer based on quantile strategy

func (t *KBinsDiscretizer) Fit(vals []float64)

func (t *KBinsDiscretizer) Transform(v float64) float64

type MaxAbsScaler ¶

type MaxAbsScaler struct {
	Max float64
}

MaxAbsScaler transforms value into -1 to +1 range linearly

func (t *MaxAbsScaler) Fit(vals []float64)

func (t *MaxAbsScaler) Transform(v float64) float64

type MinMaxScaler ¶

type MinMaxScaler struct {
	Min float64
	Max float64
}

MinMaxScaler is a transformer that rescales value into range between min and max

func (t *MinMaxScaler) Fit(vals []float64)

func (t *MinMaxScaler) Transform(v float64) float64

type OneHotEncoder ¶

type OneHotEncoder struct {
	Mapping map[string]uint // word to index
}

OneHotEncoder encodes string value to corresponding index

Mapping should contain all values from 0 to N where N is len(Mapping). Responsibility to ensure this is on caller. If some index is higher than N or lower than 0, then code will panic. If some index is not set, then that index will be skipped. If some index is set twice, then index will have effect of either of words.

func (t *OneHotEncoder) FeatureNames() []string

func (t *OneHotEncoder) Fit(vs []string)

func (t *OneHotEncoder) NumFeatures() int

func (t *OneHotEncoder) Transform(v string) []float64

func (t *OneHotEncoder) TransformInplace(dest []float64, v string)

type OrdinalEncoder ¶

type OrdinalEncoder struct {
	Mapping map[string]uint
}

OrdinalEncoder returns 0 for string that is not found, or else a number for that string

Mapping should contain all values from 0 to N where N is len(Mapping). Responsibility to ensure this is on caller. If some index is higher than N or lower than 0, then code will panic. If some index is not set, then that index will be skipped. If some index is set twice, then index will have effect of either of words.

func (t *OrdinalEncoder) Fit(vals []string)

func (t *OrdinalEncoder) Transform(v string) float64

type QuantileScaler ¶

type QuantileScaler struct {
	Quantiles []float64
}

QuantileScaler transforms any distribution to uniform distribution This is done by mapping values to quantiles they belong to.

func (t *QuantileScaler) Fit(vals []float64)

func (t *QuantileScaler) Transform(v float64) float64

type SampleNormalizerL1 ¶

type SampleNormalizerL1 struct{}

SampleNormalizerL1 transforms features for single sample to have norm L1=1

func (t *SampleNormalizerL1) Fit(_ []float64)

func (t *SampleNormalizerL1) Transform(vs []float64) []float64

func (t *SampleNormalizerL1) TransformInplace(dest []float64, vs []float64)

type SampleNormalizerL2 ¶

type SampleNormalizerL2 struct{}

SampleNormalizerL2 transforms features for single sample to have norm L2=1

func (t *SampleNormalizerL2) Fit(_ []float64)

func (t *SampleNormalizerL2) Transform(vs []float64) []float64

func (t *SampleNormalizerL2) TransformInplace(dest []float64, vs []float64)

type StandardScaler ¶

type StandardScaler struct {
	Mean float64
	STD  float64
}

StandardScaler transforms feature into normal standard distribution.

func (t *StandardScaler) Fit(vals []float64)

func (t *StandardScaler) Transform(v float64) float64

type StructTransformer ¶

type StructTransformer struct {
	Transformers []interface{}
}

StructTransformer uses reflection to encode struct into feature vector. It uses struct tags to create feature transformers for each field. Since it is using reflection, there is a slight overhead for large structs, which can be seen in benchmarks. For better performance, use codegen version for your struct, refer to README of this repo.

func (s *StructTransformer) Fit(_ []interface{})

func (s *StructTransformer) Transform(v interface{}) []float64

type TFIDFVectorizer ¶

type TFIDFVectorizer struct {
	CountVectorizer
	DocCount     []uint // number of documents where i-th word from CountVectorizer appeared in
	NumDocuments int
	Normalizer   SampleNormalizerL2
}

TFIDFVectorizer performs tf-idf vectorization on top of count vectorization. Based on: https://scikit-learn.org/stable/modules/feature_extraction.html Using non-smooth version, adding 1 to log instead of denominator in idf.

DocCount should have len of len(CountVectorizer.Mapping). It is responsibility of a caller to sensure it is so.

func (t *TFIDFVectorizer) FeatureNames() []string

func (t *TFIDFVectorizer) Fit(vals []string)

func (t *TFIDFVectorizer) NumFeatures() int

func (t *TFIDFVectorizer) Transform(v string) []float64

func (t *TFIDFVectorizer) TransformInplace(dest []float64, v string)