linear

package
v0.0.0-...-00e0c84 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 15, 2022 License: MIT Imports: 8 Imported by: 9

README

Generalized Linear Models

import "github.com/cdipaolo/goml/linear"

GoDoc

This part of the goml package implements varied generalized linear models using gradient descent (currently, though more options for optimization methods might be available in the future.)

implemented models

Linear Least Squares Regression Logistic Regression Classification (Color is Ground Truth Class)
Linear Least Squares Regression Results Logistic Regression Results

example ordinary least squares

this is mostly from from the linear_test.go tests. You can find more examples from the testing files. The line given is z = 10 + (x/10) + (y/5)

// initialize data
threeDLineX = [][]float64{}
threeDLineY = []float64{}
// the line z = 10 + (x/10) + (y/5)
for i := -10; i < 10; i++ {
    for j := -10; j < 10; j++ {
        threeDLineX = append(threeDLineX, []float64{float64(i), float64(j)})
        threeDLineY = append(threeDLineY, 10+float64(i)/10+float64(j)/5)
    }
}

// initialize model
//
// use optimization method of Stochastic Gradient Ascent
// use α (learning rate) = .0001 / 1e-4
// use λ (regularization term) = 13.06
// set the max iteration cap for gradient
//     descent to be 1000/1e3 iterations
// and finally pass in the data
model := linear.NewLeastSquares(base.StochasticGA, 1e-4, 13.06, 1e3, threeDLineX, threeDLineY)

// learn
err = model.Learn()
if err != nil {
    panic("There was some error learning")
}

// predict based on model
guess, err = model.Predict([]float64{12.016, 6.523})
if err != nil {
    panic("There was some error in the prediction")
}

// persist the model to disk
//
// path to file will be '/tmp/.goml/LeastSquares'
// and it stores the parameter vector θ as a JSON
// array
err = model.PersistToFile("/tmp/.goml/LeastSquares")
if err != nil {
    panic("There was some error persisting the model to a file!")
}

// restore the model from file
//
// note that you could have a file with a JSON
// array of floats from some other tool and it
// would import correctly as well
err = model.RestoreFromFile("/tmp/.goml/LeastSquares")
if err != nil {
    panic("There was some error persisting the model to a file!")
}

gradient descent optimization

Here's some data relating the cost function J(θ) and the number of iterations of the data using a 3d model. Note that in this case the data modeled off of was perfectly linear, so obviously the cost function wouldn't and shouldn't bottom out at 0.000... for real world data!

Nice Looking Graph!

Documentation

Overview

Package linear implements commonly used General Linear Models.

https://en.wikipedia.org/wiki/General_linear_model

Models implemented as of yet include:

  • Ordinary Least Squares
  • Logistic Regression

General Usage: Find the model you want to use. Then find the 'NewXXXXXX' function, such as 

func NewLeastSquares(method base.OptimizationMethod, alpha, regularization float64, maxIterations int, trainingSet [][]float64, expectedResults []float64) *LeastSquares

load in the given parameters, then run 

func Learn() error

Now you can predict off of the model! 

func Predict([]float64) ([]float64, error)

Full example assuming testX is of type [][]float64 and testY is of type []float64 where there are 2 features being inputted (ie. the size of a house and the number of bedrooms being given as x[0] and x[1], respectively.) textY[i] should be the observed result of the inputs testX[i].:

Example Model Usage (Batch Ordinary Least Squares): 

// optimization method: Batch Gradient Ascent
// Learning rate: 1e-4
// Regulatization term: 6
// Max Iterations: 800
// Dataset to learn fron: testX
// Expected results dataset: testY
model := NewLeastSquares(base.BatchGA, 1e-4, 6, 800, testX, testY)

err := model.Learn()
if err != nil {
    panic("SOME ERROR!! RUN!")
}

// now I want to predict off of this
// Ordinary Least Squares model!
guess, err = model.Predict([]float64{10000,6})
if err != nil {
    panic("AAAARGGGH! SHIVER ME TIMBERS! THESE ROTTEN SCOUNDRELS FOUND AN ERROR!!!")
}

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type LeastSquares 

type LeastSquares struct {
	Parameters []float64 `json:"theta"`

	// Output is the io.Writer used for logging
	// and printing. Defaults to os.Stdout.
	Output io.Writer
	// contains filtered or unexported fields
}

LeastSquares implements a standard linear regression model with a Least Squares cost function.

https://en.wikipedia.org/wiki/Least_squares

The model uses gradient descent, NOT regular equations.

func NewLeastSquares 

func NewLeastSquares(method base.OptimizationMethod, alpha, regularization float64, maxIterations int, trainingSet [][]float64, expectedResults []float64, features ...int) *LeastSquares

NewLeastSquares returns a pointer to the linear model initialized with the learning rate alpha, the training set trainingSet, and the expected results upon which to use the dataset to train, expectedResults.

if you're passing in no training set directly because you want to learn using the online method then just declare the number of features (it's an integer) as an extra arg after the rest of the arguments

Example Least Squares (Stochastic GA): 

// optimization method: Stochastic Gradient Ascent
// Learning rate: 1e-4
// Regularization term: 6
// Max Iterations: 800
// Dataset to learn from: testX
// Expected results dataset: testY
model := NewLeastSquares(base.StochasticGA, 1e-4, 6, 800, testX, testY)

err := model.Learn()
if err != nil {
    panic("SOME ERROR!! RUN!")
}

// now I want to predict off of this
// Ordinary Least Squares model!
guess, err = model.Predict([]float64{10000,6})
if err != nil {
    panic("AAAARGGGH! SHIVER ME TIMBERS! THESE ROTTEN SCOUNDRELS FOUND AN ERROR!!!")
}

func (*LeastSquares) Dij 

func (l *LeastSquares) Dij(i int, j int) (float64, error)

Dij returns the derivative of the cost function J(θ) with respect to the j-th parameter of the hypothesis, θ[j], for the training example x[i]. Used in Stochastic Gradient Descent.

assumes that i,j is within the bounds of the data they are looking up! (because this is getting called so much, it needs to be efficient with comparisons)

func (*LeastSquares) Dj 

func (l *LeastSquares) Dj(j int) (float64, error)

Dj returns the partial derivative of the cost function J(θ) with respect to theta[j] where theta is the parameter vector associated with our hypothesis function Predict (upon which we are optimizing

func (*LeastSquares) Examples 

func (l *LeastSquares) Examples() int

Examples returns the number of training examples (m) that the model currently is training from.

func (*LeastSquares) J 

func (l *LeastSquares) J() (float64, error)

J returns the Least Squares cost function of the given linear model. Could be useful in testing convergence

func (*LeastSquares) Learn 

func (l *LeastSquares) Learn() error

Learn takes the struct's dataset and expected results and runs batch gradient descent on them, optimizing theta so you can predict based on those results

func (*LeastSquares) LearningRate 

func (l *LeastSquares) LearningRate() float64

LearningRate returns the learning rate α for gradient descent to optimize the model. Could vary as a function of something else later, potentially.

func (*LeastSquares) MaxIterations 

func (l *LeastSquares) MaxIterations() int

MaxIterations returns the number of maximum iterations the model will go through in GradientAscent, in the worst case

func (*LeastSquares) OnlineLearn 

func (l *LeastSquares) OnlineLearn(errors chan error, dataset chan base.Datapoint, onUpdate func([][]float64), normalize ...bool)

OnlineLearn runs similar to using a fixed dataset with Stochastic Gradient Descent, but it handles data by passing it as a channel, and returns errors through a channel, which lets it run responsive to inputted data from outside the model itself (like using data from the stock market at timed intervals or using realtime data about the weather.)

The onUpdate callback is called whenever the parameter vector theta is changed, so you are able to persist the model with the most up to date vector at all times (you could persist to a database within the callback, for example.) Don't worry about it taking too long and blocking, because the callback is spawned into another goroutine.

NOTE that this function is suggested to run in it's own goroutine, or at least is designed as such.

NOTE part 2: You can pass in an empty dataset, so long as it's not nil, and start pushing after.

NOTE part 3: each example is only looked at as it goes through the channel, so if you want to have each example looked at more than once you must manually pass the data yourself.

NOTE part 4: the optional parameter 'normalize' will not do anything with a linear model.It is included so the model fits the OnlineModel interface.

Example Online Linear Least Squares: 

// create the channel of data and errors
stream := make(chan base.Datapoint, 100)
errors := make(chan error)

// notice how we are adding another integer
// to the end of the NewLogistic call. This
// tells the model to use that number of features
// (4) in leu of finding that from the dataset
// like you would with batch/stochastic GD
//
// Also – the 'base.StochasticGA' doesn't affect
// anything. You could put batch.
model := NewLeastSquares(base.StochasticGA, .0001, 0, 0, nil, nil, 4)

go model.OnlineLearn(errors, stream, func(theta [][]float64) {
    // do something with the new theta (persist
    // to database?) in here.
})

go func() {
    for iterations := 0; iterations < 20; iterations++ {
        for i := -200.0; abs(i) > 1; i *= -0.75 {
            for j := -200.0; abs(j) > 1; j *= -0.75 {
                for k := -200.0; abs(k) > 1; k *= -0.75 {
                    for l := -200.0; abs(l) > 1; l *= -0.75 {
                         stream <- base.Datapoint{
                             X: []float64{i, j, k, l},
                             Y: []float64{i/2 + 2*k - 4*j + 2*l + 3},
                         }
                    }
                }
            }
        }
    }

  // close the dataset to tell the model
  // to stop learning when it finishes reading
  // what's left in the channel
  close(stream)
}()

// this will block until the error
// channel is closed in the learning
// function (it will, don't worry!)
for {
    err, more := <-errors
    if err != nil {
        panic("THERE WAS AN ERROR!!! RUN!!!!")
  }
    if !more {
        break
    }
}

// Below here all the learning is completed

// predict like usual
guess, err = model.Predict([]float64{42,6,10,-32})
if err != nil {
    panic("AAAARGGGH! SHIVER ME TIMBERS! THESE ROTTEN SCOUNDRELS FOUND AN ERROR!!!")
}

func (*LeastSquares) PersistToFile 

func (l *LeastSquares) PersistToFile(path string) error

PersistToFile takes in an absolute filepath and saves the parameter vector θ to the file, which can be restored later. The function will take paths from the current directory, but functions

The data is stored as JSON because it's one of the most efficient storage method (you only need one comma extra per feature + two brackets, total!) And it's extendable.

func (*LeastSquares) Predict 

func (l *LeastSquares) Predict(x []float64, normalize ...bool) ([]float64, error)

Predict takes in a variable x (an array of floats,) and finds the value of the hypothesis function given the current parameter vector θ

if normalize is given as true, then the input will first be normalized to unit length. Only use this if you trained off of normalized inputs and are feeding an un-normalized input

func (*LeastSquares) RestoreFromFile 

func (l *LeastSquares) RestoreFromFile(path string) error

RestoreFromFile takes in a path to a parameter vector theta and assigns the model it's operating on's parameter vector to that.

The path must ba an absolute path or a path from the current directory

This would be useful in persisting data between running a model on data, or for graphing a dataset with a fit in another framework like Julia/Gadfly.

func (*LeastSquares) String 

func (l *LeastSquares) String() string

String implements the fmt interface for clean printing. Here we're using it to print the model as the equation h(θ)=... where h is the linear hypothesis model

func (*LeastSquares) Theta 

func (l *LeastSquares) Theta() []float64

Theta returns the parameter vector θ for use in persisting the model, and optimizing the model through gradient descent ( or other methods like Newton's Method)

func (*LeastSquares) UpdateLearningRate 

func (l *LeastSquares) UpdateLearningRate(a float64)

UpdateLearningRate set's the learning rate of the model to the given float64.

func (*LeastSquares) UpdateTrainingSet 

func (l *LeastSquares) UpdateTrainingSet(trainingSet [][]float64, expectedResults []float64) error

UpdateTrainingSet takes in a new training set (variable x) as well as a new result set (y). This could be useful if you want to retrain a model starting with the parameter vector of a previous training session, but most of the time wouldn't be used.

type LocalLinear 

type LocalLinear struct {
	Parameters []float64 `json:"theta"`

	// Output is the io.Writer used for logging
	// and printing. Defaults to os.Stdout.
	Output io.Writer
	// contains filtered or unexported fields
}

LocalLinear implements a locally weighted linear least squares regression.

https://en.wikipedia.org/wiki/Least_squares http://cs229.stanford.edu/notes/cs229-notes1.pdf

Note that this is not modeled to work in an online setting, so the model does not implement that interface. Also, because a new hypothesis is needed for every point you try to predict, there is no 'Learn' function. Instead, when calling predict the model first learns from the data set with weights set with respect to the given input, then returns the trained hypothesis when evaluated at the given input.

While that may sound really inefficient, sometimes this model can perform well with little tweaking, especially if you're working with a smaller data set to train off of. Andrew Ng said in one of his Stanford CS229 lectures that Locally Weighted Linear Regression is one of his mentor's 'favourite' 'off-the-shelf' learning algorithm. Obviously model selection plays a large role in this.

NOTE that there is no file persistance of this model because you need to retrain at the time of every prediction anyway.

Example Locally Weighted Linear Regression Usage: 

x := [][]float64{}
y := []float64{}

// throw in some junk points which
// should be more-or-less ignored
// by the weighting
for i := -70.0; i < -65; i += 2 {
    for j := -70.0; j < -65; j += 2 {
        x = append(x, []float64{i, j})
        y = append(y, 20*(rand.Float64()-0.5))
    }
}
for i := 65.0; i < 70; i += 2 {
    for j := 65.0; j < 70; j += 2 {
        x = append(x, []float64{i, j})
        y = append(y, 20*(rand.Float64()-0.5))
    }
}

// put in some linear points
for i := -20.0; i < 20; i++ {
    for j := -20.0; j < 20; j++ {
        x = append(x, []float64{i, j})
        y = append(y, 5*i-5*j-10)
    }
}

model := NewLocalLinear(base.StochasticGA, 1e-4, 0, 0.75, 1500, x, y)

// now when you predict it'll train off the
// dataset, weighting points closer to the
// targer evaluation more, then return
// the prediction.
guess, err := model.Predict([]float64{10.0, -13.666})

func NewLocalLinear 

func NewLocalLinear(method base.OptimizationMethod, alpha, regularization, bandwidth float64, maxIterations int, trainingSet [][]float64, expectedResults []float64) *LocalLinear

NewLocalLinear returns a pointer to the linear model initialized with the learning rate alpha, the training set trainingSet, and the expected results upon which to use the dataset to train, expectedResults.

if you're passing in no training set directly because you want to learn using the online method then just declare the number of features (it's an integer) as an extra arg after the rest of the arguments

Example Least Squares (Stochastic GA): 

// optimization method: Stochastic Gradient Ascent
// Learning rate: 1e-4
// Regulatization term: 6
// Weight Bandwidth: 1.0
// Max Iterations: 800
// Dataset to learn fron: testX
// Expected results dataset: testY
model := NewLocalLinear(base.StochasticGA, 1e-4, 6, 1.0, 800, testX, testY)

err := model.Learn()
if err != nil {
    panic("SOME ERROR!! RUN!")
}

// now I want to predict off of this
// Ordinary Least Squares model!
guess, err = model.Predict([]float64{10000,6})
if err != nil {
    panic("AAAARGGGH! SHIVER ME TIMBERS! THESE ROTTEN SCOUNDRELS FOUND AN ERROR!!!")
}

func (*LocalLinear) Dij 

func (l *LocalLinear) Dij(input []float64, i, j int) (float64, error)

Dij returns the derivative of the cost function J(θ) with respect to the j-th parameter of the hypothesis, θ[j], for the training example x[i]. Used in Stochastic Gradient Descent.

assumes that i,j is within the bounds of the data they are looking up! (because this is getting called so much, it needs to be efficient with comparisons)

func (*LocalLinear) Dj 

func (l *LocalLinear) Dj(input []float64, j int) (float64, error)

Dj returns the partial derivative of the cost function J(θ) with respect to theta[j] where theta is the parameter vector associated with our hypothesis function Predict (upon which we are optimizing

func (*LocalLinear) Examples 

func (l *LocalLinear) Examples() int

Examples returns the number of training examples (m) that the model currently is training from.

func (*LocalLinear) J 

func (l *LocalLinear) J() (float64, error)

J returns the Least Squares cost function of the given linear model. Could be usefull in testing convergance

func (*LocalLinear) LearningRate 

func (l *LocalLinear) LearningRate() float64

LearningRate returns the learning rate α for gradient descent to optimize the model. Could vary as a function of something else later, potentially.

func (*LocalLinear) MaxIterations 

func (l *LocalLinear) MaxIterations() int

MaxIterations returns the number of maximum iterations the model will go through in GradientAscent, in the worst case

func (*LocalLinear) Predict 

func (l *LocalLinear) Predict(x []float64, normalize ...bool) ([]float64, error)

Predict takes in a variable x (an array of floats,) and finds the value of the hypothesis function given the current parameter vector θ

if normalize is given as true, then the input will first be normalized to unit length. Only use this if you trained off of normalized inputs and are feeding an un-normalized input

func (*LocalLinear) String 

func (l *LocalLinear) String() string

String implements the fmt interface for clean printing. Here we're using it to print the model as the equation h(θ)=... where h is the linear hypothesis model

func (*LocalLinear) UpdateLearningRate 

func (l *LocalLinear) UpdateLearningRate(a float64)

UpdateLearningRate set's the learning rate of the model to the given float64.

func (*LocalLinear) UpdateTrainingSet 

func (l *LocalLinear) UpdateTrainingSet(trainingSet [][]float64, expectedResults []float64) error

UpdateTrainingSet takes in a new training set (variable x) as well as a new result set (y). This could be useful if you want to retrain a model starting with the parameter vector of a previous training session, but most of the time wouldn't be used.

type Logistic 

type Logistic struct {
	Parameters []float64 `json:"theta"`

	// Output is the io.Writer used for logging
	// and printing. Defaults to os.Stdout.
	Output io.Writer
	// contains filtered or unexported fields
}

Logistic represents the logistic classification model with a sigmoidal hypothesis

https://en.wikipedia.org/wiki/Logistic_regression

The model is currently optimized using Gradient Ascent, not Newton's method, etc.

The model expects all expected results in the []float64 to come as either a 0 or a 1, and will predict the probability that, based on inputs x, whether y is 1

func NewLogistic 

func NewLogistic(method base.OptimizationMethod, alpha, regularization float64, maxIterations int, trainingSet [][]float64, expectedResults []float64, features ...int) *Logistic

NewLogistic takes in a learning rate alpha, a regularization parameter value (0 means no regularization, higher value means higher bias on the model,) the maximum number of iterations the data can go through in gradient descent, as well as a training set and expected results for that training set.

if you're passing in no training set directly because you want to learn using the online method then just declare the number of features (it's an integer) as an extra arg after the rest of the arguments

DATA FORMAT: The Logistic model expects expected results to be either a 0 or a 1. Predict returns the probability that the item inputted is a 1. Obviously this means that the probability that the inputted x is a 0 is 1-TheGuess

Example Binary Logistic Regression (Batch GA): 

// optimization method: Batch Gradient Ascent
// Learning rate: 1e-4
// Regulatization term: 6
// Max Iterations: 800
// Dataset to learn fron: testX
// Expected results dataset: testY
model := NewLogistic(base.BatchGA, 1e-4, 6, 800, testX, testY)

err := model.Learn()
if err != nil {
    panic("SOME ERROR!! RUN!")
}

// now I want to predict off of this
// Ordinary Least Squares model!
guess, err = model.Predict([]float64{10000,6})
if err != nil {
    panic("AAAARGGGH! SHIVER ME TIMBERS! THESE ROTTEN SCOUNDRELS FOUND AN ERROR!!!")
}

func (*Logistic) Dij 

func (l *Logistic) Dij(i int, j int) (float64, error)

Dij returns the derivative of the cost function J(θ) with respect to the j-th parameter of the hypothesis, θ[j], for the training example x[i]. Used in Stochastic Gradient Descent.

assumes that i,j is within the bounds of the data they are looking up! (because this is getting called so much, it needs to be efficient with comparisons)

func (*Logistic) Dj 

func (l *Logistic) Dj(j int) (float64, error)

Dj returns the partial derivative of the cost function J(θ) with respect to theta[j] where theta is the parameter vector associated with our hypothesis function Predict (upon which we are optimizing

func (*Logistic) Examples 

func (l *Logistic) Examples() int

Examples returns the number of training examples (m) that the model currently is training from.

func (*Logistic) Learn 

func (l *Logistic) Learn() error

Learn takes the struct's dataset and expected results and runs batch gradient descent on them, optimizing theta so you can predict based on those results

func (*Logistic) LearningRate 

func (l *Logistic) LearningRate() float64

LearningRate returns the learning rate α for gradient descent to optimize the model. Could vary as a function of something else later, potentially.

func (*Logistic) MaxIterations 

func (l *Logistic) MaxIterations() int

MaxIterations returns the number of maximum iterations the model will go through in GradientAscent, in the worst case

func (*Logistic) OnlineLearn 

func (l *Logistic) OnlineLearn(errors chan error, dataset chan base.Datapoint, onUpdate func([][]float64), normalize ...bool)

OnlineLearn runs similar to using a fixed dataset with Stochastic Gradient Descent, but it handles data by passing it as a channal, and returns errors through a channel, which lets it run responsive to inputted data from outside the model itself (like using data from the stock market at timed intervals or using realtime data about the weather.)

The onUpdate callback is called whenever the parameter vector theta is changed, so you are able to persist the model with the most up to date vector at all times (you could persist to a database within the callback, for example.) Don't worry about it taking too long and blocking, because the callback is spawned into another goroutine.

NOTE that this function is suggested to run in it's own goroutine, or at least is designed as such.

NOTE part 2: You can pass in an empty dataset, so long as it's not nil, and start pushing after.

NOTE part 3: each example is only looked at as it goes through the channel, so if you want to have each example looked at more than once you must manually pass the data yourself.

NOTE part 4: the optional parameter 'normalize' will , if true, normalize all data streamed through the channel to unit length. This will affect the outcome of the hypothesis, though it could be favorable if your data comes in drastically different scales.

Example Online Logistic Regression: 

// create the channel of data and errors
stream := make(chan base.Datapoint, 100)
errors := make(chan error)

// notice how we are adding another integer
// to the end of the NewLogistic call. This
// tells the model to use that number of features
// (4) in leu of finding that from the dataset
// like you would with batch/stochastic GD
//
// Also – the 'base.StochasticGA' doesn't affect
// anything. You could put batch.
model := NewLogistic(base.StochasticGA, .0001, 0, 0, nil, nil, 4)

go model.OnlineLearn(errors, stream, func(theta [][]float64) {
    // do something with the new theta (persist
    // to database?) in here.
})

go func() {
    for iterations := 0; iterations < 20; iterations++ {
        for i := -200.0; abs(i) > 1; i *= -0.75 {
            for j := -200.0; abs(j) > 1; j *= -0.75 {
                for k := -200.0; abs(k) > 1; k *= -0.75 {
                    for l := -200.0; abs(l) > 1; l *= -0.75 {
                        if i/2+2*k-4*j+2*l+3 > 0 {
                            stream <- base.Datapoint{
                                X: []float64{i, j, k, l},
                                Y: []float64{1.0},
                            }
                        } else {
                            stream <- base.Datapoint{
                                X: []float64{i, j, k, l},
                                Y: []float64{0.0},
                            }
                        }
                    }
                }
            }
        }
    }

    // close the dataset to tell the model
    // to stop learning when it finishes reading
    // what's left in the channel
    close(stream)
}()

// this will block until the error
// channel is closed in the learning
// function (it will, don't worry!)
for {
    err, more := <-errors
    if err != nil {
        panic("THERE WAS AN ERROR!!! RUN!!!!")
    }
    if !more {
        break
    }
}

// Below here all the learning is completed

// predict like usual
guess, err = model.Predict([]float64{42,6,10,-32})
if err != nil {
    panic("AAAARGGGH! SHIVER ME TIMBERS! THESE ROTTEN SCOUNDRELS FOUND AN ERROR!!!")
}

func (*Logistic) PersistToFile 

func (l *Logistic) PersistToFile(path string) error

PersistToFile takes in an absolute filepath and saves the parameter vector θ to the file, which can be restored later. The function will take paths from the current directory, but functions

The data is stored as JSON because it's one of the most efficient storage method (you only need one comma extra per feature + two brackets, total!) And it's extendable.

func (*Logistic) Predict 

func (l *Logistic) Predict(x []float64, normalize ...bool) ([]float64, error)

Predict takes in a variable x (an array of floats,) and finds the value of the hypothesis function given the current parameter vector θ

if normalize is given as true, then the input will first be normalized to unit length. Only use this if you trained off of normalized inputs and are feeding an un-normalized input

func (*Logistic) RestoreFromFile 

func (l *Logistic) RestoreFromFile(path string) error

RestoreFromFile takes in a path to a parameter vector theta and assigns the model it's operating on's parameter vector to that.

The path must ba an absolute path or a path from the current directory

This would be useful in persisting data between running a model on data, or for graphing a dataset with a fit in another framework like Julia/Gadfly.

func (*Logistic) String 

func (l *Logistic) String() string

String implements the fmt interface for clean printing. Here we're using it to print the model as the equation h(θ)=... where h is the logistic hypothesis model

func (*Logistic) Theta 

func (l *Logistic) Theta() []float64

Theta returns the parameter vector θ for use in persisting the model, and optimizing the model through gradient descent ( or other methods like Newton's Method)

func (*Logistic) UpdateLearningRate 

func (l *Logistic) UpdateLearningRate(a float64)

UpdateLearningRate set's the learning rate of the model to the given float64.

func (*Logistic) UpdateTrainingSet 

func (l *Logistic) UpdateTrainingSet(trainingSet [][]float64, expectedResults []float64) error

UpdateTrainingSet takes in a new training set (variable x) as well as a new result set (y). This could be useful if you want to retrain a model starting with the parameter vector of a previous training session, but most of the time wouldn't be used.

type Softmax 

type Softmax struct {
	Parameters [][]float64 `json:"theta"`

	// Output is the io.Writer used for logging
	// and printing. Defaults to os.Stdout.
	Output io.Writer
	// contains filtered or unexported fields
}

Softmax represents a softmax classification model in 'k' demensions. It is generally thought of as a generalization of the Logistic Regression model. Prediction will return a vector ([]float64) that corresponds to the probabilty (where i is the index) that the inputted features is 'i'. Softmax classification operates assuming the Multinomial probablility distribution of data.

TODO: add wikipedia link

Expected results expects an 'integer' (it's still passed as a float64) between 0 and k-1. K must be passed when creating the model.

func NewSoftmax 

func NewSoftmax(method base.OptimizationMethod, alpha, regularization float64, k, maxIterations int, trainingSet [][]float64, expectedResults []float64, features ...int) *Softmax

NewSoftmax takes in a learning rate alpha, a regularization parameter value (0 means no regularization, higher value means higher bias on the model,) the maximum number of iterations the data can go through in gradient descent, as well as a training set and expected results for that training set.

func (*Softmax) Dij 

func (s *Softmax) Dij(i, k int) ([]float64, error)

Dij returns the derivative of the cost function J(θ) with respect to the j-th parameter of the hypothesis, θ[j], for the training example x[i]. Used in Stochastic Gradient Descent.

assumes that i,j is within the bounds of the data they are looking up! (because this is getting called so much, it needs to be efficient with comparisons)

func (*Softmax) Dj 

func (s *Softmax) Dj(k int) ([]float64, error)

Dj returns the partial derivative of the cost function J(θ) with respect to theta[k] where theta is the parameter vector associated with our hypothesis function Predict (upon which we are optimizing.

k is the classification value you are finding the gradient for (because the parameter vactor is actually a vector _of_ vectors!)

func (*Softmax) Examples 

func (s *Softmax) Examples() int

Examples returns the number of training examples (m) that the model currently is training from.

func (*Softmax) Learn 

func (s *Softmax) Learn() error

Learn takes the struct's dataset and expected results and runs gradient descent on them, optimizing theta so you can predict accurately based on those results

func (*Softmax) LearningRate 

func (s *Softmax) LearningRate() float64

LearningRate returns the learning rate α for gradient descent to optimize the model. Could vary as a function of something else later, potentially.

func (*Softmax) MaxIterations 

func (s *Softmax) MaxIterations() int

MaxIterations returns the number of maximum iterations the model will go through in GradientAscent, in the worst case

func (*Softmax) OnlineLearn 

func (s *Softmax) OnlineLearn(errors chan error, dataset chan base.Datapoint, onUpdate func([][]float64), normalize ...bool)

OnlineLearn runs similar to using a fixed dataset with Stochastic Gradient Descent, but it handles data by passing it as a channal, and returns errors through a channel, which lets it run responsive to inputted data from outside the model itself (like using data from the stock market at timed intervals or using realtime data about the weather.)

The onUpdate callback is called whenever the parameter vector theta is changed, so you are able to persist the model with the most up to date vector at all times (you could persist to a database within the callback, for example.) Don't worry about it taking too long and blocking, because the callback is spawned into another goroutine.

NOTE that this function is suggested to run in it's own goroutine, or at least is designed as such.

NOTE part 2: You can pass in an empty dataset, so long as it's not nil, and start pushing after.

NOTE part 3: each example is only looked at as it goes through the channel, so if you want to have each example looked at more than once you must manually pass the data yourself.

NOTE part 4: the optional parameter 'normalize' will , if true, normalize all data streamed through the channel to unit length. This will affect the outcome of the hypothesis, though it could be favorable if your data comes in drastically different scales.

Example Online Logistic Regression: 

    // create the channel of data and errors
    stream := make(chan base.Datapoint, 100)
    errors := make(chan error)

    // notice how we are adding another integer
    // to the end of the NewSoftmax call. This
    // tells the model to use that number of features
    // (2) in leu of finding that from the dataset
    // like you would with batch/stochastic GD
    //
    // Also – the 'base.StochasticGA' doesn't affect
    // anything. You could put batch or any other model.
    model := NewSoftmax(base.StochasticGA, 5e-5, 0, 3, 0, nil, nil, 2)

    go model.OnlineLearn(errors, stream, func(theta [][]float64) {
        // do something with the new theta (persist
        // to database?) in here.
    })

    go model.OnlineLearn(errors, stream, func(theta [][]float64) {})

    // start passing data to our datastream
    //
    // we could have data already in our channel
    // when we instantiated the Perceptron, though
	   go func() {
        for iter := 0; iter < 3; iter++ {
            for i := -2.0; i < 2.0; i += 0.15 {
                for j := -2.0; j < 2.0; j += 0.15 {

                    if -2*i+j/2-0.5 > 0 && -1*i-j < 0 {
                             stream <- base.Datapoint{
                                X: []float64{float64(i), float64(j)},
                                Y: []float64{2.0},
                            }
                    } else if -2*i+j/2-0.5 > 0 && -1*i-j > 0 {
                            stream <- base.Datapoint{
                                X: []float64{float64(i), float64(j)},
                                Y: []float64{1.0},
                            }
                    } else {
                        stream <- base.Datapoint{
                                X: []float64{float64(i), float64(j)},
                                Y: []float64{0.0},
                            }
                    }
                }
            }
        }

        // close the dataset
        close(stream)
    }()

    // this will block until the error
    // channel is closed in the learning
    // function (it will, don't worry!)
    for {
        err, more := <-errors
        if err != nil {
            panic("THERE WAS AN ERROR!!! RUN!!!!")
        }
        if !more {
            break
        }
    }

    // Below here all the learning is completed

    // predict like usual
    guess, err = model.Predict([]float64{42,6,10,-32})
    if err != nil {
        panic("AAAARGGGH! SHIVER ME TIMBERS! THESE ROTTEN SCOUNDRELS FOUND AN ERROR!!!")
    }

func (*Softmax) PersistToFile 

func (s *Softmax) PersistToFile(path string) error

PersistToFile takes in an absolute filepath and saves the parameter vector θ to the file, which can be restored later. The function will take paths from the current directory, but functions

The data is stored as JSON because it's one of the most efficient storage method (you only need one comma extra per feature + two brackets, total!) And it's extendable.

func (*Softmax) Predict 

func (s *Softmax) Predict(x []float64, normalize ...bool) ([]float64, error)

Predict takes in a variable x (an array of floats,) and finds the value of the hypothesis function given the current parameter vector θ

func (*Softmax) RestoreFromFile 

func (s *Softmax) RestoreFromFile(path string) error

RestoreFromFile takes in a path to a parameter vector theta and assigns the model it's operating on's parameter vector to that.

The path must ba an absolute path or a path from the current directory

This would be useful in persisting data between running a model on data, or for graphing a dataset with a fit in another framework like Julia/Gadfly.

func (*Softmax) String 

func (s *Softmax) String() string

String implements the fmt interface for clean printing. Here we're using it to print the model as the equation h(θ)=... where h is the softmax hypothesis model

func (*Softmax) Theta 

func (s *Softmax) Theta() [][]float64

Theta returns the parameter vector θ for use in persisting the model, and optimizing the model through gradient descent ( or other methods like Newton's Method)

func (*Softmax) UpdateLearningRate 

func (s *Softmax) UpdateLearningRate(a float64)

UpdateLearningRate set's the learning rate of the model to the given float64.

func (*Softmax) UpdateTrainingSet 

func (s *Softmax) UpdateTrainingSet(trainingSet [][]float64, expectedResults []float64) error

UpdateTrainingSet takes in a new training set (variable x) as well as a new result set (y). This could be useful if you want to retrain a model starting with the parameter vector of a previous training session, but most of the time wouldn't be used.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.