README
¶
Perceptron
import "github.com/cdipaolo/goml/perceptron"
The perceptron model holds easy to implement, online, reactive perceptrons that work with Golang channels (streams) of data to make pipelined training more efficient.
The perceptron is similar to regular Logistic Regression, except that it returns discrete values rather than the probability of a result being true (or 1,) which could be 0.76543, for example. The perceptron uses a setp function instead of a sigmoid transform its inputs into a hypothesis. It's model comes primarily from biology theory, representing a neuron in the brain.
The optimization method for a perceptron also operates differently than logistic regression, which results in some cool properties. The perceptron doesn't update for each training example. Instead, it guesses what the correct classification should be, then only if it gets it wrong will the perceptron update it's parameter vector θ (also known as 'the weights.') What this allows you to do is constantly feed data into a perceptron, whereby it can be continually update and learn (when learning, obviously this won't work when predicting as this is not an unsupervised model.) Note the update rule below that we're using within the binary, online perceptron if this doesn't make much sense initially.
// update the parameters if the guess
// is wrong
if guess[0] != point.Y[0] {
p.Parameters[0] += p.alpha * (point.Y[0] - guess[0])
for i := 1; i < len(p.Parameters); i++ {
p.Parameters[i] += p.alpha * (point.Y[0] - guess[0]) * point.X[i-1]
}
// call the OnUpdate callback with the new theta
// appended to a blank slice so the vector is
// passed by value and not by reference
go p.OnUpdate(append([]float64{}, p.Parameters...))
}
It should be noted that you want interspersion between the results you pass, that is, you want to 'alternate' sending positive and negative results with a decently quick frequency. If you instead pass a string of 500 negative results followed by 500 positive ones, the perceptron will start by thinking that all results are negative, then correcting completely and thinking that most results are positive, but it will never converge along the optimal boundary hyperplane between the binary classification.
implemented models
- binary, online perceptron
- binary, online kernel perceptron
- this model uses more memory than the regular perceptron, but by using the kernel trick it allows you to input theoretically infinite feature spaces into it as well as fitting non-linear decision boundaries with the model! You can use ready-made (though custimizable) kernels from the
goml/basepackage. It will take longer to train, as well.
- this model uses more memory than the regular perceptron, but by using the kernel trick it allows you to input theoretically infinite feature spaces into it as well as fitting non-linear decision boundaries with the model! You can use ready-made (though custimizable) kernels from the
example binary, online perceptron
This example is pretty much verbatim from the tests. If you want to see other simple examples of Perceptrons in action, check out the tests for each model!
// create the channel of data and errors
//
// note that we are buffering the data stream
// channel
stream := make(chan base.Datapoint, 100)
errors := make(chan error)
model := NewPerceptron(0.1, 1, func (theta []float64) {}, stream)
// make the model learn in a different goroutine,
// passing errors to our error channel so we know
// the lowdown on our training
go model.Learn(errors)
// start passing data to our stream
//
// we could have data already in our channel
// when we instantiated the Perceptron, though
//
// and note that this data could be coming from
// some web server, or whatever!!
for i := -500.0; abs(i) > 1; i *= -0.997 {
if 10 + (i-20)/2 > 0 {
stream <- base.Datapoint{
X: []float64{i-20},
Y: []float64{1.0},
}
} else {
stream <- base.Datapoint{
X: []float64{i-20},
Y: []float64{0},
}
}
}
// close the dataset
close(stream)
for {
err, more := <- errors
if more {
// there is another error
fmt.Printf("Error passed: %v", err)
} else {
// training is done!
break
}
}
// now you can predict!!
// note that guess is a []float64 of len() == 1
// when it isn't nil (it is in the case of
// an error)
guess, err := model.Predict([]float64{i})
if err != nil {
panic("EGATZ!! I FOUND AN ERROR! BETTER CHECK YOUR INPUT DIMENSIONS!")
}
Documentation
¶
Overview ¶
Package perceptron holds the online perceptron model of learning. A perceptron works by 'reacting' to bad predictions, and only updating it's parameter vector when it does make a bad prediction. If you want to read more about the details of the perceptron itself, go to the Perceptron struct documentation.
The package implements the training of a perceptron as running on a buffered channel of base.Datapoint's. This lets you run the learning of the model reactively off of a data stream, an API for example, only pushing new data into the pipeline when it's recieved. You are given an OnUpdate callback with the Perceptron struct, which is called whenever the model updates it parameter vector. It passes a copy of the new parameter vector as a copy and runs the callback in a new goroutine. This would let the user persist the model to a database of their choosing in realtime, calling update to a table consistantly within the callback.
The Perceptron also takes in a channel of errors when it learns, which lets the user see any errors while learning but not actually interrupting the learning itself. The model just ignores errors (usually caused by a mismatch of dimension on the input vector) and goes to the next datapoint. The channel of errors is closed when learning is done so you know when your model is finished working its way though the dataset (this implies that you closed the data stream, though.)
Example Online, Binary Perceptron (no layers, etc.):
// create the channel of data and errors
stream := make(chan base.Datapoint, 100)
errors := make(chan error)
model := NewPerceptron(0.1, 1, stream)
go model.OnlineLearn(errors, stream, func (theta []float64) {
fmt.Fprintf(p.Output, "Theta updated to %v!\n", theta)
})
// start passing data to our datastream
//
// we could have data already in our channel
// when we instantiated the Perceptron, though
//
// and note that this data could be coming from
// some web server, or whatever!!
go func() {
for i := -500.0; abs(i) > 1; i *= -0.997 {
if 10 + (i-20)/2 > 0 {
stream <- base.Datapoint{
X: []float64{i-20},
Y: []float64{1.0},
}
} else {
stream <- base.Datapoint{
X: []float64{i-20},
Y: []float64{0},
}
}
}
}()
// close the dataset
close(stream)
for {
err, more := <- errors
if err != nil {
fmt.Fprintf(p.Output, "Error passed: %v", err)
} else {
// training is done!
break
}
}
// now you can predict!!
// note that guess is a []float64 of len() == 1
// when it isn't nil
guess, err := model.Predict([]float64{i})
if err != nil {
panic("EGATZ!! I FOUND AN ERROR! BETTER CHECK YOUR INPUT DIMENSIONS!")
}
Index ¶
- type KernelPerceptron
- func (p *KernelPerceptron) OnlineLearn(errors chan error, dataset chan base.Datapoint, onUpdate func([][]float64), ...)
- func (p *KernelPerceptron) PersistToFile(path string) error
- func (p *KernelPerceptron) Predict(x []float64, normalize ...bool) ([]float64, error)
- func (p *KernelPerceptron) RestoreFromFile(path string) error
- func (p *KernelPerceptron) String() string
- type Perceptron
- func (p *Perceptron) OnlineLearn(errors chan error, dataset chan base.Datapoint, onUpdate func([][]float64), ...)
- func (p *Perceptron) PersistToFile(path string) error
- func (p *Perceptron) Predict(x []float64, normalize ...bool) ([]float64, error)
- func (p *Perceptron) RestoreFromFile(path string) error
- func (p *Perceptron) String() string
- func (p *Perceptron) UpdateLearningRate(a float64)
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type KernelPerceptron ¶
type KernelPerceptron struct {
// SV stores the KernelPerceptron's support
// vectors
SV []base.Datapoint `json:"support_vectors,omitempty"`
Kernel func([]float64, []float64) float64
// Output is the io.Writer used for logging
// and printing. Defaults to os.Stdout.
Output io.Writer
}
KernelPerceptron represents the perceptron online learning model, where you input features and the model's state reacts to the input and changes weights (parameter vector theta) only when the guess by the algorithm is wrong. Unlike the base Perceptron model, which maps inputs using an affine feature space, the KernelPerceptron can use different kernels which may map to infinite-dimension feature spaces.
The hypothesis is a generalized version of the regular perceptron, where i∈M where M are all misclassified examples:
sgn(Σ αy[i] * K(x[i], x))
In this implementation, data is passed through a channel, where the learn function is run in a separate goroutine and stops when the channel is closed.
http://cs229.stanford.edu/notes/cs229-notes6.pdf https://en.wikipedia.org/wiki/Perceptron https://en.wikipedia.org/wiki/Kernel_perceptron
KernelPerceptron implements the OnlineModel interface, not the Model interface, because it uses online learning only
Data results in this binary class model are expected to be either -1 or 1 (ie. the base.Datapoint's you pass should, called point, have point.Y be either [-1] or [1])
You must pass in a valid kernel function with the NewKernelPerceptron function. You can find premade, valid kernels in the `base` package if you want to use those.
func NewKernelPerceptron ¶
func NewKernelPerceptron(kernel func([]float64, []float64) float64) *KernelPerceptron
NewKernelPerceptron takes in a learning rate alpha, the number of features (not including the constant term) being evaluated by the model, the update callback called whenever the perceptron updates the parameter vector theta (whenever it makes a wrong guess), and a channel of datapoints that will be used in training and returns an instantiated model.
Again! Features _does not_ include the constant term!
Also, learning rate of 0.1 seems to work well in many cases. (I also heard that in a lecture video from a UW professor)
Weight is the importance given to newer support vectors in prediction. Should be 0 < w
func (*KernelPerceptron) OnlineLearn ¶
func (p *KernelPerceptron) OnlineLearn(errors chan error, dataset chan base.Datapoint, onUpdate func([][]float64), normalize ...bool)
OnlineLearn runs off of the datastream within the Perceptron structure. Whenever the model makes a wrong prediction the parameter vector theta is updated to reflect that, as discussed in the documentation for the Perceptron struct itself, and the OnUpdate function is called with the newly updated parameter vector. Learning will stop when the data channel is closed and all remaining datapoints within the channel have been read.
The errors channel will be closed when learning is completed so you know when it's done if you're relying on that for whatever reason
onUpdate func ([]float64):
onUpdate is a function that is called whenever the perceptron updates it's support vectors This acts almost like a callback and passes the newly added support vector as a slice of floats.
This might be useful is you want to maintain an up to date persisted model in a database of your choosing and you'd like to update it constantly.
This will be spawned into a new goroutine, so don't worry about the function taking a long time, or blocking.
If you want to monitor errors happening within this function, just have a channel of errors you send do within this channel, or some other method if it fits your scenario better.
NOTE that there is an optional last parameter which, when true, will normalize all data given on the stream. This will potentially help the optimization converge faster. This is given as a parameter because you won't have direct access to the dataset before hand like you would in batch/stochastic settings.
Example Online Kernel Perceptron:
// create the channel of data and errors
stream := make(chan base.Datapoint, 100)
errors := make(chan error)
// The kernel could be any kernel from the Base
// package, or it could be your own function!
// I suggest you look at the code for the kernels
// if you want to make sense of them. It's pretty
// intuitive and simple.
model := NewKernelPerceptron(base.GaussianKernel(50))
go model.OnlineLearn(errors, stream, func(SV [][]float64) {
// do something with the newly added support
// vector (persist to database?) in here.
})
go func() {
for iterations := 0; iterations < 20; iterations++ {
for i := -200.0; abs(i) > 1; i *= -0.7 {
for j := -200.0; abs(j) > 1; j *= -0.7 {
for k := -200.0; abs(k) > 1; k *= -0.7 {
for l := -200.0; abs(l) > 1; l *= -0.7 {
if i/2+2*k-4*j+2*l+3 > 0 {
stream <- base.Datapoint{
X: []float64{i, j, k, l},
Y: []float64{1.0},
}
} else {
stream <- base.Datapoint{
X: []float64{i, j, k, l},
Y: []float64{-1.0},
}
}
}
}
}
}
}
// close the dataset to tell the model
// to stop learning when it finishes reading
// what's left in the channel
close(stream)
}()
// this will block until the error
// channel is closed in the learning
// function (it will, don't worry!)
for {
err, more := <-errors
if err != nil {
panic("THERE WAS AN ERROR!!! RUN!!!!")
}
if !more {
break
}
}
// Below here all the learning is completed
// predict like usual
guess, err = model.Predict([]float64{42,6,10,-32})
if err != nil {
panic("AAAARGGGH! SHIVER ME TIMBERS! THESE ROTTEN SCOUNDRELS FOUND AN ERROR!!!")
}
func (*KernelPerceptron) PersistToFile ¶
func (p *KernelPerceptron) PersistToFile(path string) error
PersistToFile takes in an absolute filepath and saves the parameter vector θ to the file, which can be restored later. The function will take paths from the current directory, but functions
The data is stored as JSON because it's one of the most efficient storage method (you only need one comma extra per feature + two brackets, total!) And it's extendable.
func (*KernelPerceptron) Predict ¶
func (p *KernelPerceptron) Predict(x []float64, normalize ...bool) ([]float64, error)
Predict takes in a variable x (an array of floats,) and finds the value of the hypothesis function given the current parameter vector θ
func (*KernelPerceptron) RestoreFromFile ¶
func (p *KernelPerceptron) RestoreFromFile(path string) error
RestoreFromFile takes in a path to a parameter vector theta and assigns the model it's operating on's parameter vector to that.
The path must ba an absolute path or a path from the current directory
This would be useful in persisting data between running a model on data, or for graphing a dataset with a fit in another framework like Julia/Gadfly.
func (*KernelPerceptron) String ¶
func (p *KernelPerceptron) String() string
String implements the fmt interface for clean printing. Here we're using it to print the model as the equation h(θ)=... where h is the perceptron hypothesis model.
Note that I'm using the terniary operator to represent the perceptron:
h(θ,x) = Σ y[i]*K(x[i], x`) > 0 ? 1 : 0
type Perceptron ¶
type Perceptron struct {
Parameters []float64 `json:"theta"`
// Output is the io.Writer used for logging
// and printing. Defaults to os.Stdout.
Output io.Writer
// contains filtered or unexported fields
}
Perceptron represents the perceptron online learning model, where you input features and the model's state reacts to the input and changes weights (parameter vector theta) only when the guess by the algorithm is wrong
The hypothesis of the Perceptron is similar to logistic regression in that the extremes tend to 0 and 1, but the actual hypothesis runs the input with weights through a step function, not a sigmoid, namely:
if (θx * y < 0) {
θ := θ + α*yx
}
In this implementation, data is passed through a channel, where the learn function is run in a separate goroutine and stops when the channel is closed.
http://cs229.stanford.edu/notes/cs229-notes6.pdf https://en.wikipedia.org/wiki/Perceptron
Perceptron implements the OnlineModel interface, not the Model interface, because it uses online learning
Unlike the General Linear Models, for example, the data is expected to be passed as a Dataset struct so it can be easily passed through the data pipeline channel
Data results in this binary class model are expected to be either -1 or 1 (ie. the base.Datapoint's you pass should, called point, have point.Y be either [-1] or [1])
func NewPerceptron ¶
func NewPerceptron(alpha float64, features int) *Perceptron
NewPerceptron takes in a learning rate alpha, the number of features (not including the constant term) being evaluated by the model, the update callback called whenever the perceptron updates the parameter vector theta (whenever it makes a wrong guess), and a channel of datapoints that will be used in training and returns an instantiated model.
Again! Features _does not_ include the constant term!
Also, learning rate of 0.1 seems to work well in many cases. (I also heard that in a lecture video from a UW professor)
func (*Perceptron) OnlineLearn ¶
func (p *Perceptron) OnlineLearn(errors chan error, dataset chan base.Datapoint, onUpdate func([][]float64), normalize ...bool)
OnlineLearn runs off of the datastream within the Perceptron structure. Whenever the model makes a wrong prediction the parameter vector theta is updated to reflect that, as discussed in the documentation for the Perceptron struct itself, and the OnUpdate function is called with the newly updated parameter vector. Learning will stop when the data channel is closed and all remaining datapoints within the channel have been read.
The errors channel will be closed when learning is completed so you know when it's done if you're relying on that for whatever reason
onUpdate func ([]float64):
onUpdate is a function that is called whenever the perceptron updates it's parameter vector theta. This acts almost like a callback and passes the newly updated parameter vector theta as a slice of floats.
This might be useful is you want to maintain an up to date persisted model in a database of your choosing and you'd like to update it constantly.
This will be spawned into a new goroutine, so don't worry about the function taking a long time, or blocking.
If you want to monitor errors happening within this function, just have a channel of errors you send do within this channel, or some other method if it fits your scenario better.
NOTE that there is an optional last parameter which, when true, will normalize all data given on the stream. This will potentially help gradient descent converge faster. This is given as a parameter because you won't have direct access to the dataset before hand like you would in batch/stochastic settings.
Example Online, Binary Perceptron (no layers, etc.):
// create the channel of data and errors
stream := make(chan base.Datapoint, 100)
errors := make(chan error)
model := NewPerceptron(0.1, 1, stream)
go model.OnlineLearn(errors, stream, func (theta []float64) {
// do something with the new theta (persist
// to database?) in here.
fmt.Fprintf(p.Output, "Theta updated to %v!\n", theta)
})
// start passing data to our datastream
//
// we could have data already in our channel
// when we instantiated the Perceptron, though
//
// and note that this data could be coming from
// some web server, or whatever!!
go func() {
for i := -500.0; abs(i) > 1; i *= -0.997 {
if 10 + (i-20)/2 > 0 {
stream <- base.Datapoint{
X: []float64{i-20},
Y: []float64{1.0},
}
} else {
stream <- base.Datapoint{
X: []float64{i-20},
Y: []float64{0},
}
}
}
}()
// close the dataset
close(stream)
for {
err, more := <- errors
if err != nil {
fmt.Fprintf(p.Output, "Error passed: %v", err)
} else {
// training is done!
break
}
}
// now you can predict!!
// note that guess is a []float64 of len() == 1
// when it isn't nil
guess, err := model.Predict([]float64{i})
if err != nil {
panic("EGATZ!! I FOUND AN ERROR! BETTER CHECK YOUR INPUT DIMENSIONS!")
}
func (*Perceptron) PersistToFile ¶
func (p *Perceptron) PersistToFile(path string) error
PersistToFile takes in an absolute filepath and saves the parameter vector θ to the file, which can be restored later. The function will take paths from the current directory, but functions
The data is stored as JSON because it's one of the most efficient storage method (you only need one comma extra per feature + two brackets, total!) And it's extendable.
func (*Perceptron) Predict ¶
func (p *Perceptron) Predict(x []float64, normalize ...bool) ([]float64, error)
Predict takes in a variable x (an array of floats,) and finds the value of the hypothesis function given the current parameter vector θ
func (*Perceptron) RestoreFromFile ¶
func (p *Perceptron) RestoreFromFile(path string) error
RestoreFromFile takes in a path to a parameter vector theta and assigns the model it's operating on's parameter vector to that.
The path must ba an absolute path or a path from the current directory
This would be useful in persisting data between running a model on data, or for graphing a dataset with a fit in another framework like Julia/Gadfly.
func (*Perceptron) String ¶
func (p *Perceptron) String() string
String implements the fmt interface for clean printing. Here we're using it to print the model as the equation h(θ)=... where h is the perceptron hypothesis model.
Note that I'm using the terniary operator to represent the perceptron:
h(θ,x) = θx > 0 ? 1 : 0
func (*Perceptron) UpdateLearningRate ¶
func (p *Perceptron) UpdateLearningRate(a float64)
UpdateLearningRate set's the learning rate of the model to the given float64.
Source Files