README
¶
TF-IDF
Illustration
TF-IDF (Term Frequency-Inverse Document Frequency)：The importance of a term is in proportion to the number of appearance in the document, and in inverse proportion to the number of appearance in the corpus.
TF (Term Frequency) = # of term appeared in doc / total # of words in doc
IDF (Inverse Document Frequency) = log( total # of docs / # of docs containing the term )
denominator needed to be adjusted by +1 if none of the doc in corpus contains the term
TF-IDF = TF * IDF
Cosine Similarity = cos(( A * B ) / ( |A| * |B| ) )
Usage
t := tfidf.NewTFIDF()
t.AddDocs("hi there", "how are you", "how do you do")
doc := "where are you"
weight := t.Cal(doc)
fmt.Printf("weight of %s is %+v.\n", doc, weight)
Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Corpus ¶
type Corpus struct {
// TermCount stores all terms appeared in the corpus as key
// and the # of docs containing the term as value. It is
// used for calculating idf.
TermCount map[string]float64
// Corpus stores all documents with it's content hash as key
// and the document as value. It is used for calculating idf.
Documents map[string]Document
}
Corpus defines Corpus
type Document ¶
type Document struct {
ID string
Content string
Terms []string
// contains filtered or unexported fields
}
Document defines Document
type TFIDF ¶
type TFIDF struct {
Corpus *Corpus
// contains filtered or unexported fields
}
TFIDF defines TFIDF
func NewTFIDFWithTokenizer ¶
NewTFIDFWithTokenizer factory
func (*TFIDF) AddDoc ¶
AddDoc adds doc to the corpus by: 1. Update Corupus for later calculation of other doc's idf (as numerator) 2. Update TermDocMap for later calculation of other docs's idf (as denominator)
Source Files
Directories