Documentation
¶
Overview ¶
Package dedupe provides an interface to dedupe anything. It was originally written for deduping the firehose. Given a prefix and a size it will create a unique hash based on the content of the message to be deduped. It will test if that hash has been seen before in whatever storage mechanism its been given.
DeduperStorage can be any type of storage that implements the given interface. See the memcache package for an example.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ( ErrorKeyExistsOnWrite = errors.New("Dedupe: key exists while attempting write") ErrorKeyExistsOnRead = errors.New("Dedupe: key exists while attempting read") ErrorUnknown = errors.New("Dedupe: unknown error occured") ErrorUnknownType = errors.New("Dedupe: unknown filtering algorithm") )
Error codes
Functions ¶
This section is empty.
Types ¶
type Deduper ¶
type Deduper struct {
Type string
Size int
Prefix string
Storage DeduperStorage
}
Type - the type of filtering algorithm to use Size - the of the filter, the bigger the number the more unique (for lossy hashmaps only) Prefix - some kind of namespace that relates to your implementation Storage - storage mechanism for searching for duplicates, see memcache package for example
func (*Deduper) Contains ¶
After implementing your storage interface and instantiating a Deduper struct, call this method to determine if the message is a duplicate (true) or is not a duplicate (false). Will return an error if there is a failure in the storage mechanism.
d := &Deduper{Type: "hashmap", Size: 1000000, Prefix: "dedupe:firehose:memcache", Storage: &MemcacheDeduper{}}
if d.Contans(j.Message) == false {
// process message
}
Source Files