Documentation
¶
Overview ¶
Package repo contains utility modules to open offset logs and create different kinds of indexes.
Index ¶
- Constants
- func AllKeyPairs(r Interface) (map[string]ssb.KeyPair, error)
- func DefaultKeyPair(r Interface, algo refs.RefAlgo) (ssb.KeyPair, error)
- func LoadKeyPair(r Interface, name string) (ssb.KeyPair, error)
- func NewFilteredLog(b margaret.Log, fn FilterFunc) margaret.Log
- func NewKeyPair(r Interface, name string, algo refs.RefAlgo) (ssb.KeyPair, error)
- func NewKeyPairFromSeed(r Interface, name string, algo refs.RefAlgo, seed io.Reader) (ssb.KeyPair, error)
- func OpenBadgerDB(path string) (*badger.DB, error)
- func OpenBadgerIndex(r Interface, name string, f LibrarianIndexCreater) (*badger.DB, librarian.SeqSetterIndex, librarian.SinkIndex, error)
- func OpenBlobStore(r Interface) (ssb.BlobStore, error)
- func OpenFileSystemMultiLog(r Interface, name string, f multilog.Func) (*roaring.MultiLog, librarian.SinkIndex, error)
- func OpenIndex(db *badger.DB, name string, ...) (librarian.Index, librarian.SinkIndex, error)
- func OpenLog(r Interface, path ...string) (multimsg.AlterableLog, error)
- func OpenStandaloneMultiLog(r Interface, name string, f multilog.Func) (multilog.MultiLog, librarian.SinkIndex, error)
- type FilterFunc
- type FilteredLog
- type Interface
- type LibrarianIndexCreater
- type MakeMultiLog
- type MakeSimpleIndex
- type MultiLogMaker
- type ResolverFilter
- type SequenceResolver
- func (sr *SequenceResolver) Append(seq int64, feed int64, claimed, received time.Time) error
- func (sr SequenceResolver) Close() error
- func (sr *SequenceResolver) Load() (int64, error)
- func (sr SequenceResolver) Seq() int64
- func (sr *SequenceResolver) Serialize() error
- func (sr SequenceResolver) SortAndFilter(seqs []int64, by SortDomain, ok ResolverFilter, desc bool) (SortedSeqSlice, error)
- func (sr SequenceResolver) SortAndFilterAll(by SortDomain, ok ResolverFilter, desc bool) (SortedSeqSlice, error)
- func (sr SequenceResolver) SortAndFilterBitmap(seqs *bmap.Bitmap, by SortDomain, ok ResolverFilter, desc bool) (SortedSeqSlice, error)
- func (sr SequenceResolver) String() string
- type SimpleIndexMaker
- type SortDomain
- type SortedAscending
- type SortedDescending
- type SortedSeqSlice
- type SortedSequence
Constants ¶
const PrefixIndex = "indexes"
const PrefixMultiLog = "sublogs"
Variables ¶
This section is empty.
Functions ¶
func NewFilteredLog ¶ added in v0.2.1
func NewFilteredLog(b margaret.Log, fn FilterFunc) margaret.Log
NewFilteredLog wraps the passed log into a new one, using the FilterFunc to decide if a message is in the log.
func NewKeyPairFromSeed ¶
func OpenBadgerDB ¶ added in v0.2.1
func OpenBadgerIndex ¶
func OpenBadgerIndex(r Interface, name string, f LibrarianIndexCreater) (*badger.DB, librarian.SeqSetterIndex, librarian.SinkIndex, error)
func OpenFileSystemMultiLog ¶
Types ¶
type FilterFunc ¶ added in v0.2.1
FilterFunc works on messages of a FilteredLog. If the func returns true, the log is in the filtered log.
type FilteredLog ¶ added in v0.2.1
type FilteredLog struct {
// contains filtered or unexported fields
}
FilteredLog omits entries in the backing log as decided by the configured FilterFunc. It does so by claiming the entries are deleted (via returning margaret.ErrNulled instead)
func (FilteredLog) Append ¶ added in v0.2.1
func (fl FilteredLog) Append(val interface{}) (int64, error)
func (FilteredLog) Changes ¶ added in v0.2.1
func (fl FilteredLog) Changes() luigi.Observable
func (FilteredLog) Get ¶ added in v0.2.1
func (fl FilteredLog) Get(s int64) (interface{}, error)
Get retrieves the message object by traversing the authors sublog to the root log
func (FilteredLog) Seq ¶ added in v0.2.1
func (fl FilteredLog) Seq() int64
type LibrarianIndexCreater ¶
type LibrarianIndexCreater func(*badger.DB) (librarian.SeqSetterIndex, librarian.SinkIndex)
type MakeMultiLog ¶
type MakeSimpleIndex ¶
type MultiLogMaker ¶
type ResolverFilter ¶
ResolverFilter get's passed the value from the domain that is searched. Should return true if the value should be included and sorted
type SequenceResolver ¶
type SequenceResolver struct {
// contains filtered or unexported fields
}
SequenceResolver holds three gigantic arrays for each of the understood ResolveDomains.
It should be hooked into a receive log, and filled with Append.
TODO: a better approach might be to fetch these lazyly from disk if they become too large. At 1mio messages we roughly look at 8mb per domain.
func NewSequenceResolver ¶
func NewSequenceResolver(r Interface) (*SequenceResolver, error)
NewSequenceResolver opens the stored resolver at r.GetPath("seqmaps") and Loads existing values.
func NewSequenceResolverFromLog ¶
func NewSequenceResolverFromLog(l margaret.Log) (*SequenceResolver, error)
NewSequenceResolverFromLog creates a fresh resolver reading the full margaret log. Expects to read refs.Message from the log. Useful for testing.
func (SequenceResolver) Close ¶
func (sr SequenceResolver) Close() error
Close serialzes the resolver to disk.
func (*SequenceResolver) Load ¶
func (sr *SequenceResolver) Load() (int64, error)
Load reads the files from repo and deserializes them.
func (SequenceResolver) Seq ¶
func (sr SequenceResolver) Seq() int64
Seq returns the number of entries held by the resolver.
func (*SequenceResolver) Serialize ¶
func (sr *SequenceResolver) Serialize() error
Serialize does the reverse from Load. It saves the three domains to disk.
func (SequenceResolver) SortAndFilter ¶
func (sr SequenceResolver) SortAndFilter(seqs []int64, by SortDomain, ok ResolverFilter, desc bool) (SortedSeqSlice, error)
SortAndFilter goes through seqs in the passed domain using the filter function to include wanted elements. desc: true means descending, desc: false means ascending.
func (SequenceResolver) SortAndFilterAll ¶
func (sr SequenceResolver) SortAndFilterAll(by SortDomain, ok ResolverFilter, desc bool) (SortedSeqSlice, error)
func (SequenceResolver) SortAndFilterBitmap ¶
func (sr SequenceResolver) SortAndFilterBitmap(seqs *bmap.Bitmap, by SortDomain, ok ResolverFilter, desc bool) (SortedSeqSlice, error)
func (SequenceResolver) String ¶
func (sr SequenceResolver) String() string
type SimpleIndexMaker ¶
type SortDomain ¶
type SortDomain uint
SortDomain an enum for the understood domains
const ( SortByClaimed SortDomain SortByReceived SortByFeedSeq )
The known domains are: Claimed timestamp, Received Timestamp and Sequence number of the messageon the feed (this is important for partial replication, where feeds are not fetched in full and correct order)
type SortedAscending ¶
type SortedAscending struct{ SortedSeqSlice }
SortedAscending wraps around SortedSeqSlice to give it a Less that sorts values from small to large.
type SortedDescending ¶
type SortedDescending struct{ SortedSeqSlice }
SortedDescending wraps around SortedSeqSlice to give it a Less that sorts values from large to small.
type SortedSeqSlice ¶
type SortedSeqSlice []SortedSequence
SortedSeqSlice a slice of SortedSequences that can be sorted
func (SortedSeqSlice) AsLuigiSource ¶
func (ts SortedSeqSlice) AsLuigiSource() luigi.Source
AsLuigiSource returns a luigi.Source to iterate over the sorted array. Helpful for retrofitting into existing margaret code.
func (SortedSeqSlice) Len ¶
func (ts SortedSeqSlice) Len() int
func (SortedSeqSlice) Swap ¶
func (ts SortedSeqSlice) Swap(i int, j int)
Swap swaps the elements with indexes i and j.
type SortedSequence ¶
type SortedSequence struct {
By int64 // fill with the value of the domain
Seq int64 // the sequence of the entry we are looking for
}
SortedSequence holds the sequence value of the message and the domain value it should be sorted by.
Source Files