Documentation
¶
Index ¶
- func Compact(filenames []string, path string, recordsPerSst int, keysPerSegment int, ...) (string, error)
- func Create(filename string, keys []string, m map[string]SstEntry, seqNum uint64)
- func DumpBin(filename string)
- func DumpIndex(filename string)
- func Filenames(path string) []string
- func Find(key string, lvl []SstLevel, path string) ([]byte, bool)
- func Levels(path string) []string
- func Load(filename string) ([]SstEntry, SstFileHeader)
- func NextFilename(path string) string
- func PathForLevel(base string, level int) string
- func Remove(filename string)
- func RemoveAll(path string)
- type SstEntry
- type SstFile
- type SstFileHeader
- type SstHeap
- type SstHeapNode
- type SstIndex
- type SstIndexData
- type SstLevel
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Compact ¶
func Compact(filenames []string, path string, recordsPerSst int, keysPerSegment int, removeDeleted bool) (string, error)
Compact performs a k-way merge of data from the given SST files under the given path.
The recordsPerSst parameter determines the maximum number of records written to each new SST file.
The removeDeleted parameter indicates whether tombstones will be permanently removed (true) or carried through to the new SST files (false).
Data must be sorted within each input SST file. That data is streamed from each file one entry at a time and added to a min heap. We then read from the heap to get the next sorted element and write it out to a new SST file.
This process is performed for all input data, generating a new set of SST files containing sorted and non-overlapping data.
Thus we can handle large files as only a small portion of data is kept in memory at once.
If there are any duplicate keys, only the most recent entry (IE largest sequence number) is written. Note this is only applicable to SST level 0 which contains SST files that may contain overlapping data.
func Create ¶
TODO: input is name of .bin file. write that and corresponding .index file Create creates a new SST file from given data
func Levels ¶
Levels returns the names of any directories containing consolidated SST files at levels greater than level 0. This implies the data is organized in non-overlapping regions across files at that level.
func Load ¶
func Load(filename string) ([]SstEntry, SstFileHeader)
TODO: input is name of .bin file. read that and corresponding .index file and load into memory
func NextFilename ¶
NextFilename returns the name of the next SST binary file in given directory
func PathForLevel ¶
Types ¶
type SstFile ¶
type SstFile struct {
Filename string
Filter *bloom.Filter
Index []SstIndex
Cache []SstIndexData
}
type SstFileHeader ¶
type SstFileHeader struct {
Seq uint64
}
type SstHeap ¶
type SstHeap []*SstHeapNode
An min-heap of SST entries Provides an easy way to sort large numbers of entries
Source Files