index

package
v0.0.0-...-e951c9a Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 6, 2022 License: AGPL-3.0 Imports: 25 Imported by: 0

Documentation

Index

Constants

View Source
const (
	// MagicIndex 4 bytes at the head of an index file.
	MagicIndex = 0xBAAAD700
	// HeaderLen represents number of bytes reserved of index for header.
	HeaderLen = 5

	// FormatV1 represents 1 version of index.
	FormatV1 = 1
	// FormatV2 represents 2 version of index.
	FormatV2 = 2

	IndexFilename = "index"
)
View Source
const (
	// ShardLabel is a reserved label referencing a cortex shard
	ShardLabel = "__tsdb_shard__"
	// ShardLabelFmt is the fmt of the ShardLabel key.
	ShardLabelFmt = "%d_of_%d"
)

Variables

This section is empty.

Functions

func AllPostingsKey

func AllPostingsKey() (name, value string)

AllPostingsKey returns the label key that is used to store the postings list of all existing IDs.

func ExpandPostings

func ExpandPostings(p Postings) (res []storage.SeriesRef, err error)

ExpandPostings returns the postings expanded as a slice.

func ReadOffsetTable

func ReadOffsetTable(bs ByteSlice, off uint64, f func([]string, uint64, int) error) error

ReadOffsetTable reads an offset table and at the given position calls f for each found entry. If f returns an error it stops decoding and returns the received error.

Types

type ByteSlice

type ByteSlice interface {
	Len() int
	Range(start, end int) []byte
}

ByteSlice abstracts a byte slice.

type ChunkMeta

type ChunkMeta struct {
	Checksum uint32

	MinTime, MaxTime int64

	// Bytes stored, rounded to nearest KB
	KB uint32

	Entries uint32
}

Meta holds information about a chunk of data.

func (ChunkMeta) Bounds

func (c ChunkMeta) Bounds() (model.Time, model.Time)

func (ChunkMeta) From

func (c ChunkMeta) From() model.Time

func (ChunkMeta) Through

func (c ChunkMeta) Through() model.Time

type ChunkMetas

type ChunkMetas []ChunkMeta

func (ChunkMetas) Add

func (c ChunkMetas) Add(chk ChunkMeta) ChunkMetas

Add adds ChunkMeta at the right place in order. It assumes existing ChunkMetas have already been sorted by using Finalize. There is no chance of a data loss even if the chunks are not sorted because the chunk would anyways be added so the assumption is relatively safe.

func (ChunkMetas) Bounds

func (c ChunkMetas) Bounds() (mint, maxt model.Time)

func (ChunkMetas) Drop

func (c ChunkMetas) Drop(chk ChunkMeta) (ChunkMetas, bool)

Drop drops ChunkMeta. It assumes existing ChunkMetas have already been sorted by using Finalize. Calling Drop on non-sorted result can result in not dropping the chunk even if it exists. It returns a boolean indicating if the chunk was found and dropped or not so the caller can take appropriate actions if not. ToDo(Sandeep): See if we can do something about the assumption on sorted chunks. Maybe always build ChunkMetas using Add method which should always keep the ChunkMetas deduped and sorted.

func (ChunkMetas) Finalize

func (c ChunkMetas) Finalize() ChunkMetas

Finalize sorts and dedupes TODO(owen-d): can we remove the need for this by ensuring we only push in order and without duplicates?

func (ChunkMetas) Len

func (c ChunkMetas) Len() int

func (ChunkMetas) Less

func (c ChunkMetas) Less(i, j int) bool

Sort by (MinTime, MaxTime, Checksum)

func (ChunkMetas) Swap

func (c ChunkMetas) Swap(i, j int)

type Decoder

type Decoder struct {
	LookupSymbol func(uint32) (string, error)
}

Decoder provides decoding methods for the v1 and v2 index file format.

It currently does not contain decoding methods for all entry types but can be extended by them if there's demand.

func (*Decoder) LabelNamesOffsetsFor

func (dec *Decoder) LabelNamesOffsetsFor(b []byte) ([]uint32, error)

LabelNamesOffsetsFor decodes the offsets of the name symbols for a given series. They are returned in the same order they're stored, which should be sorted lexicographically.

func (*Decoder) LabelValueFor

func (dec *Decoder) LabelValueFor(b []byte, label string) (string, error)

LabelValueFor decodes a label for a given series.

func (*Decoder) Postings

func (dec *Decoder) Postings(b []byte) (int, Postings, error)

Postings returns a postings list for b and its number of elements.

func (*Decoder) Series

func (dec *Decoder) Series(b []byte, lbls *labels.Labels, chks *[]ChunkMeta) (uint64, error)

Series decodes a series entry from the given byte slice into lset and chks.

type FileWriter

type FileWriter struct {
	// contains filtered or unexported fields
}

func NewFileWriter

func NewFileWriter(name string) (*FileWriter, error)

func (*FileWriter) AddPadding

func (fw *FileWriter) AddPadding(size int) error

AddPadding adds zero byte padding until the file size is a multiple size.

func (*FileWriter) Close

func (fw *FileWriter) Close() error

func (*FileWriter) Flush

func (fw *FileWriter) Flush() error

func (*FileWriter) Pos

func (fw *FileWriter) Pos() uint64

func (*FileWriter) Remove

func (fw *FileWriter) Remove() error

func (*FileWriter) Write

func (fw *FileWriter) Write(bufs ...[]byte) error

func (*FileWriter) WriteAt

func (fw *FileWriter) WriteAt(buf []byte, pos uint64) error

type FingerprintOffsets

type FingerprintOffsets [][2]uint64

(SeriesRef, Fingerprint) tuples

func (FingerprintOffsets) Range

func (xs FingerprintOffsets) Range(shard ShardAnnotation) (minOffset, maxOffset uint64)

type ListPostings

type ListPostings struct {
	// contains filtered or unexported fields
}

ListPostings implements the Postings interface over a plain list.

func (*ListPostings) At

func (it *ListPostings) At() storage.SeriesRef

func (*ListPostings) Err

func (it *ListPostings) Err() error

func (*ListPostings) Next

func (it *ListPostings) Next() bool

func (*ListPostings) Seek

func (it *ListPostings) Seek(x storage.SeriesRef) bool

type MemPostings

type MemPostings struct {
	// contains filtered or unexported fields
}

MemPostings holds postings list for series ID per label pair. They may be written to out of order. EnsureOrder() must be called once before any reads are done. This allows for quick unordered batch fills on startup.

func NewMemPostings

func NewMemPostings() *MemPostings

NewMemPostings returns a memPostings that's ready for reads and writes.

func NewUnorderedMemPostings

func NewUnorderedMemPostings() *MemPostings

NewUnorderedMemPostings returns a memPostings that is not safe to be read from until EnsureOrder() was called once.

func (*MemPostings) Add

func (p *MemPostings) Add(id storage.SeriesRef, lset labels.Labels)

Add a label set to the postings index.

func (*MemPostings) All

func (p *MemPostings) All() Postings

All returns a postings list over all documents ever added.

func (*MemPostings) Delete

func (p *MemPostings) Delete(deleted map[storage.SeriesRef]struct{})

Delete removes all ids in the given map from the postings lists.

func (*MemPostings) EnsureOrder

func (p *MemPostings) EnsureOrder()

EnsureOrder ensures that all postings lists are sorted. After it returns all further calls to add and addFor will insert new IDs in a sorted manner.

func (*MemPostings) Get

func (p *MemPostings) Get(name, value string) Postings

Get returns a postings list for the given label pair.

func (*MemPostings) Iter

func (p *MemPostings) Iter(f func(labels.Label, Postings) error) error

Iter calls f for each postings list. It aborts if f returns an error and returns it.

func (*MemPostings) LabelNames

func (p *MemPostings) LabelNames() []string

LabelNames returns all the unique label names.

func (*MemPostings) LabelValues

func (p *MemPostings) LabelValues(name string) []string

LabelValues returns label values for the given name.

func (*MemPostings) SortedKeys

func (p *MemPostings) SortedKeys() []labels.Label

SortedKeys returns a list of sorted label keys of the postings.

func (*MemPostings) Stats

func (p *MemPostings) Stats(label string) *PostingsStats

Stats calculates the cardinality statistics from postings.

func (*MemPostings) Symbols

func (p *MemPostings) Symbols() StringIter

Symbols returns an iterator over all unique name and value strings, in order.

type Metadata

type Metadata struct {
	From, Through int64
	Checksum      uint32
}

Metadata is TSDB-level metadata

func (*Metadata) EnsureBounds

func (m *Metadata) EnsureBounds(from, through int64)

type PoolChunkMetas

type PoolChunkMetas struct {
	// contains filtered or unexported fields
}
var ChunkMetasPool PoolChunkMetas

func (*PoolChunkMetas) Get

func (p *PoolChunkMetas) Get() []ChunkMeta

func (*PoolChunkMetas) Put

func (p *PoolChunkMetas) Put(xs []ChunkMeta)

type Postings

type Postings interface {
	// Next advances the iterator and returns true if another value was found.
	Next() bool

	// Seek advances the iterator to value v or greater and returns
	// true if a value was found.
	Seek(v storage.SeriesRef) bool

	// At returns the value at the current iterator position.
	At() storage.SeriesRef

	// Err returns the last error of the iterator.
	Err() error
}

Postings provides iterative access over a postings list.

func EmptyPostings

func EmptyPostings() Postings

EmptyPostings returns a postings list that's always empty. NOTE: Returning EmptyPostings sentinel when Postings struct has no postings is recommended. It triggers optimized flow in other functions like Intersect, Without etc.

func ErrPostings

func ErrPostings(err error) Postings

ErrPostings returns new postings that immediately error.

func Intersect

func Intersect(its ...Postings) Postings

Intersect returns a new postings list over the intersection of the input postings.

func Merge

func Merge(its ...Postings) Postings

Merge returns a new iterator over the union of the input iterators.

func NewListPostings

func NewListPostings(list []storage.SeriesRef) Postings

func Without

func Without(full, drop Postings) Postings

Without returns a new postings list that contains all elements from the full list that are not in the drop list.

type PostingsStats

type PostingsStats struct {
	CardinalityMetricsStats []Stat
	CardinalityLabelStats   []Stat
	LabelValueStats         []Stat
	LabelValuePairsStats    []Stat
	NumLabelPairs           int
}

PostingsStats contains cardinality based statistics for postings.

type Range

type Range struct {
	Start, End int64
}

Range marks a byte range.

type Reader

type Reader struct {
	// contains filtered or unexported fields
}

func NewFileReader

func NewFileReader(path string) (*Reader, error)

NewFileReader returns a new index reader against the given index file.

func NewReader

func NewReader(b ByteSlice) (*Reader, error)

NewReader returns a new index reader on the given byte slice. It automatically handles different format versions.

func (*Reader) Bounds

func (r *Reader) Bounds() (int64, int64)

func (*Reader) Checksum

func (r *Reader) Checksum() uint32

func (*Reader) Close

func (r *Reader) Close() error

Close the reader and its underlying resources.

func (*Reader) LabelNames

func (r *Reader) LabelNames(matchers ...*labels.Matcher) ([]string, error)

LabelNames returns all the unique label names present in the index. TODO(twilkie) implement support for matchers

func (*Reader) LabelNamesFor

func (r *Reader) LabelNamesFor(ids ...storage.SeriesRef) ([]string, error)

LabelNamesFor returns all the label names for the series referred to by IDs. The names returned are sorted.

func (*Reader) LabelValueFor

func (r *Reader) LabelValueFor(id storage.SeriesRef, label string) (string, error)

LabelValueFor returns label value for the given label name in the series referred to by ID.

func (*Reader) LabelValues

func (r *Reader) LabelValues(name string, matchers ...*labels.Matcher) ([]string, error)

LabelValues returns value tuples that exist for the given label name. It is not safe to use the return value beyond the lifetime of the byte slice passed into the Reader. TODO(replay): Support filtering by matchers

func (*Reader) Postings

func (r *Reader) Postings(name string, shard *ShardAnnotation, values ...string) (Postings, error)

func (*Reader) PostingsRanges

func (r *Reader) PostingsRanges() (map[labels.Label]Range, error)

PostingsRanges returns a new map of byte range in the underlying index file for all postings lists.

func (*Reader) RawFileReader

func (r *Reader) RawFileReader() (io.ReadSeeker, error)

func (*Reader) Series

func (r *Reader) Series(id storage.SeriesRef, lbls *labels.Labels, chks *[]ChunkMeta) (uint64, error)

Series reads the series with the given ID and writes its labels and chunks into lbls and chks.

func (*Reader) Size

func (r *Reader) Size() int64

Size returns the size of an index file.

func (*Reader) SortedLabelValues

func (r *Reader) SortedLabelValues(name string, matchers ...*labels.Matcher) ([]string, error)

SortedLabelValues returns value tuples that exist for the given label name. It is not safe to use the return value beyond the lifetime of the byte slice passed into the Reader.

func (*Reader) SymbolTableSize

func (r *Reader) SymbolTableSize() uint64

SymbolTableSize returns the symbol table size in bytes.

func (*Reader) Symbols

func (r *Reader) Symbols() StringIter

Symbols returns an iterator over the symbols that exist within the index.

func (*Reader) Version

func (r *Reader) Version() int

Version returns the file format version of the underlying index.

type RealByteSlice

type RealByteSlice []byte

func (RealByteSlice) Len

func (b RealByteSlice) Len() int

func (RealByteSlice) Range

func (b RealByteSlice) Range(start, end int) []byte

func (RealByteSlice) Sub

func (b RealByteSlice) Sub(start, end int) ByteSlice

type ShardAnnotation

type ShardAnnotation struct {
	Shard uint32
	Of    uint32
}

ShardAnnotation is a convenience struct which holds data from a parsed shard label Of MUST be a power of 2 to ensure sharding logic works correctly.

func NewShard

func NewShard(x, of uint32) ShardAnnotation

func (ShardAnnotation) Bounds

Bounds shows the [minimum, maximum) fingerprints. If there is no maximum fingerprint (for example the last shard), math.MaxUint64 is used as the maximum.

func (ShardAnnotation) Match

func (shard ShardAnnotation) Match(fp model.Fingerprint) bool

Match returns whether a fingerprint belongs to a certain shard. The Shard must be a power of 2. Inclusion in a shard is calculated by determining the arbitrary bit prefix for a shard, then ensuring the fingerprint has the same prefix

func (ShardAnnotation) RequiredBits

func (shard ShardAnnotation) RequiredBits() uint64

func (ShardAnnotation) String

func (shard ShardAnnotation) String() string

String encodes a shardAnnotation into a label value

func (ShardAnnotation) Validate

func (shard ShardAnnotation) Validate() error

type ShardedPostings

type ShardedPostings struct {
	// contains filtered or unexported fields
}

func NewShardedPostings

func NewShardedPostings(p Postings, shard ShardAnnotation, offsets FingerprintOffsets) *ShardedPostings

Note: shardedPostings can technically return more series than just those that correspond to the requested shard. This is because we do fingperint/offset sampling in TSDB so we won't know exactly which offsets to start/end at, but will likely buffer a little on each end, so they still need to be checked for shard inclusiveness. For example (below), given a shard, we'll likely return a slight superset of offsets surrounding the shard. ---[shard0]--- # Shard membership -[--shard0--]- # Series returned by shardedPostings

func (*ShardedPostings) At

At returns the value at the current iterator position.

func (*ShardedPostings) Err

func (sp *ShardedPostings) Err() (err error)

Err returns the last error of the iterator.

func (*ShardedPostings) Next

func (sp *ShardedPostings) Next() bool

Next advances the iterator and returns true if another value was found.

func (*ShardedPostings) Seek

func (sp *ShardedPostings) Seek(v storage.SeriesRef) (res bool)

Seek advances the iterator to value v or greater and returns true if a value was found.

type Stat

type Stat struct {
	Name  string
	Count uint64
}

Stat holds values for a single cardinality statistic.

type StringIter

type StringIter interface {
	// Next advances the iterator and returns true if another value was found.
	Next() bool

	// At returns the value at the current iterator position.
	At() string

	// Err returns the last error of the iterator.
	Err() error
}

StringIter iterates over a sorted list of strings.

func NewStringListIter

func NewStringListIter(s []string) StringIter

NewStringListIter returns a StringIter for the given sorted list of strings.

type Symbols

type Symbols struct {
	// contains filtered or unexported fields
}

func NewSymbols

func NewSymbols(bs ByteSlice, version, off int) (*Symbols, error)

NewSymbols returns a Symbols object for symbol lookups.

func (Symbols) Iter

func (s Symbols) Iter() StringIter

func (Symbols) Lookup

func (s Symbols) Lookup(o uint32) (string, error)

func (Symbols) ReverseLookup

func (s Symbols) ReverseLookup(sym string) (uint32, error)

func (Symbols) Size

func (s Symbols) Size() int

type TOC

type TOC struct {
	Symbols            uint64
	Series             uint64
	LabelIndices       uint64
	LabelIndicesTable  uint64
	Postings           uint64
	PostingsTable      uint64
	FingerprintOffsets uint64
	Metadata           Metadata
}

TOC represents index Table Of Content that states where each section of index starts.

func NewTOCFromByteSlice

func NewTOCFromByteSlice(bs ByteSlice) (*TOC, error)

NewTOCFromByteSlice return parsed TOC from given index byte slice.

type Writer

type Writer struct {
	Version int
	// contains filtered or unexported fields
}

Writer implements the IndexWriter interface for the standard serialization format.

func NewWriter

func NewWriter(ctx context.Context, fn string) (*Writer, error)

NewWriter returns a new Writer to the given filename. It serializes data in format version 2.

func (*Writer) AddSeries

func (w *Writer) AddSeries(ref storage.SeriesRef, lset labels.Labels, fp model.Fingerprint, chunks ...ChunkMeta) error

AddSeries adds the series one at a time along with its chunks. Requires a specific fingerprint to be passed in the case where the "desired" fingerprint differs from what labels.Hash() produces. For example, multitenant TSDBs embed a tenant label, but the actual series has no such label and so the derived fingerprint differs.

func (*Writer) AddSymbol

func (w *Writer) AddSymbol(sym string) error

func (*Writer) Close

func (w *Writer) Close() error

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL