tsi1

package
v1.2.3-0...-3d68b7a Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 17, 2017 License: MIT Imports: 30 Imported by: 0

Documentation

Overview

Package tsi1 provides a memory-mapped index implementation that supports high cardinality series.

Overview

The top-level object in tsi1 is the Index. It is the primary access point from the rest of the system. The Index is composed of LogFile and IndexFile objects.

Log files are small write-ahead log files that record new series immediately in the order that they are received. The data within the file is indexed in-memory so it can be quickly accessed. When the system is restarted, this log file is replayed and the in-memory representation is rebuilt.

Index files also contain series information, however, they are highly indexed so that reads can be performed quickly. Index files are built through a process called compaction where a log file or multiple index files are merged together.

Operations

The index can perform many tasks related to series, measurement, & tag data. All data is inserted by adding a series to the index. When adding a series, the measurement, tag keys, and tag values are all extracted and indexed separately.

Once a series has been added, it can be removed in several ways. First, the individual series can be removed. Second, it can be removed as part of a bulk operation by deleting the entire measurement.

The query engine needs to be able to look up series in a variety of ways such as by measurement name, by tag value, or by using regular expressions. The index provides an API to iterate over subsets of series and perform set operations such as unions and intersections.

Log File Layout

The write-ahead file that series initially are inserted into simply appends all new operations sequentially. It is simply composed of a series of log entries. An entry contains a flag to specify the operation type, the measurement name, the tag set, and a checksum.

┏━━━━━━━━━LogEntry━━━━━━━━━┓
┃ ┌──────────────────────┐ ┃
┃ │         Flag         │ ┃
┃ ├──────────────────────┤ ┃
┃ │     Measurement      │ ┃
┃ ├──────────────────────┤ ┃
┃ │      Key/Value       │ ┃
┃ ├──────────────────────┤ ┃
┃ │      Key/Value       │ ┃
┃ ├──────────────────────┤ ┃
┃ │      Key/Value       │ ┃
┃ ├──────────────────────┤ ┃
┃ │       Checksum       │ ┃
┃ └──────────────────────┘ ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━┛

When the log file is replayed, if the checksum is incorrect or the entry is incomplete (because of a partially failed write) then the log is truncated.

Index File Layout

The index file is composed of 3 main block types: one series block, one or more tag blocks, and one measurement block. At the end of the index file is a trailer that records metadata such as the offsets to these blocks.

Series Block Layout

The series block stores raw series keys in sorted order. It also provides hash indexes so that series can be looked up quickly. Hash indexes are inserted periodically so that memory size is limited at write time. Once all the series and hash indexes have been written then a list of index entries are written so that hash indexes can be looked up via binary search.

The end of the block contains two HyperLogLog++ sketches which track the estimated number of created series and deleted series. After the sketches is a trailer which contains metadata about the block.

┏━━━━━━━SeriesBlock━━━━━━━━┓
┃ ┌──────────────────────┐ ┃
┃ │      Series Key      │ ┃
┃ ├──────────────────────┤ ┃
┃ │      Series Key      │ ┃
┃ ├──────────────────────┤ ┃
┃ │      Series Key      │ ┃
┃ ├──────────────────────┤ ┃
┃ │                      │ ┃
┃ │      Hash Index      │ ┃
┃ │                      │ ┃
┃ ├──────────────────────┤ ┃
┃ │      Series Key      │ ┃
┃ ├──────────────────────┤ ┃
┃ │      Series Key      │ ┃
┃ ├──────────────────────┤ ┃
┃ │      Series Key      │ ┃
┃ ├──────────────────────┤ ┃
┃ │                      │ ┃
┃ │      Hash Index      │ ┃
┃ │                      │ ┃
┃ ├──────────────────────┤ ┃
┃ │    Index Entries     │ ┃
┃ ├──────────────────────┤ ┃
┃ │     HLL Sketches     │ ┃
┃ ├──────────────────────┤ ┃
┃ │       Trailer        │ ┃
┃ └──────────────────────┘ ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━┛

Tag Block Layout

After the series block is one or more tag blocks. One of these blocks exists for every measurement in the index file. The block is structured as a sorted list of values for each key and then a sorted list of keys. Each of these lists has their own hash index for fast direct lookups.

┏━━━━━━━━Tag Block━━━━━━━━━┓
┃ ┌──────────────────────┐ ┃
┃ │        Value         │ ┃
┃ ├──────────────────────┤ ┃
┃ │        Value         │ ┃
┃ ├──────────────────────┤ ┃
┃ │        Value         │ ┃
┃ ├──────────────────────┤ ┃
┃ │                      │ ┃
┃ │      Hash Index      │ ┃
┃ │                      │ ┃
┃ └──────────────────────┘ ┃
┃ ┌──────────────────────┐ ┃
┃ │        Value         │ ┃
┃ ├──────────────────────┤ ┃
┃ │        Value         │ ┃
┃ ├──────────────────────┤ ┃
┃ │                      │ ┃
┃ │      Hash Index      │ ┃
┃ │                      │ ┃
┃ └──────────────────────┘ ┃
┃ ┌──────────────────────┐ ┃
┃ │         Key          │ ┃
┃ ├──────────────────────┤ ┃
┃ │         Key          │ ┃
┃ ├──────────────────────┤ ┃
┃ │                      │ ┃
┃ │      Hash Index      │ ┃
┃ │                      │ ┃
┃ └──────────────────────┘ ┃
┃ ┌──────────────────────┐ ┃
┃ │       Trailer        │ ┃
┃ └──────────────────────┘ ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━┛

Each entry for values contains a sorted list of offsets for series keys that use that value. Series iterators can be built around a single tag key value or multiple iterators can be merged with set operators such as union or intersection.

Measurement block

The measurement block stores a sorted list of measurements, their associated series offsets, and the offset to their tag block. This allows all series for a measurement to be traversed quickly and it allows fast direct lookups of measurements and their tags.

This block also contains HyperLogLog++ sketches for new and deleted measurements.

┏━━━━Measurement Block━━━━━┓
┃ ┌──────────────────────┐ ┃
┃ │     Measurement      │ ┃
┃ ├──────────────────────┤ ┃
┃ │     Measurement      │ ┃
┃ ├──────────────────────┤ ┃
┃ │     Measurement      │ ┃
┃ ├──────────────────────┤ ┃
┃ │                      │ ┃
┃ │      Hash Index      │ ┃
┃ │                      │ ┃
┃ ├──────────────────────┤ ┃
┃ │     HLL Sketches     │ ┃
┃ ├──────────────────────┤ ┃
┃ │       Trailer        │ ┃
┃ └──────────────────────┘ ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━┛

Manifest file

The index is simply an ordered set of log and index files. These files can be merged together or rewritten but their order must always be the same. This is because series, measurements, & tags can be marked as deleted (aka tombstoned) and this action needs to be tracked in time order.

Whenever the set of active files is changed, a manifest file is written to track the set. The manifest specifies the ordering of files and, on startup, all files not in the manifest are removed from the index directory.

Compacting index files

Compaction is the process of taking files and merging them together into a single file. There are two stages of compaction within TSI.

First, once log files exceed a size threshold then they are compacted into an index file. This threshold is relatively small because log files must maintain their index in the heap which TSI tries to avoid. Small log files are also very quick to convert into an index file so this is done aggressively.

Second, once a contiguous set of index files exceed a factor (e.g. 10x) then they are all merged together into a single index file and the old files are discarded. Because all blocks are written in sorted order, the new index file can be streamed and minimize memory use.

Concurrency

Index files are immutable so they do not require fine grained locks, however, compactions require that we track which files are in use so they are not discarded too soon. This is done by using reference counting with file sets.

A file set is simply an ordered list of index files. When the current file set is obtained from the index, a counter is incremented to track its usage. Once the user is done with the file set, it is released and the counter is decremented. A file cannot be removed from the file system until this counter returns to zero.

Besides the reference counting, there are no other locking mechanisms when reading or writing index files. Log files, however, do require a lock whenever they are accessed. This is another reason to minimize log file size.

Index

Constants

View Source
const (
	// IndexName is the name of the index.
	IndexName = "tsi1"

	// Version is the current version of the TSI index.
	Version = 1
)
View Source
const (
	LogFileExt   = ".tsl"
	IndexFileExt = ".tsi"

	CompactingExt = ".compacting"
)

File extensions.

View Source
const (
	// IndexFile trailer fields
	IndexFileVersionSize       = 2
	SeriesBlockOffsetSize      = 8
	SeriesBlockSizeSize        = 8
	MeasurementBlockOffsetSize = 8
	MeasurementBlockSizeSize   = 8

	IndexFileTrailerSize = IndexFileVersionSize +
		SeriesBlockOffsetSize +
		SeriesBlockSizeSize +
		MeasurementBlockOffsetSize +
		MeasurementBlockSizeSize
)

IndexFile field size constants.

View Source
const (
	LogEntrySeriesTombstoneFlag      = 0x01
	LogEntryMeasurementTombstoneFlag = 0x02
	LogEntryTagKeyTombstoneFlag      = 0x04
	LogEntryTagValueTombstoneFlag    = 0x08
)

Log entry flag constants.

View Source
const (
	// 1 byte offset for the block to ensure non-zero offsets.
	MeasurementFillSize = 1

	// Measurement trailer fields
	MeasurementTrailerSize = 0 +
		2 +
		8 + 8 +
		8 + 8 +
		8 + 8 +
		8 + 8 // tombstone measurement sketch offset/size

	// Measurement key block fields.
	MeasurementNSize      = 8
	MeasurementOffsetSize = 8
)

Measurement field size constants.

View Source
const (
	// Series list trailer field sizes.
	SeriesBlockTrailerSize = 0 +
		4 + 4 +
		4 + 4 + 4 +
		8 + 4 + 4 +
		4 + 4 +
		4 + 4 +
		4 + 4 +
		0

	// Other field sizes
	SeriesCountSize = 4
	SeriesIDSize    = 4
)

Series list field size constants.

View Source
const (
	// Marks the series as having been deleted.
	SeriesTombstoneFlag = 0x01

	// Marks the following bytes as a hash index.
	// These bytes should be skipped by an iterator.
	SeriesHashIndexFlag = 0x02
)

Series flag constants.

View Source
const (
	// TagBlock key block fields.
	TagKeyNSize      = 8
	TagKeyOffsetSize = 8

	// TagBlock value block fields.
	TagValueNSize      = 8
	TagValueOffsetSize = 8
)

TagBlock variable size constants.

View Source
const (
	DefaultMaxLogFileSize = 5 * 1024 * 1024
)

Default compaction thresholds.

View Source
const FileSignature = "TSI1"

FileSignature represents a magic number at the header of the index file.

View Source
const IndexFileVersion = 1

IndexFileVersion is the current TSI1 index file version.

View Source
const LoadFactor = 80

LoadFactor is the fill percent for RHH indexes.

View Source
const ManifestFileName = "MANIFEST"

ManifestFileName is the name of the index manifest file.

View Source
const MaxIndexFileSize = 4 * (1 << 30)

MaxIndexFileSize is the maximum expected size of an index file.

View Source
const MaxIndexMergeCount = 2

MaxIndexMergeCount is the maximum number of files that can be merged together at once.

View Source
const MaxSeriesBlockHashSize = (1048576 * LoadFactor) / 100

MaxSeriesBlockHashSize is the maximum number of series in a single hash.

View Source
const MeasurementBlockVersion = 1

MeasurementBlockVersion is the version of the measurement block.

View Source
const (
	MeasurementTombstoneFlag = 0x01
)

Measurement flag constants.

View Source
const TagBlockTrailerSize = 0 +
	8 + 8 +
	8 + 8 +
	8 + 8 +
	8 +

	2 // version

TagBlockTrailerSize is the total size of the on-disk trailer.

View Source
const TagBlockVersion = 1

TagBlockVersion is the version of the tag block.

View Source
const (
	TagKeyTombstoneFlag = 0x01
)

Tag key flag constants.

View Source
const (
	TagValueTombstoneFlag = 0x01
)

Tag value flag constants.

Variables

View Source
var (
	ErrInvalidIndexFile            = errors.New("invalid index file")
	ErrUnsupportedIndexFileVersion = errors.New("unsupported index file version")
)

IndexFile errors.

View Source
var (
	ErrUnsupportedMeasurementBlockVersion = errors.New("unsupported measurement block version")
	ErrMeasurementBlockSizeMismatch       = errors.New("measurement block size mismatch")
)

Measurement errors.

View Source
var (
	ErrUnsupportedTagBlockVersion = errors.New("unsupported tag block version")
	ErrTagBlockSizeMismatch       = errors.New("tag block size mismatch")
)

TagBlock errors.

View Source
var DefaultCompactionLevels = []CompactionLevel{
	{M: 0, K: 0},
	{M: 1 << 25, K: 6},
	{M: 1 << 25, K: 6},
	{M: 1 << 26, K: 6},
	{M: 1 << 27, K: 6},
	{M: 1 << 28, K: 6},
	{M: 1 << 29, K: 6},
	{M: 1 << 30, K: 6},
}

DefaultCompactionLevels is the default settings used by the index.

View Source
var ErrIncompatibleVersion = errors.New("incompatible tsi1 index MANIFEST")

ErrIncompatibleVersion is returned when attempting to read from an incompatible tsi1 manifest file.

View Source
var (
	ErrLogEntryChecksumMismatch = errors.New("log entry checksum mismatch")
)

Log errors.

View Source
var ErrSeriesOverflow = errors.New("series overflow")

ErrSeriesOverflow is returned when too many series are added to a series writer.

Functions

func AppendSeriesElem

func AppendSeriesElem(dst []byte, flag byte, name []byte, tags models.Tags) []byte

AppendSeriesElem serializes flag/name/tags to dst and returns the new buffer.

func AppendSeriesKey

func AppendSeriesKey(dst []byte, name []byte, tags models.Tags) []byte

AppendSeriesKey serializes name and tags to a byte slice. The total length is prepended as a uvarint.

func CompareSeriesElem

func CompareSeriesElem(a, b tsdb.SeriesElem) int

CompareSeriesElem returns -1 if a < b, 1 if a > b, and 0 if equal.

func CompareSeriesKeys

func CompareSeriesKeys(a, b []byte) int

func DifferenceSeriesIterators

func DifferenceSeriesIterators(itr0, itr1 tsdb.SeriesIterator) tsdb.SeriesIterator

DifferenceSeriesIterators returns an iterator that only returns series which occur the first iterator but not the second iterator.

func FilterUndeletedSeriesIterator

func FilterUndeletedSeriesIterator(itr tsdb.SeriesIterator) tsdb.SeriesIterator

FilterUndeletedSeriesIterator returns an iterator which filters all deleted series.

func FormatIndexFileName

func FormatIndexFileName(id, level int) string

FormatIndexFileName generates an index filename for the given index.

func FormatLogFileName

func FormatLogFileName(id int) string

FormatLogFileName generates a log filename for the given index.

func IntersectSeriesIterators

func IntersectSeriesIterators(itr0, itr1 tsdb.SeriesIterator) tsdb.SeriesIterator

IntersectSeriesIterators returns an iterator that only returns series which occur in both iterators. If both series have associated expressions then they are combined together.

func MergeSeriesIterators

func MergeSeriesIterators(itrs ...tsdb.SeriesIterator) tsdb.SeriesIterator

MergeSeriesIterators returns an iterator that merges a set of iterators. Iterators that are first in the list take precendence and a deletion by those early iterators will invalidate elements by later iterators.

func ParseFilename

func ParseFilename(name string) (level, id int)

ParseFilename extracts the numeric id from a log or index file path. Returns 0 if it cannot be parsed.

func ReadSeriesKey

func ReadSeriesKey(data []byte) []byte

ReadSeriesKey returns the series key from the beginning of the buffer.

func SeriesElemKey

func SeriesElemKey(e tsdb.SeriesElem) []byte

SeriesElemKey encodes e as a series key.

func UnionSeriesIterators

func UnionSeriesIterators(itr0, itr1 tsdb.SeriesIterator) tsdb.SeriesIterator

UnionSeriesIterators returns an iterator that returns series from both both iterators. If both series have associated expressions then they are combined together.

func WriteManifestFile

func WriteManifestFile(path string, m *Manifest) error

WriteManifestFile writes a manifest to a file path.

Types

type CompactionLevel

type CompactionLevel struct {
	// Bloom filter bit size & hash count
	M uint64 `json:"m,omitempty"`
	K uint64 `json:"k,omitempty"`
}

CompactionLevel represents a grouping of index files based on bloom filter settings. By having the same bloom filter settings, the filters can be merged and evaluated at a higher level.

type File

type File interface {
	Close() error
	Path() string

	ID() int
	Level() int

	Measurement(name []byte) MeasurementElem
	MeasurementIterator() MeasurementIterator
	HasSeries(name []byte, tags models.Tags, buf []byte) (exists, tombstoned bool)
	Series(name []byte, tags models.Tags) tsdb.SeriesElem
	SeriesN() uint64

	TagKey(name, key []byte) TagKeyElem
	TagKeyIterator(name []byte) TagKeyIterator

	TagValue(name, key, value []byte) TagValueElem
	TagValueIterator(name, key []byte) TagValueIterator

	// Series iteration.
	SeriesIterator() tsdb.SeriesIterator
	MeasurementSeriesIterator(name []byte) tsdb.SeriesIterator
	TagKeySeriesIterator(name, key []byte) tsdb.SeriesIterator
	TagValueSeriesIterator(name, key, value []byte) tsdb.SeriesIterator

	// Sketches for cardinality estimation
	MergeSeriesSketches(s, t estimator.Sketch) error
	MergeMeasurementsSketches(s, t estimator.Sketch) error

	// Series existence bloom filter.
	Filter() *bloom.Filter

	// Reference counting.
	Retain()
	Release()
}

File represents a log or index file.

type FileSet

type FileSet struct {
	// contains filtered or unexported fields
}

FileSet represents a collection of files.

func NewFileSet

func NewFileSet(database string, levels []CompactionLevel, files []File) (*FileSet, error)

NewFileSet returns a new instance of FileSet.

func (FileSet) Close

func (p FileSet) Close() error

Close closes all the files in the file set.

func (*FileSet) Files

func (fs *FileSet) Files() []File

Files returns all files in the set.

func (*FileSet) FilterNamesTags

func (fs *FileSet) FilterNamesTags(names [][]byte, tagsSlice []models.Tags) ([][]byte, []models.Tags)

FilterNamesTags filters out any series which already exist. It modifies the provided slices of names and tags.

func (*FileSet) HasSeries

func (fs *FileSet) HasSeries(name []byte, tags models.Tags, buf []byte) bool

HasSeries returns true if the series exists and is not tombstoned.

func (*FileSet) HasTagKey

func (fs *FileSet) HasTagKey(name, key []byte) bool

HasTagKey returns true if the tag key exists.

func (*FileSet) HasTagValue

func (fs *FileSet) HasTagValue(name, key, value []byte) bool

HasTagValue returns true if the tag value exists.

func (*FileSet) IndexFiles

func (fs *FileSet) IndexFiles() []*IndexFile

IndexFiles returns all index files from the file set.

func (*FileSet) LastContiguousIndexFilesByLevel

func (fs *FileSet) LastContiguousIndexFilesByLevel(level int) []*IndexFile

LastContiguousIndexFilesByLevel returns the last contiguous files by level. These can be used by the compaction scheduler.

func (*FileSet) LogFiles

func (fs *FileSet) LogFiles() []*LogFile

LogFiles returns all log files from the file set.

func (*FileSet) MatchTagValueSeriesIterator

func (fs *FileSet) MatchTagValueSeriesIterator(name, key []byte, value *regexp.Regexp, matches bool) tsdb.SeriesIterator

MatchTagValueSeriesIterator returns a series iterator for tags which match value. If matches is false, returns iterators which do not match value.

func (*FileSet) MaxID

func (fs *FileSet) MaxID() int

MaxID returns the highest file identifier.

func (*FileSet) Measurement

func (fs *FileSet) Measurement(name []byte) MeasurementElem

Measurement returns a measurement by name.

func (*FileSet) MeasurementIterator

func (fs *FileSet) MeasurementIterator() MeasurementIterator

MeasurementIterator returns an iterator over all measurements in the index.

func (*FileSet) MeasurementNamesByExpr

func (fs *FileSet) MeasurementNamesByExpr(expr influxql.Expr) ([][]byte, error)

func (*FileSet) MeasurementSeriesByExprIterator

func (fs *FileSet) MeasurementSeriesByExprIterator(name []byte, expr influxql.Expr, fieldset *tsdb.MeasurementFieldSet) (tsdb.SeriesIterator, error)

MeasurementSeriesByExprIterator returns a series iterator for a measurement that is filtered by expr. If expr only contains time expressions then this call is equivalent to MeasurementSeriesIterator().

func (*FileSet) MeasurementSeriesIterator

func (fs *FileSet) MeasurementSeriesIterator(name []byte) tsdb.SeriesIterator

MeasurementSeriesIterator returns an iterator over all non-tombstoned series in the index for the provided measurement.

func (*FileSet) MeasurementSeriesKeysByExpr

func (fs *FileSet) MeasurementSeriesKeysByExpr(name []byte, expr influxql.Expr, fieldset *tsdb.MeasurementFieldSet) ([][]byte, error)

MeasurementSeriesKeysByExpr returns a list of series keys matching expr.

func (*FileSet) MeasurementTagKeysByExpr

func (fs *FileSet) MeasurementTagKeysByExpr(name []byte, expr influxql.Expr) (map[string]struct{}, error)

MeasurementTagKeysByExpr extracts the tag keys wanted by the expression.

func (*FileSet) MeasurementsSketches

func (fs *FileSet) MeasurementsSketches() (estimator.Sketch, estimator.Sketch, error)

MeasurementsSketches returns the merged measurement sketches for the FileSet.

func (*FileSet) MustReplace

func (fs *FileSet) MustReplace(oldFiles []File, newFile File) *FileSet

MustReplace swaps a list of files for a single file and returns a new file set. The caller should always guarentee that the files exist and are contiguous.

func (*FileSet) PrependLogFile

func (fs *FileSet) PrependLogFile(f *LogFile) *FileSet

PrependLogFile returns a new file set with f added at the beginning. Filters do not need to be rebuilt because log files have no bloom filter.

func (*FileSet) Release

func (fs *FileSet) Release()

Release removes a reference count from all files.

func (*FileSet) Retain

func (fs *FileSet) Retain()

Retain adds a reference count to all files.

func (*FileSet) SeriesIterator

func (fs *FileSet) SeriesIterator() tsdb.SeriesIterator

SeriesIterator returns an iterator over all series in the index.

func (*FileSet) SeriesSketches

func (fs *FileSet) SeriesSketches() (estimator.Sketch, estimator.Sketch, error)

SeriesSketches returns the merged series sketches for the FileSet.

func (*FileSet) TagKeyIterator

func (fs *FileSet) TagKeyIterator(name []byte) TagKeyIterator

TagKeyIterator returns an iterator over all tag keys for a measurement.

func (*FileSet) TagKeySeriesIterator

func (fs *FileSet) TagKeySeriesIterator(name, key []byte) tsdb.SeriesIterator

TagKeySeriesIterator returns a series iterator for all values across a single key.

func (*FileSet) TagValueIterator

func (fs *FileSet) TagValueIterator(name, key []byte) TagValueIterator

TagValueIterator returns a value iterator for a tag key.

func (*FileSet) TagValueSeriesIterator

func (fs *FileSet) TagValueSeriesIterator(name, key, value []byte) tsdb.SeriesIterator

TagValueSeriesIterator returns a series iterator for a single tag value.

type Files

type Files []File

func (Files) IDs

func (a Files) IDs() []int

type Index

type Index struct {

	// Associated shard info.
	ShardID uint64

	// Name of database.
	Database string

	// Root directory of the index files.
	Path string

	// Log file compaction thresholds.
	MaxLogFileSize int64

	// Frequency of compaction checks.
	CompactionEnabled         bool
	CompactionMonitorInterval time.Duration
	// contains filtered or unexported fields
}

Index represents a collection of layered index files and WAL.

func NewIndex

func NewIndex() *Index

NewIndex returns a new instance of Index.

func (*Index) AssignShard

func (i *Index) AssignShard(k string, shardID uint64)

func (*Index) CheckLogFile

func (i *Index) CheckLogFile() error

func (*Index) Close

func (i *Index) Close() error

Close closes the index.

func (*Index) Compact

func (i *Index) Compact()

Compact requests a compaction of log files.

func (*Index) CreateSeriesIfNotExists

func (i *Index) CreateSeriesIfNotExists(key, name []byte, tags models.Tags) error

CreateSeriesIfNotExists creates a series if it doesn't exist or is deleted.

func (*Index) CreateSeriesListIfNotExists

func (i *Index) CreateSeriesListIfNotExists(_, names [][]byte, tagsSlice []models.Tags) error

CreateSeriesListIfNotExists creates a list of series if they doesn't exist in bulk.

func (*Index) DropMeasurement

func (i *Index) DropMeasurement(name []byte) error

DropMeasurement deletes a measurement from the index.

func (*Index) DropSeries

func (i *Index) DropSeries(key []byte, ts int64) error

func (*Index) FileN

func (i *Index) FileN() int

FileN returns the active files in the file set.

func (*Index) ForEachMeasurementName

func (i *Index) ForEachMeasurementName(fn func(name []byte) error) error

ForEachMeasurementName iterates over all measurement names in the index.

func (*Index) ForEachMeasurementTagKey

func (i *Index) ForEachMeasurementTagKey(name []byte, fn func(key []byte) error) error

ForEachMeasurementTagKey iterates over all tag keys in a measurement.

func (*Index) HasTagKey

func (i *Index) HasTagKey(name, key []byte) (bool, error)

HasTagKey returns true if tag key exists.

func (*Index) InitializeSeries

func (i *Index) InitializeSeries(key, name []byte, tags models.Tags) error

InitializeSeries is a no-op. This only applies to the in-memory index.

func (*Index) Manifest

func (i *Index) Manifest() *Manifest

Manifest returns a manifest for the index.

func (*Index) ManifestPath

func (i *Index) ManifestPath() string

ManifestPath returns the path to the index's manifest file.

func (*Index) MeasurementExists

func (i *Index) MeasurementExists(name []byte) (bool, error)

MeasurementExists returns true if a measurement exists.

func (*Index) MeasurementNamesByExpr

func (i *Index) MeasurementNamesByExpr(expr influxql.Expr) ([][]byte, error)

func (*Index) MeasurementNamesByRegex

func (i *Index) MeasurementNamesByRegex(re *regexp.Regexp) ([][]byte, error)

func (*Index) MeasurementSeriesKeysByExpr

func (i *Index) MeasurementSeriesKeysByExpr(name []byte, expr influxql.Expr) ([][]byte, error)

MeasurementSeriesKeysByExpr returns a list of series keys matching expr.

func (*Index) MeasurementSeriesKeysByExprIterator

func (i *Index) MeasurementSeriesKeysByExprIterator(name []byte, condition influxql.Expr) (tsdb.SeriesIterator, error)

func (*Index) MeasurementTagKeyValuesByExpr

func (i *Index) MeasurementTagKeyValuesByExpr(auth query.Authorizer, name []byte, keys []string, expr influxql.Expr, keysSorted bool) ([][]string, error)

MeasurementTagKeyValuesByExpr returns a set of tag values filtered by an expression.

See tsm1.Engine.MeasurementTagKeyValuesByExpr for a fuller description of this method.

func (*Index) MeasurementTagKeysByExpr

func (i *Index) MeasurementTagKeysByExpr(name []byte, expr influxql.Expr) (map[string]struct{}, error)

MeasurementTagKeysByExpr extracts the tag keys wanted by the expression.

func (*Index) MeasurementsSketches

func (i *Index) MeasurementsSketches() (estimator.Sketch, estimator.Sketch, error)

MeasurementsSketches returns the two sketches for the index by merging all instances of the type sketch types in all the index files.

func (*Index) NextSequence

func (i *Index) NextSequence() int

NextSequence returns the next file identifier.

func (*Index) Open

func (i *Index) Open() error

Open opens the index.

func (*Index) Rebuild

func (i *Index) Rebuild()

func (*Index) RemoveShard

func (i *Index) RemoveShard(shardID uint64)

func (*Index) RetainFileSet

func (i *Index) RetainFileSet() *FileSet

RetainFileSet returns the current fileset and adds a reference count.

func (*Index) SeriesN

func (i *Index) SeriesN() int64

SeriesN returns the number of unique non-tombstoned series in the index. Since indexes are not shared across shards, the count returned by SeriesN cannot be combined with other shard's results. If you need to count series across indexes then use SeriesSketches and merge the results from other indexes.

func (*Index) SeriesPointIterator

func (i *Index) SeriesPointIterator(opt query.IteratorOptions) (query.Iterator, error)

SeriesPointIterator returns an influxql iterator over all series.

func (*Index) SeriesSketches

func (i *Index) SeriesSketches() (estimator.Sketch, estimator.Sketch, error)

SeriesSketches returns the two sketches for the index by merging all instances sketches from TSI files and the WAL.

func (*Index) SetFieldName

func (i *Index) SetFieldName(measurement []byte, name string)

func (*Index) SetFieldSet

func (i *Index) SetFieldSet(fs *tsdb.MeasurementFieldSet)

SetFieldSet sets a shared field set from the engine.

func (*Index) SnapshotTo

func (i *Index) SnapshotTo(path string) error

SnapshotTo creates hard links to the file set into path.

func (*Index) TagKeyCardinality

func (i *Index) TagKeyCardinality(name, key []byte) int

TagKeyCardinality always returns zero. It is not possible to determine cardinality of tags across index files.

func (*Index) TagSets

func (i *Index) TagSets(name []byte, opt query.IteratorOptions) ([]*query.TagSet, error)

TagSets returns an ordered list of tag sets for a measurement by dimension and filtered by an optional conditional expression.

func (*Index) Type

func (i *Index) Type() string

func (*Index) UnassignShard

func (i *Index) UnassignShard(k string, shardID uint64, ts int64) error

func (*Index) Wait

func (i *Index) Wait()

Wait returns once outstanding compactions have finished.

func (*Index) WithLogger

func (i *Index) WithLogger(logger *zap.Logger)

WithLogger sets the logger for the index.

type IndexFile

type IndexFile struct {
	// contains filtered or unexported fields
}

IndexFile represents a collection of measurement, tag, and series data.

func NewIndexFile

func NewIndexFile() *IndexFile

NewIndexFile returns a new instance of IndexFile.

func (*IndexFile) Close

func (f *IndexFile) Close() error

Close unmaps the data file.

func (*IndexFile) Compacting

func (f *IndexFile) Compacting() bool

Compacting returns true if the file is being compacted.

func (*IndexFile) Filter

func (f *IndexFile) Filter() *bloom.Filter

Filter returns the series existence filter for the file.

func (*IndexFile) HasSeries

func (f *IndexFile) HasSeries(name []byte, tags models.Tags, buf []byte) (exists, tombstoned bool)

HasSeries returns flags indicating if the series exists and if it is tombstoned.

func (*IndexFile) ID

func (f *IndexFile) ID() int

ID returns the file sequence identifier.

func (*IndexFile) Level

func (f *IndexFile) Level() int

Level returns the compaction level for the file.

func (*IndexFile) Measurement

func (f *IndexFile) Measurement(name []byte) MeasurementElem

Measurement returns a measurement element.

func (*IndexFile) MeasurementIterator

func (f *IndexFile) MeasurementIterator() MeasurementIterator

MeasurementIterator returns an iterator over all measurements.

func (*IndexFile) MeasurementN

func (f *IndexFile) MeasurementN() (n uint64)

MeasurementN returns the number of measurements in the file.

func (*IndexFile) MeasurementSeriesIterator

func (f *IndexFile) MeasurementSeriesIterator(name []byte) tsdb.SeriesIterator

MeasurementSeriesIterator returns an iterator over a measurement's series.

func (*IndexFile) MergeMeasurementsSketches

func (f *IndexFile) MergeMeasurementsSketches(s, t estimator.Sketch) error

MergeMeasurementsSketches merges the index file's series sketches into the provided sketches.

func (*IndexFile) MergeSeriesSketches

func (f *IndexFile) MergeSeriesSketches(s, t estimator.Sketch) error

MergeSeriesSketches merges the index file's series sketches into the provided sketches.

func (*IndexFile) Open

func (f *IndexFile) Open() error

Open memory maps the data file at the file's path.

func (*IndexFile) Path

func (f *IndexFile) Path() string

Path returns the file path.

func (*IndexFile) Release

func (f *IndexFile) Release()

Release removes a reference count from the file.

func (*IndexFile) Retain

func (f *IndexFile) Retain()

Retain adds a reference count to the file.

func (*IndexFile) Series

func (f *IndexFile) Series(name []byte, tags models.Tags) tsdb.SeriesElem

Series returns the series and a flag indicating if the series has been tombstoned by the measurement.

func (*IndexFile) SeriesIterator

func (f *IndexFile) SeriesIterator() tsdb.SeriesIterator

SeriesIterator returns an iterator over all series.

func (*IndexFile) SeriesN

func (f *IndexFile) SeriesN() uint64

SeriesN returns the total number of non-tombstoned series for the index file.

func (*IndexFile) SetPath

func (f *IndexFile) SetPath(path string)

SetPath sets the file's path.

func (*IndexFile) Size

func (f *IndexFile) Size() int64

Size returns the size of the index file, in bytes.

func (*IndexFile) TagKey

func (f *IndexFile) TagKey(name, key []byte) TagKeyElem

TagKey returns a tag key.

func (*IndexFile) TagKeyIterator

func (f *IndexFile) TagKeyIterator(name []byte) TagKeyIterator

TagKeyIterator returns an iterator over all tag keys for a measurement.

func (*IndexFile) TagKeySeriesIterator

func (f *IndexFile) TagKeySeriesIterator(name, key []byte) tsdb.SeriesIterator

TagKeySeriesIterator returns a series iterator for a tag key and a flag indicating if a tombstone exists on the measurement or key.

func (*IndexFile) TagValue

func (f *IndexFile) TagValue(name, key, value []byte) TagValueElem

TagValue returns a tag value.

func (*IndexFile) TagValueElem

func (f *IndexFile) TagValueElem(name, key, value []byte) TagValueElem

TagValueElem returns an element for a measurement/tag/value.

func (*IndexFile) TagValueIterator

func (f *IndexFile) TagValueIterator(name, key []byte) TagValueIterator

TagValueIterator returns a value iterator for a tag key and a flag indicating if a tombstone exists on the measurement or key.

func (*IndexFile) TagValueSeriesIterator

func (f *IndexFile) TagValueSeriesIterator(name, key, value []byte) tsdb.SeriesIterator

TagValueSeriesIterator returns a series iterator for a tag value and a flag indicating if a tombstone exists on the measurement, key, or value.

func (*IndexFile) UnmarshalBinary

func (f *IndexFile) UnmarshalBinary(data []byte) error

UnmarshalBinary opens an index from data. The byte slice is retained so it must be kept open.

type IndexFileTrailer

type IndexFileTrailer struct {
	Version     int
	SeriesBlock struct {
		Offset int64
		Size   int64
	}
	MeasurementBlock struct {
		Offset int64
		Size   int64
	}
}

IndexFileTrailer represents meta data written to the end of the index file.

func ReadIndexFileTrailer

func ReadIndexFileTrailer(data []byte) (IndexFileTrailer, error)

ReadIndexFileTrailer returns the index file trailer from data.

func (*IndexFileTrailer) WriteTo

func (t *IndexFileTrailer) WriteTo(w io.Writer) (n int64, err error)

WriteTo writes the trailer to w.

type IndexFiles

type IndexFiles []*IndexFile

IndexFiles represents a layered set of index files.

func (IndexFiles) CompactTo

func (p IndexFiles) CompactTo(w io.Writer, m, k uint64) (n int64, err error)

CompactTo merges all index files and writes them to w.

func (IndexFiles) Files

func (p IndexFiles) Files() []File

Files returns p as a list of File objects.

func (IndexFiles) IDs

func (p IndexFiles) IDs() []int

IDs returns the ids for all index files.

func (IndexFiles) MeasurementIterator

func (p IndexFiles) MeasurementIterator() MeasurementIterator

MeasurementIterator returns an iterator that merges measurements across all files.

func (*IndexFiles) MeasurementNames

func (p *IndexFiles) MeasurementNames() [][]byte

MeasurementNames returns a sorted list of all measurement names for all files.

func (IndexFiles) MeasurementSeriesIterator

func (p IndexFiles) MeasurementSeriesIterator(name []byte) tsdb.SeriesIterator

MeasurementSeriesIterator returns an iterator that merges series across all files.

func (IndexFiles) Release

func (p IndexFiles) Release()

Release removes a reference count from all files.

func (IndexFiles) Retain

func (p IndexFiles) Retain()

Retain adds a reference count to all files.

func (IndexFiles) SeriesIterator

func (p IndexFiles) SeriesIterator() tsdb.SeriesIterator

SeriesIterator returns an iterator that merges series across all files.

func (IndexFiles) Stat

func (p IndexFiles) Stat() (*IndexFilesInfo, error)

Stat returns the max index file size and the total file size for all index files.

func (*IndexFiles) TagKeyIterator

func (p *IndexFiles) TagKeyIterator(name []byte) (TagKeyIterator, error)

TagKeyIterator returns an iterator that merges tag keys across all files.

func (IndexFiles) TagValueSeriesIterator

func (p IndexFiles) TagValueSeriesIterator(name, key, value []byte) tsdb.SeriesIterator

TagValueSeriesIterator returns an iterator that merges series across all files.

type IndexFilesInfo

type IndexFilesInfo struct {
	MaxSize int64     // largest file size
	Size    int64     // total file size
	ModTime time.Time // last modified
}

type LogEntry

type LogEntry struct {
	Flag     byte        // flag
	Name     []byte      // measurement name
	Tags     models.Tags // tagset
	Checksum uint32      // checksum of flag/name/tags.
	Size     int         // total size of record, in bytes.
}

LogEntry represents a single log entry in the write-ahead log.

func (*LogEntry) UnmarshalBinary

func (e *LogEntry) UnmarshalBinary(data []byte) error

UnmarshalBinary unmarshals data into e.

type LogFile

type LogFile struct {
	// contains filtered or unexported fields
}

LogFile represents an on-disk write-ahead log file.

func NewLogFile

func NewLogFile(path string) *LogFile

NewLogFile returns a new instance of LogFile.

func (*LogFile) AddSeries

func (f *LogFile) AddSeries(name []byte, tags models.Tags) error

AddSeries adds a series to the log file.

func (*LogFile) AddSeriesList

func (f *LogFile) AddSeriesList(names [][]byte, tagsSlice []models.Tags) error

AddSeriesList adds a list of series to the log file in bulk.

func (*LogFile) Close

func (f *LogFile) Close() error

Close shuts down the file handle and mmap.

func (*LogFile) CompactTo

func (f *LogFile) CompactTo(w io.Writer, m, k uint64) (n int64, err error)

CompactTo compacts the log file and writes it to w.

func (*LogFile) DeleteMeasurement

func (f *LogFile) DeleteMeasurement(name []byte) error

DeleteMeasurement adds a tombstone for a measurement to the log file.

func (*LogFile) DeleteSeries

func (f *LogFile) DeleteSeries(name []byte, tags models.Tags) error

DeleteSeries adds a tombstone for a series to the log file.

func (*LogFile) DeleteTagKey

func (f *LogFile) DeleteTagKey(name, key []byte) error

DeleteTagKey adds a tombstone for a tag key to the log file.

func (*LogFile) DeleteTagValue

func (f *LogFile) DeleteTagValue(name, key, value []byte) error

DeleteTagValue adds a tombstone for a tag value to the log file.

func (*LogFile) Filter

func (f *LogFile) Filter() *bloom.Filter

Filter returns the bloom filter for the file.

func (*LogFile) FilterNamesTags

func (f *LogFile) FilterNamesTags(names [][]byte, tagsSlice []models.Tags) ([][]byte, []models.Tags)

FilterNamesTags filters out any series which already exist. It modifies the provided slices of names and tags.

func (*LogFile) Flush

func (f *LogFile) Flush() error

Flush flushes buffered data to disk.

func (*LogFile) HasSeries

func (f *LogFile) HasSeries(name []byte, tags models.Tags, buf []byte) (exists, tombstoned bool)

HasSeries returns flags indicating if the series exists and if it is tombstoned.

func (*LogFile) ID

func (f *LogFile) ID() int

ID returns the file sequence identifier.

func (*LogFile) Level

func (f *LogFile) Level() int

Level returns the log level of the file.

func (*LogFile) Measurement

func (f *LogFile) Measurement(name []byte) MeasurementElem

Measurement returns a measurement element.

func (*LogFile) MeasurementIterator

func (f *LogFile) MeasurementIterator() MeasurementIterator

MeasurementIterator returns an iterator over all the measurements in the file.

func (*LogFile) MeasurementN

func (f *LogFile) MeasurementN() (n uint64)

MeasurementN returns the total number of measurements.

func (*LogFile) MeasurementNames

func (f *LogFile) MeasurementNames() []string

MeasurementNames returns an ordered list of measurement names.

func (*LogFile) MeasurementSeriesIterator

func (f *LogFile) MeasurementSeriesIterator(name []byte) tsdb.SeriesIterator

MeasurementSeriesIterator returns an iterator over all series for a measurement.

func (*LogFile) MergeMeasurementsSketches

func (f *LogFile) MergeMeasurementsSketches(sketch, tsketch estimator.Sketch) error

MergeMeasurementsSketches merges the measurement sketches belonging to this LogFile into the provided sketches.

MergeMeasurementsSketches is safe for concurrent use by multiple goroutines.

func (*LogFile) MergeSeriesSketches

func (f *LogFile) MergeSeriesSketches(sketch, tsketch estimator.Sketch) error

MergeSeriesSketches merges the series sketches belonging to this LogFile into the provided sketches.

MergeSeriesSketches is safe for concurrent use by multiple goroutines.

func (*LogFile) Open

func (f *LogFile) Open() error

Open reads the log from a file and validates all the checksums.

func (*LogFile) Path

func (f *LogFile) Path() string

Path returns the file path.

func (*LogFile) Release

func (f *LogFile) Release()

Release removes a reference count from the file.

func (*LogFile) Retain

func (f *LogFile) Retain()

Retain adds a reference count to the file.

func (*LogFile) Series

func (f *LogFile) Series(name []byte, tags models.Tags) tsdb.SeriesElem

Series returns a series by name/tags.

func (*LogFile) SeriesIterator

func (f *LogFile) SeriesIterator() tsdb.SeriesIterator

SeriesIterator returns an iterator over all series in the log file.

func (*LogFile) SeriesN

func (f *LogFile) SeriesN() (n uint64)

SeriesN returns the total number of series in the file.

func (*LogFile) SeriesWithBuffer

func (f *LogFile) SeriesWithBuffer(name []byte, tags models.Tags, buf []byte) tsdb.SeriesElem

SeriesWithBuffer returns a series by name/tags.

func (*LogFile) SetPath

func (f *LogFile) SetPath(path string)

SetPath sets the log file's path.

func (*LogFile) Size

func (f *LogFile) Size() int64

Size returns the size of the file, in bytes.

func (*LogFile) Stat

func (f *LogFile) Stat() (int64, time.Time)

Stat returns size and last modification time of the file.

func (*LogFile) TagKey

func (f *LogFile) TagKey(name, key []byte) TagKeyElem

TagKey returns a tag key element.

func (*LogFile) TagKeyIterator

func (f *LogFile) TagKeyIterator(name []byte) TagKeyIterator

TagKeyIterator returns a value iterator for a measurement.

func (*LogFile) TagKeyN

func (f *LogFile) TagKeyN() (n uint64)

TagKeyN returns the total number of keys.

func (*LogFile) TagKeySeriesIterator

func (f *LogFile) TagKeySeriesIterator(name, key []byte) tsdb.SeriesIterator

TagKeySeriesIterator returns a series iterator for a tag key.

func (*LogFile) TagValue

func (f *LogFile) TagValue(name, key, value []byte) TagValueElem

TagValue returns a tag value element.

func (*LogFile) TagValueIterator

func (f *LogFile) TagValueIterator(name, key []byte) TagValueIterator

TagValueIterator returns a value iterator for a tag key.

func (*LogFile) TagValueN

func (f *LogFile) TagValueN() (n uint64)

TagValueN returns the total number of values.

func (*LogFile) TagValueSeriesIterator

func (f *LogFile) TagValueSeriesIterator(name, key, value []byte) tsdb.SeriesIterator

TagValueSeriesIterator returns a series iterator for a tag value.

type Manifest

type Manifest struct {
	Levels []CompactionLevel `json:"levels,omitempty"`
	Files  []string          `json:"files,omitempty"`

	// Version should be updated whenever the TSI format has changed.
	Version int `json:"version,omitempty"`
}

Manifest represents the list of log & index files that make up the index. The files are listed in time order, not necessarily ID order.

func NewManifest

func NewManifest() *Manifest

NewManifest returns a new instance of Manifest with default compaction levels.

func ReadManifestFile

func ReadManifestFile(path string) (*Manifest, error)

ReadManifestFile reads a manifest from a file path.

func (*Manifest) HasFile

func (m *Manifest) HasFile(name string) bool

HasFile returns true if name is listed in the log files or index files.

func (*Manifest) Validate

func (m *Manifest) Validate() error

Validate checks if the Manifest's version is compatible with this version of the tsi1 index.

type MeasurementBlock

type MeasurementBlock struct {
	// contains filtered or unexported fields
}

MeasurementBlock represents a collection of all measurements in an index.

func (*MeasurementBlock) Elem

func (blk *MeasurementBlock) Elem(name []byte) (e MeasurementBlockElem, ok bool)

Elem returns an element for a measurement.

func (*MeasurementBlock) Iterator

func (blk *MeasurementBlock) Iterator() MeasurementIterator

Iterator returns an iterator over all measurements.

func (*MeasurementBlock) UnmarshalBinary

func (blk *MeasurementBlock) UnmarshalBinary(data []byte) error

UnmarshalBinary unpacks data into the block. Block is not copied so data should be retained and unchanged after being passed into this function.

func (*MeasurementBlock) Version

func (blk *MeasurementBlock) Version() int

Version returns the encoding version parsed from the data. Only valid after UnmarshalBinary() has been successfully invoked.

type MeasurementBlockElem

type MeasurementBlockElem struct {
	// contains filtered or unexported fields
}

MeasurementBlockElem represents an internal measurement element.

func (*MeasurementBlockElem) Deleted

func (e *MeasurementBlockElem) Deleted() bool

Deleted returns true if the tombstone flag is set.

func (*MeasurementBlockElem) HasSeries

func (e *MeasurementBlockElem) HasSeries() bool

func (*MeasurementBlockElem) Name

func (e *MeasurementBlockElem) Name() []byte

Name returns the measurement name.

func (*MeasurementBlockElem) SeriesData

func (e *MeasurementBlockElem) SeriesData() []byte

SeriesData returns the raw series data.

func (*MeasurementBlockElem) SeriesID

func (e *MeasurementBlockElem) SeriesID(i int) uint32

SeriesID returns series ID at an index.

func (*MeasurementBlockElem) SeriesIDs

func (e *MeasurementBlockElem) SeriesIDs() []uint32

SeriesIDs returns a list of decoded series ids.

NOTE: This should be used for testing and diagnostics purposes only. It requires loading the entire list of series in-memory.

func (*MeasurementBlockElem) SeriesN

func (e *MeasurementBlockElem) SeriesN() uint32

SeriesN returns the number of series associated with the measurement.

func (*MeasurementBlockElem) Size

func (e *MeasurementBlockElem) Size() int

Size returns the size of the element.

func (*MeasurementBlockElem) TagBlockOffset

func (e *MeasurementBlockElem) TagBlockOffset() int64

TagBlockOffset returns the offset of the measurement's tag block.

func (*MeasurementBlockElem) TagBlockSize

func (e *MeasurementBlockElem) TagBlockSize() int64

TagBlockSize returns the size of the measurement's tag block.

func (*MeasurementBlockElem) UnmarshalBinary

func (e *MeasurementBlockElem) UnmarshalBinary(data []byte) error

UnmarshalBinary unmarshals data into e.

type MeasurementBlockTrailer

type MeasurementBlockTrailer struct {
	Version int // Encoding version

	// Offset & size of data section.
	Data struct {
		Offset int64
		Size   int64
	}

	// Offset & size of hash map section.
	HashIndex struct {
		Offset int64
		Size   int64
	}

	// Offset and size of cardinality sketch for measurements.
	Sketch struct {
		Offset int64
		Size   int64
	}

	// Offset and size of cardinality sketch for tombstoned measurements.
	TSketch struct {
		Offset int64
		Size   int64
	}
}

MeasurementBlockTrailer represents meta data at the end of a MeasurementBlock.

func ReadMeasurementBlockTrailer

func ReadMeasurementBlockTrailer(data []byte) (MeasurementBlockTrailer, error)

ReadMeasurementBlockTrailer returns the block trailer from data.

func (*MeasurementBlockTrailer) WriteTo

func (t *MeasurementBlockTrailer) WriteTo(w io.Writer) (n int64, err error)

WriteTo writes the trailer to w.

type MeasurementBlockWriter

type MeasurementBlockWriter struct {
	// contains filtered or unexported fields
}

MeasurementBlockWriter writes a measurement block.

func NewMeasurementBlockWriter

func NewMeasurementBlockWriter() *MeasurementBlockWriter

NewMeasurementBlockWriter returns a new MeasurementBlockWriter.

func (*MeasurementBlockWriter) Add

func (mw *MeasurementBlockWriter) Add(name []byte, deleted bool, offset, size int64, seriesIDs []uint32)

Add adds a measurement with series and tag set offset/size.

func (*MeasurementBlockWriter) WriteTo

func (mw *MeasurementBlockWriter) WriteTo(w io.Writer) (n int64, err error)

WriteTo encodes the measurements to w.

type MeasurementElem

type MeasurementElem interface {
	Name() []byte
	Deleted() bool
	HasSeries() bool
}

MeasurementElem represents a generic measurement element.

type MeasurementElems

type MeasurementElems []MeasurementElem

MeasurementElems represents a list of MeasurementElem.

func (MeasurementElems) Len

func (a MeasurementElems) Len() int

func (MeasurementElems) Less

func (a MeasurementElems) Less(i, j int) bool

func (MeasurementElems) Swap

func (a MeasurementElems) Swap(i, j int)

type MeasurementIterator

type MeasurementIterator interface {
	Next() MeasurementElem
}

MeasurementIterator represents a iterator over a list of measurements.

func FilterUndeletedMeasurementIterator

func FilterUndeletedMeasurementIterator(itr MeasurementIterator) MeasurementIterator

FilterUndeletedMeasurementIterator returns an iterator which filters all deleted measurement.

func MergeMeasurementIterators

func MergeMeasurementIterators(itrs ...MeasurementIterator) MeasurementIterator

MergeMeasurementIterators returns an iterator that merges a set of iterators. Iterators that are first in the list take precendence and a deletion by those early iterators will invalidate elements by later iterators.

type SeriesBlock

type SeriesBlock struct {
	// contains filtered or unexported fields
}

SeriesBlock represents the section of the index that holds series data.

func (*SeriesBlock) HasSeries

func (blk *SeriesBlock) HasSeries(name []byte, tags models.Tags, buf []byte) (exists, tombstoned bool)

HasSeries returns flags indicating if the series exists and if it is tombstoned.

func (*SeriesBlock) Offset

func (blk *SeriesBlock) Offset(name []byte, tags models.Tags, buf []byte) (offset uint32, tombstoned bool)

Offset returns the byte offset of the series within the block.

func (*SeriesBlock) Series

func (blk *SeriesBlock) Series(name []byte, tags models.Tags) tsdb.SeriesElem

Series returns a series element.

func (*SeriesBlock) SeriesCount

func (blk *SeriesBlock) SeriesCount() uint32

SeriesCount returns the number of series.

func (*SeriesBlock) SeriesIterator

func (blk *SeriesBlock) SeriesIterator() tsdb.SeriesIterator

SeriesIterator returns an iterator over all the series.

func (*SeriesBlock) UnmarshalBinary

func (blk *SeriesBlock) UnmarshalBinary(data []byte) error

UnmarshalBinary unpacks data into the series list.

If data is an mmap then it should stay open until the series list is no longer used because data access is performed directly from the byte slice.

type SeriesBlockElem

type SeriesBlockElem struct {
	// contains filtered or unexported fields
}

SeriesBlockElem represents a series element in the series list.

func (*SeriesBlockElem) Deleted

func (e *SeriesBlockElem) Deleted() bool

Deleted returns true if the tombstone flag is set.

func (*SeriesBlockElem) Expr

func (e *SeriesBlockElem) Expr() influxql.Expr

Expr always returns a nil expression. This is only used by higher level query planning.

func (*SeriesBlockElem) Name

func (e *SeriesBlockElem) Name() []byte

Name returns the measurement name.

func (*SeriesBlockElem) Tags

func (e *SeriesBlockElem) Tags() models.Tags

Tags returns the tag set.

func (*SeriesBlockElem) UnmarshalBinary

func (e *SeriesBlockElem) UnmarshalBinary(data []byte) error

UnmarshalBinary unmarshals data into e.

type SeriesBlockEncoder

type SeriesBlockEncoder struct {
	// contains filtered or unexported fields
}

SeriesBlockEncoder encodes series to a SeriesBlock in an underlying writer.

func NewSeriesBlockEncoder

func NewSeriesBlockEncoder(w io.Writer, n uint32, m, k uint64) *SeriesBlockEncoder

NewSeriesBlockEncoder returns a new instance of SeriesBlockEncoder.

func (*SeriesBlockEncoder) Close

func (enc *SeriesBlockEncoder) Close() error

Close writes the index and trailer. This should be called at the end once all series have been encoded.

func (*SeriesBlockEncoder) Encode

func (enc *SeriesBlockEncoder) Encode(name []byte, tags models.Tags, deleted bool) error

Encode writes a series to the underlying writer. The series must be lexicographical sorted after the previous encoded series.

func (*SeriesBlockEncoder) N

func (enc *SeriesBlockEncoder) N() int64

N returns the number of bytes written.

type SeriesBlockTrailer

type SeriesBlockTrailer struct {
	Series struct {
		Data struct {
			Offset int32
			Size   int32
		}
		Index struct {
			Offset int32
			Size   int32
			N      int32
		}
	}

	// Bloom filter info.
	Bloom struct {
		K      uint64
		Offset int32
		Size   int32
	}

	// Offset and size of cardinality sketch for measurements.
	Sketch struct {
		Offset int32
		Size   int32
	}

	// Offset and size of cardinality sketch for tombstoned measurements.
	TSketch struct {
		Offset int32
		Size   int32
	}

	SeriesN    int32
	TombstoneN int32
}

SeriesBlockTrailer represents meta data written to the end of the series list.

func ReadSeriesBlockTrailer

func ReadSeriesBlockTrailer(data []byte) SeriesBlockTrailer

ReadSeriesBlockTrailer returns the series list trailer from data.

func (SeriesBlockTrailer) WriteTo

func (t SeriesBlockTrailer) WriteTo(w io.Writer) (n int64, err error)

type TagBlock

type TagBlock struct {
	// contains filtered or unexported fields
}

TagBlock represents tag key/value block for a single measurement.

func (*TagBlock) TagKeyElem

func (blk *TagBlock) TagKeyElem(key []byte) TagKeyElem

TagKeyElem returns an element for a tag key. Returns an element with a nil key if not found.

func (*TagBlock) TagKeyIterator

func (blk *TagBlock) TagKeyIterator() TagKeyIterator

TagKeyIterator returns an iterator over all the keys in the block.

func (*TagBlock) TagValueElem

func (blk *TagBlock) TagValueElem(key, value []byte) TagValueElem

TagValueElem returns an element for a tag value.

func (*TagBlock) UnmarshalBinary

func (blk *TagBlock) UnmarshalBinary(data []byte) error

UnmarshalBinary unpacks data into the tag block. Tag block is not copied so data should be retained and unchanged after being passed into this function.

func (*TagBlock) Version

func (blk *TagBlock) Version() int

Version returns the encoding version parsed from the data. Only valid after UnmarshalBinary() has been successfully invoked.

type TagBlockEncoder

type TagBlockEncoder struct {
	// contains filtered or unexported fields
}

TagBlockEncoder encodes a tags to a TagBlock section.

func NewTagBlockEncoder

func NewTagBlockEncoder(w io.Writer) *TagBlockEncoder

NewTagBlockEncoder returns a new TagBlockEncoder.

func (*TagBlockEncoder) Close

func (enc *TagBlockEncoder) Close() error

Close flushes the trailer of the encoder to the writer.

func (*TagBlockEncoder) EncodeKey

func (enc *TagBlockEncoder) EncodeKey(key []byte, deleted bool) error

EncodeKey writes a tag key to the underlying writer.

func (*TagBlockEncoder) EncodeValue

func (enc *TagBlockEncoder) EncodeValue(value []byte, deleted bool, seriesIDs []uint32) error

EncodeValue writes a tag value to the underlying writer. The tag key must be lexicographical sorted after the previous encoded tag key.

func (*TagBlockEncoder) N

func (enc *TagBlockEncoder) N() int64

N returns the number of bytes written.

type TagBlockKeyElem

type TagBlockKeyElem struct {
	// contains filtered or unexported fields
}

TagBlockKeyElem represents a tag key element in a TagBlock.

func (*TagBlockKeyElem) Deleted

func (e *TagBlockKeyElem) Deleted() bool

Deleted returns true if the key has been tombstoned.

func (*TagBlockKeyElem) Key

func (e *TagBlockKeyElem) Key() []byte

Key returns the key name of the element.

func (*TagBlockKeyElem) TagValueIterator

func (e *TagBlockKeyElem) TagValueIterator() TagValueIterator

TagValueIterator returns an iterator over the key's values.

type TagBlockTrailer

type TagBlockTrailer struct {
	Version int   // Encoding version
	Size    int64 // Total size w/ trailer

	// Offset & size of value data section.
	ValueData struct {
		Offset int64
		Size   int64
	}

	// Offset & size of key data section.
	KeyData struct {
		Offset int64
		Size   int64
	}

	// Offset & size of hash map section.
	HashIndex struct {
		Offset int64
		Size   int64
	}
}

TagBlockTrailer represents meta data at the end of a TagBlock.

func ReadTagBlockTrailer

func ReadTagBlockTrailer(data []byte) (TagBlockTrailer, error)

ReadTagBlockTrailer returns the tag block trailer from data.

func (*TagBlockTrailer) WriteTo

func (t *TagBlockTrailer) WriteTo(w io.Writer) (n int64, err error)

WriteTo writes the trailer to w.

type TagBlockValueElem

type TagBlockValueElem struct {
	// contains filtered or unexported fields
}

TagBlockValueElem represents a tag value element.

func (*TagBlockValueElem) Deleted

func (e *TagBlockValueElem) Deleted() bool

Deleted returns true if the element has been tombstoned.

func (*TagBlockValueElem) SeriesData

func (e *TagBlockValueElem) SeriesData() []byte

SeriesData returns the raw series data.

func (*TagBlockValueElem) SeriesID

func (e *TagBlockValueElem) SeriesID(i int) uint32

SeriesID returns series ID at an index.

func (*TagBlockValueElem) SeriesIDs

func (e *TagBlockValueElem) SeriesIDs() []uint32

SeriesIDs returns a list decoded series ids.

func (*TagBlockValueElem) SeriesN

func (e *TagBlockValueElem) SeriesN() uint32

SeriesN returns the series count.

func (*TagBlockValueElem) Size

func (e *TagBlockValueElem) Size() int

Size returns the size of the element.

func (*TagBlockValueElem) Value

func (e *TagBlockValueElem) Value() []byte

Value returns the value for the element.

type TagKeyElem

type TagKeyElem interface {
	Key() []byte
	Deleted() bool
	TagValueIterator() TagValueIterator
}

TagKeyElem represents a generic tag key element.

type TagKeyIterator

type TagKeyIterator interface {
	Next() TagKeyElem
}

TagKeyIterator represents a iterator over a list of tag keys.

func MergeTagKeyIterators

func MergeTagKeyIterators(itrs ...TagKeyIterator) TagKeyIterator

MergeTagKeyIterators returns an iterator that merges a set of iterators. Iterators that are first in the list take precendence and a deletion by those early iterators will invalidate elements by later iterators.

type TagValueElem

type TagValueElem interface {
	Value() []byte
	Deleted() bool
}

TagValueElem represents a generic tag value element.

type TagValueIterator

type TagValueIterator interface {
	Next() TagValueElem
}

TagValueIterator represents a iterator over a list of tag values.

func MergeTagValueIterators

func MergeTagValueIterators(itrs ...TagValueIterator) TagValueIterator

MergeTagValueIterators returns an iterator that merges a set of iterators. Iterators that are first in the list take precendence and a deletion by those early iterators will invalidate elements by later iterators.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL