whisperconverter

package

v1.1.2 Latest Latest Go to latest Published: Mar 19, 2024 License: AGPL-3.0 Imports: 25 Imported by: 0

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

github.com/grafana/mimir-proxies

Links

Open Source Insights

Documentation ¶

Index ¶

func CreateWhisperFile(path string, timestamps []*time.Time) error
func ListFilesInDir(dirPath string) ([]string, error)
func ReadPoints(w Archive, name string) ([]whisper.Point, error)
func SplitSamplesByDays(samples []mimirpb.Sample) [][]mimirpb.Sample
func ToMimirSamples(points []whisper.Point) ([]mimirpb.Sample, error)
func ToTime(in string) (time.Time, error)
func ToTimes(in []string) ([]*time.Time, error)
func WhisperToMimirSamples(whisperFile, name string) ([]mimirpb.Sample, error)
type Archive
type WhisperConverter
- func NewWhisperConverter(namePrefix, whisperDirectory string, fileFilter *regexp.Regexp, ...) *WhisperConverter

Constants ¶

This section is empty.

Variables ¶

This section is empty.

Functions ¶

func CreateWhisperFile ¶

func CreateWhisperFile(path string, timestamps []*time.Time) error

func ListFilesInDir ¶

func ListFilesInDir(dirPath string) ([]string, error)

func ReadPoints ¶

func ReadPoints(w Archive, name string) ([]whisper.Point, error)

ReadPoints reads and concatenates all of the points in a whisper Archive.

func SplitSamplesByDays ¶

func SplitSamplesByDays(samples []mimirpb.Sample) [][]mimirpb.Sample

SplitSamplesByDays separates a slice of samples over the midnight UTC boundary.

func ToMimirSamples ¶

func ToMimirSamples(points []whisper.Point) ([]mimirpb.Sample, error)

ToMimirSamples converts a Whisper metric with the given name to a slice of labels and series of mimir samples. Returns error if no points.

func ToTime ¶

func ToTime(in string) (time.Time, error)

func ToTimes ¶

func ToTimes(in []string) ([]*time.Time, error)

func WhisperToMimirSamples ¶

func WhisperToMimirSamples(whisperFile, name string) ([]mimirpb.Sample, error)

WhisperToMimirSamples opens the given whisper file, applying the given metric name, and writes it to the given block directory with blocks covering the given duration.

Types ¶

type Archive ¶

type Archive interface {
	GetArchives() []whisper.ArchiveInfo
	DumpArchive(int) ([]whisper.Point, error)
}

Archive provides a testable interface for converting whisper databases.

type WhisperConverter ¶

type WhisperConverter struct {
	// contains filtered or unexported fields
}

WhisperConverter is an object for performing various steps of the conversion from Whisper files to Mimir blocks.

func NewWhisperConverter ¶

func NewWhisperConverter(
	namePrefix,
	whisperDirectory string,
	fileFilter *regexp.Regexp,
	threads,
	workerCount,
	workerID int,
	customLabels labels.Labels,
	dates []time.Time,
	logger log.Logger) *WhisperConverter

func (*WhisperConverter) CommandDateRange ¶

func (c *WhisperConverter) CommandDateRange(targetWhisperFiles string)

CommandDateRange prints the minimum and maximum timestamps for the dataset in a format suitable for use as arguments to pass1 and pass2. targetWhisperFiles is a filename containing the list of files to process, or if blank, files will be walked using c.whisperDirectory.

func (*WhisperConverter) CommandFileList ¶

func (c *WhisperConverter) CommandFileList(targetWhisperFiles string) error

CommandFileList traverses the file structure starting from whisperDirectory, discovers files that match fileFilter, and outputs those files to the destination specified by targetWhisperFiles.

func (*WhisperConverter) CommandPass1 ¶

func (c *WhisperConverter) CommandPass1(targetWhisperFiles, intermediateDir string, resumeIntermediate bool) error

CommandPass1 performs the first pass conversion to intermediate files. Each metric is dumped, sorted, and split into days, and the individual days are written to the intermediate files. If this stage crashes, rerunning the stage will automatically resume. targetWhisperFiles is a filename containing the list of files to process, or if blank, files will be walked using c.whisperDirectory.

func (*WhisperConverter) CommandPass2 ¶

func (c *WhisperConverter) CommandPass2(intermediateDir, blocksDir string, overwriteBlocks bool) error

CommandPass2 performs the second pass conversion to Mimir blocks. It reads each intermediate file, sorts the metrics by labels, and outputs the block.

func (*WhisperConverter) GetProcessedCount ¶

func (c *WhisperConverter) GetProcessedCount() uint64

func (*WhisperConverter) GetSkippedCount ¶

func (c *WhisperConverter) GetSkippedCount() uint64

Source Files ¶

View all Source files

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL