tsmdbs

TSMDBS, just another time-series database.

TSMDBS is a embeddable time-series database module designed for storage and analysis of telemetry data. TSMDBS is a package for Golang and the database intended to be embedded in Golang application. There is no server or external app that provides access to TSMDBS functionality.

What TSMDBS is ...

TSMDBS is a database for storing and querying metrics. It provides Golang API and query language that you can use through Golang API. TSMDBS designed to be ether in-momery or on-disk. TSMDBS uses SQLITE as a database backend, so if you are using it as "on-the-disk" database, you can use any SQLITE tools to access the data. TSMDBS is good enough to serve as a temporary storage for telemetry, that you are going to further process or analyze.

What TSMDBS is not ...

Due to the fact that TSMDBS primarily designed to be used as "in-memory" temporary storage for the telemetry data, TSMDBS is not replacement for the production, long term, telemetry storage, especially in large quantities. TSMDBS not designed to provide per-second/millisecond/nanosecond timestamp accuracy. While internal timestamp representation is in milliseconds, all telemetry data allocated in "per-minute" buckets. TSMDBS uses JSON for internal telemetry representation. So, there are benefits and disadvantages in that design choice.

Why shall you give TSMDBS a try ?

If you are creating an instrumentations that are working with time-based data and particularly telemetry data, you are aware that in order to do any meaningful manipulations with telemetry, you have to query multiple samples and store it somewhere. Storing and retrieving of time-series data could become a chore. TSMDBS gives you a bit of assistance, providing you following features:

• Simple, clean, thread-safe API for Golang.
• Ability to store metrics of different data types.
• Ability to add attributes to each metric, making it more meaningful.
• Ability to add "relations" to telemetry. You can store your metric with attached labels and then retrieve you metric using thouse lables.
• You can use ether API calls or TSMDBS Query language (TSQL) to store and retrieve the data.
• TSQL gives you rich logic, arithmetic, query and statistical functions capabilities. In one line of TSQL code you can sample the telemetry and perform statistical computation with it.

How to use TSMDBS

TSMDBS is a embeddable database for Golang applications and not designed to be used outside of custom Golang applications.

Installation

go get github.com/vulogov/tsmdbs

Use TSMDBS Golang API

How to create in-memory TSMDBS database ?

import "github.com/vulogov/tsmdbs"

db, err := tsmdbs.InMemory()

How to create on-disk NSMDBS database ?

import "github.com/vulogov/tsmdbs"

db, err := tsmdbs.TS("{file name}")

How to store telemetry in TSMDBS ?

import "os"
import "fmt"
import "github.com/vulogov/tsmdbs"

db, err := tsmdbs.InMemory()
if err != nil {
  fmt.Println(err)
  os.Exit(1)
}
id, err := db.Store(mtype, time.Now(), "hostname", "metric.key", theValue, []string{}, map[string]interface{})

Parameters for the Store() functions are:

• Metric type. The string. Default is "metric".
• Timestamp for the metric. You can pass time.Time, number of milliseconds or string with common string representation of date and time. If you pass nil, time.Now() will be used.
• Hostname as string.
• Telemetry key, as string.
• Some telemetry value
• List of the strings with labels for that telemetry.
• Map with attributes for that telemetry.

How to close TSMDBS instance ?

import "os"
import "fmt"
import "github.com/vulogov/tsmdbs"

db, err := tsmdbs.InMemory()
if err != nil {
  fmt.Println(err)
  os.Exit(1)
}
db.Close()

How to use TSQL

Golang API function for serving a TSQL queries

import "os"
import "fmt"
import "github.com/vulogov/tsmdbs"

db, err := tsmdbs.InMemory()
if err != nil {
  fmt.Println(err)
  os.Exit(1)
}
result, err := db.Query("{TSQL query}")
if err != nil {
  fmt.Println(err)
  os.Exit(1)
}

Basic format and functions of TSQL

42

"Hello world"

40 + 2
( 3 + 2 ) * 10

1 < 2

date("01/01/1970")

db.{hostname}.{telemetry key}(.function())

Functions are optional and here is the list of functions.

db.{hostname}.{telemetry key}.{label}.{label}(.optional function())

labels.{label}.{label}.query() or sample()

TSQL examples

db.testhost.answer.application1.application2.insert(42)

In this example, we are storing telemetry for the host = "testhost", telemetry key = "answer", and value = 42, while attaching this metric to labels "application1" and "application2"

db.testhost.answer.sample()

Sampling telemetry stored for host "testhost" and telemetry key "answer"

query(
    end(
      start(
        db.testhost.testkey, date("2022-01-01 23:59:59")
      ),
    now())
)

Querying host="testhost" and telemetry key = "testkey", for start date set to "2022-01-01 23:59:59" and end date to now()

Useful embedded functions

Example:

stddev(db.testhost.testkey.sample())

Query telemetry from host = "testhost" with telemetry key "testkey", then sampling it and then compute standard deviation for the sample.

Show me the code !

Every bit of the TSMDBS source code could be located on public GitHub repository: https://github.com/vulogov/tsmdbs