sdk

package
v0.0.0-...-6698953 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 22, 2023 License: Apache-2.0 Imports: 16 Imported by: 3

README

SDK

Data source plugin for the Tune Insight Note

A TI Note data source plugin extends the data manager to allow clients to query external data sources. It exposes operations that can have arbitrary parameters and results, and can potentially output data objects.

How to develop a data source plugin

A TI Note data source plugin is a Go plugin that exposes two variables:

  • DataSourceType a string (of the type sdk.DataSourceType) that identifies uniquely the type of data source.
  • DataSourcePluginFactory a function (of the type sdk.DataSourcePluginFactory) that can be invoked by the TI Note to create a new instance of the data source.

The factory function sdk.DataSourcePluginFactory takes as parameters:

  • logger that allows the data source plugin logging to be integrated with the one of the TI Note;
  • config a map of arbitrary config keys to allow the data source to be configured by the TI Note.

It returns a data source, which is a struct that implements the interface sdk.DataSourcePlugin with the function Query(). This function takes as arguments:

  • userID the unique identifier of the user invoking the query;
  • operation the operation requested for the query;
  • jsonParameters the parameters of the operation, that can be anything in the JSON format;
  • outputDataObjectsSharedIDs the shared IDs defined by the client the potential output data objects of the operation should have.

The Query() function returns:

  • jsonResults the results of the operation, that can be anything in the JSON format;
  • outputDataObjects the potential output data objects of the operation;
  • err the potential error of the operation.

If the operation outputs data objects, those must be of the type sdk.DataObject, with the shared ID and output name provided by the client.

Use the plugin in Tune Insight Note

The plugin compiled into a .so file must be placed under the plugins/datasources directory of the TI Note. The plugin is then dynamically loaded at TI Note startup, after which it will be possible to instantiate, query, and manage data sources of the newly defined type throug the TI Note API.

Documentation

Index

Constants

View Source
const (
	// DefaultResultKey is the default key of the results returned by `Query`.
	DefaultResultKey string = "default"
	// OutputDataObjectsKey is the key for the data object IDs generated by the data source results.
	OutputDataObjectsKey string = "outputDataObjects"
	// QueryParams is the parameters key
	QueryParams string = "params"
	// QueryOperation is the operation key
	QueryOperation string = "operation"
)
View Source
const (
	// DSCoreMetadataField is the key under which the MetadataStorage are stored in the TI Note storage.
	DSCoreMetadataField = "ds-core-metadata"
)

Variables

This section is empty.

Functions

func Checksum

func Checksum(config DatabaseConfig) int64

Checksum returns the checksum of a connection configuration which is created using the datasource name and the driver name

func DataImpl

func DataImpl(ds DataSource) map[string]interface{}

DataImpl is the function that should be called by all Data() implementations.

func IsRegistered

func IsRegistered(driver string) bool

IsRegistered returns whether or not the given db-driver is already registered

func NewConnection

func NewConnection(conf DatabaseConfig) (*sql.DB, error)

NewConnection opens a new sql.DB connection given the configuration, if the driver is not yet registered it gets registered

func WrapErrors

func WrapErrors(msg string, errs []error) error

WrapErrors wraps multiple errors into a single error message

Types

type API

type API struct {
	APIConfig
	logrus.FieldLogger
	SleepingTimeBetweenAttempts time.Duration
}

API is composed of the a logger and API configuration

func NewAPI

func NewAPI(conf APIConfig) (*API, error)

NewAPI creates a new API instance given configuration,connection, and parameters for connection attempts

type APIConfig

type APIConfig interface {
	// APIName should return the name of the for example "misp" or "generic"
	APIName() string
	// GetURL should return URL of the API
	GetURL() string
	// GetCert should return the name of the certificate used for this datasource (if applicable, otherwise "")
	GetCert() string
}

APIConfig is a general interface for specific api configs, from the specific config, the API name and URL can be retrieved as well as the datasource name to use the API.

type DBManager

type DBManager struct {
	DBManagerConfig
	sync.Mutex
	// contains filtered or unexported fields
}

DBManager manages live database connections

func NewDBManager

func NewDBManager(config DBManagerConfig) *DBManager

NewDBManager creates a new manager with an empty map of databases

func (*DBManager) Close

func (m *DBManager) Close(config DatabaseConfig) error

Close closes the db given parameters

func (*DBManager) CloseAll

func (m *DBManager) CloseAll() error

CloseAll attempts to close all db connections, concatenates any errors (locks the manager)

func (*DBManager) GetDatabase

func (m *DBManager) GetDatabase(config DatabaseConfig) (db *Database, err error)

GetDatabase return the database instance from the configuration, or creates a new one if not registered

func (*DBManager) NewDatabase

func (m *DBManager) NewDatabase(config DatabaseConfig) (db *Database, err error)

NewDatabase creates a new database given a generic configuration

type DBManagerConfig

type DBManagerConfig struct {
	MaxConnectionAttempts              int `yaml:"db-max-conn-attempts" default:"3"`
	SleepingTimeBetweenAttemptsSeconds int `yaml:"db-sleeping-time" default:"4"`
}

DBManagerConfig regroups parameters for connection to all databases

type DataObject

type DataObject struct {
	// OutputName is the output name of the DataObject.
	OutputName OutputDataObjectName

	// SharedID is the DataObjectSharedID that this DataObject should have.
	SharedID models.DataObjectSharedID

	// the ones not being used should be left to nil
	IntValue  *int64
	IntVector []int64
	IntMatrix [][]int64

	FloatValue  *float64
	FloatVector []float64
	FloatMatrix [][]float64

	Columns []string
}

DataObject defines a data object to be produced by a DataSourcePlugin.

type DataSource

type DataSource interface {
	SetID(id models.DataSourceID)
	GetID() models.DataSourceID

	SetContext(ctx *context.Context)
	GetContext() *context.Context

	// GetDataSourceCore returns the DataSourceCore of the DataSource.
	GetDataSourceCore() *DataSourceCore

	// SetDataSourceConfig sets the DataSource configuration, i.e. DataSource specific  metadata not contained in DataSourceCore.
	SetDataSourceConfig(map[string]interface{}) error
	// GetDataSourceConfig returns the DataSource configuration, i.e. DataSource specific  metadata not contained in DataSourceCore.
	GetDataSourceConfig() map[string]interface{}

	// Data must return all the DataSource data to be stored in the TI Note object storage.
	// Implementations should just call the DataImpl() function provided by the package.
	Data() map[string]interface{}

	// Config configures the DataSource. It must be called after DataSourceFactory.
	Config(logger logrus.FieldLogger, config map[string]interface{}) error

	// ConfigFromDB configures the DataSource with the info stored in the DB. It must be called after DataSourceFactory if the DataSource has been retrieved from the DB.
	ConfigFromDB(logger logrus.FieldLogger) error

	// Query data source with a specific operation. (the possible operations are defined by the data source)
	//  - userID is the ID of the user who is querying the data source (e.g. the username)
	//  - params are the query parameters. Typically two keys are "operation" (const QueryOperation) for the type of operation
	// 	  and "params" (const QueryParams) for serialized JSON payloads.
	//  - resultKeys are the keys of the results to be returned. If empty, the default key is "default"
	Query(userID string, params map[string]interface{}, resultKeys ...string) (results map[string]interface{}, err error)

	// Close is called to close all connections related to the DataSource or other instances.
	Close() error
}

DataSource defines a TI Note data source, which is instantiated by a DataSourceFactory. All DataSource implementations should embed DataSourceCore.

type DataSourceCore

type DataSourceCore struct {
	*MetadataDB
	*MetadataStorage
	Ctx *context.Context // For telemetry
}

DataSourceCore contains the common DataSource metadata. All DataSource implementations must embed DataSourceCore.

func NewDataSourceCore

func NewDataSourceCore(mdb *MetadataDB, mds *MetadataStorage) *DataSourceCore

NewDataSourceCore instantiates a DataSourceCore with the provided @mdb and @mds. If either @mdb or @mds are nil, they are set to default values.

func (*DataSourceCore) GetContext

func (dsc *DataSourceCore) GetContext() *context.Context

GetContext return of context of the data source

func (*DataSourceCore) GetDataSourceCore

func (dsc *DataSourceCore) GetDataSourceCore() *DataSourceCore

GetDataSourceCore returns the DataSourceCore of the data source.

func (*DataSourceCore) SetContext

func (dsc *DataSourceCore) SetContext(ctx *context.Context)

SetContext sets a context of the data source

type DataSourceFactory

type DataSourceFactory func(dsc *DataSourceCore, config map[string]interface{}, dbManager *DBManager) (ds DataSource, err error)

DataSourceFactory defines a TI Note data source factory, which is a function that can instantiate a DataSource. A data source plugin must expose a variable of this type and with the same name (i.e., "DataSourceFactory") for the TI Note to load it.

type DataSourceType

type DataSourceType string

DataSourceType defines a data source type, which uniquely identifies a data source plugin. A data source plugin must expose a variable of this type and with the same name (i.e., "DataSourceType") for GeCo to load it.

type Database

type Database struct {
	DatabaseConfig
	logrus.FieldLogger
	*sql.DB
	MaxConnectionAttempts       int
	SleepingTimeBetweenAttempts time.Duration
}

Database is composed of a *sql.DB, logger and Database configuration

func NewDatabase

func NewDatabase(conf DatabaseConfig, connection *sql.DB, maxConnAttempts int, sleepingTimeBetweenAttempts time.Duration) (*Database, error)

NewDatabase creates a new Database instance given configuration,connection, and parameters for connection attempts

func (*Database) Close

func (db *Database) Close() error

Close closes the sql.DB connection

func (*Database) Delete

func (db *Database) Delete(sqlStatement string, args ...interface{}) (err error)

Delete deletes records from the Database. sqlStatement must be in the form of "DELETE FROM xxx WHERE xxx RETURN id".

func (*Database) GetDBConn

func (db *Database) GetDBConn() *sql.DB

GetDBConn returns the sql.DB connection

func (*Database) Retrieve

func (db *Database) Retrieve(sqlStatement string, args ...interface{}) (rows *sql.Rows, err error)

Retrieve retrieves records from the Database. sqlStatement must be in the form of "SELECT xxx FROM xxx [WHERE xxx]".

func (*Database) Store

func (db *Database) Store(sqlStatement string, args ...interface{}) (id string, err error)

Store stores records in the Database. sqlStatement must be in the form of "INSERT INTO xxx VALUES ($1, $2, ...) RETURNING id".

func (*Database) Update

func (db *Database) Update(sqlStatement string, args ...interface{}) (id string, err error)

Update updates records in the Database. sqlStatement must be in the form of "UPDATE xxx SET xxx WHERE xxx RETURNING id".

func (*Database) WaitReady

func (db *Database) WaitReady() (err error)

WaitReady performs a health-check of the Database and returns an error if the Database is unreachable

type DatabaseConfig

type DatabaseConfig interface {
	// DriverName should return the name of the driver for example "postgres" or "sqlite3"
	DriverName() string
	// DataSourceName should return the connection string to the db: postgres example "host=localhost port=5432 user=test password=test dbname=test sslmode=disable"
	DataSourceName() string
	// Driver Should return the database driver
	Driver() driver.Driver
	// Name should return the name of the connected database
	Name() string
}

DatabaseConfig is a general interface for specific-db configs, from the specific config, the driver name can be retrieved as well as the datasource name to connect to the db and a function to register the actual driver

type DatabaseError

type DatabaseError struct {
	Err error
}

DatabaseError is wraps a database-related error

func (*DatabaseError) Error

func (r *DatabaseError) Error() string

Error prints the error message related to database

type GenericAPIConfig

type GenericAPIConfig struct {
	URL   string `yaml:"api-url" json:"api-url" default:"localhost"`
	Token string `yaml:"api-token" json:"api-token" default:""`
	Cert  string `yaml:"cert" json:"cert" default:""`
}

GenericAPIConfig is the configuration for a generic HTTP API

func (GenericAPIConfig) APIName

func (conf GenericAPIConfig) APIName() string

APIName returns the name of the API which is 'generic'

func (GenericAPIConfig) GetCert

func (conf GenericAPIConfig) GetCert() string

GetCert returns the name of the certificate used for this datasource (if applicable)

func (GenericAPIConfig) GetURL

func (conf GenericAPIConfig) GetURL() string

GetURL returns the url in configuration

type MetadataDB

type MetadataDB struct {
	gorm.Model
	ID                      models.DataSourceID      `gorm:"primaryKey"`
	Name                    string                   `gorm:"uniqueIndex:udx_name;not null"`
	Type                    DataSourceType           `gorm:"not null"`
	CredentialsProviderType credentials.ProviderType `gorm:"not null"`
	Owner                   string                   `gorm:"not null"`
	AccessScope             string
	AuthorizedUsers         pq.StringArray `gorm:"type:text[]"`
	DeletedAt               gorm.DeletedAt `gorm:"uniqueIndex:udx_name"`
}

MetadataDB contains the common DataSource metadata that are stored in the TI Note database.

func NewMetadataDB

func NewMetadataDB(id models.DataSourceID, owner, name string, dsType DataSourceType, cpType credentials.ProviderType) *MetadataDB

NewMetadataDB instantiates a MetadataDB given the required fields.

cpType is the credential provider type (e.g. localCredentialsProvider)

func (*MetadataDB) BeforeCreate

func (mdb *MetadataDB) BeforeCreate(tx *gorm.DB) (err error)

BeforeCreate sets a new id to the DataSource if it is empty upon database insert.

func (*MetadataDB) GetID

func (mdb *MetadataDB) GetID() models.DataSourceID

GetID Return the id of the DataSource.

func (*MetadataDB) SetID

func (mdb *MetadataDB) SetID(id models.DataSourceID)

SetID sets the ID of the DataSource.

func (MetadataDB) TableName

func (MetadataDB) TableName() string

TableName overrides the table name used by MetadataDB to `data_sources`

type MetadataStorage

type MetadataStorage struct {
	CredentialsProvider credentials.Provider
	Attributes          []string
	ConsentType         models.DataSourceConsentType
}

MetadataStorage contains the common DataSource metadata that are stored in the TI Note object storage.

func NewMetadataStorage

func NewMetadataStorage(cp credentials.Provider) *MetadataStorage

NewMetadataStorage instantiates a default MetadataStorage. If a nil @cp is passed, a default Local one is set.

func (*MetadataStorage) Data

func (ms *MetadataStorage) Data() map[string]interface{}

Data returns MetadataStorage in the format expected by the TI Node to store it in storage.

type OutputDataObjectName

type OutputDataObjectName string

OutputDataObjectName is a name for a data object that was output by a DataSourcePlugin.Query.

type PostgresConfig

type PostgresConfig struct {
	Host     string `yaml:"db-host" default:"localhost"`
	Port     int    `yaml:"db-port" default:"5432"`
	Database string `yaml:"db-database" default:"test"`
	User     string `yaml:"db-user" default:"postgres"`
	Password string `yaml:"db-pwd" default:"password"`
}

PostgresConfig is the configuration when using the postgres driver

func (PostgresConfig) DataSourceName

func (conf PostgresConfig) DataSourceName() string

DataSourceName returns the connection string to a postgres db: "host=localhost port=5432 user=test password=test dbname=test sslmode=disable"

func (PostgresConfig) Driver

func (conf PostgresConfig) Driver() driver.Driver

Driver Should returns the postgres database driver

func (PostgresConfig) DriverName

func (conf PostgresConfig) DriverName() string

DriverName returns "postgres"

func (PostgresConfig) Name

func (conf PostgresConfig) Name() string

Name should return the name of the connected database

type SQLiteConfig

type SQLiteConfig struct {
	Database  string `yaml:"db-database" default:"test"`
	Directory string `yaml:"db-directory" default:"db/"`
}

SQLiteConfig is the configuration when using the sqlite driver

func (SQLiteConfig) DataSourceName

func (conf SQLiteConfig) DataSourceName() string

DataSourceName should return the connection string to the db: <directory>/<database>.db

func (SQLiteConfig) Driver

func (conf SQLiteConfig) Driver() driver.Driver

Driver returns the appropriate database driver

func (SQLiteConfig) DriverName

func (conf SQLiteConfig) DriverName() string

DriverName returns the name of the driver which is sqlite3

func (SQLiteConfig) Name

func (conf SQLiteConfig) Name() string

Name returns the name of the connected database

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL