dsunit

package module
v0.11.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 4, 2023 License: Apache-2.0 Imports: 32 Imported by: 2

README

Datastore Testibility (dsunit)

Datastore testibility library for Go. GoDoc

This library is compatible with Go 1.10+

Please refer to CHANGELOG.md if you encounter breaking changes.

Introduction

Data focused testing belongs to blackbox group, where the main interest goes down to the initial and final state of the datastore.

To set the initial state of ta datastore, this framework provies utilities to either create empty datastore, or to prepare it with dataset data to test that application works correctly.

The final state testing focuses on checking that a dataset data matches an expected set of values after application logic run. In this case this library has ability to verify either complete or snapshot state of a datastore. While the first approach will be comparing all tables data with expected set of values, the latter will reduced verification to the range provided by expected dataset.

Motivation

This library has been design to provide easy and unified way of testing any datastore (SQL, NoSSQL,file logs) on any platform, language and on the cloud. It simplifies test organization by dataset auto discovery used for datastore preparation and verification. Dataset data can be loaded from various sources like: memory, local or remote csv, json files. All dataset support macro expression to dynamically evaluate value of data i.e <ds:sql ["SELECT CURRENT_DATE()"]> On top of that expected data, can also use predicate expressions to delegate verification of the data values i.e. <ds:between [11301, 11303]>. Finally a dataset like a view can be used to store data for many datastore sources in in just one dataset file.

Datastore initialization and dataset data verification can by managed locally or remotely on remote data store unit test server.

Usage

Data setup and verification
  1. With dedicated expected data folder
    
    import (
        "testing"
        "github.com/viant/dsunit"
        _ "github.com/go-sql-driver/mysql"
    )
    
    
    func Test_Usecase(t *testing.T) {
        parent := toolbox.CallerDirectory(3)
        if !dsunit.InitFromURL(t, path.Join(parent, "test", "config.yaml")) {
            return
        }
    
            ... business test logic comes here
    
    
        expectURL := path.Join(parent, "test/case1/data/expect")
        expectedData := dsunit.NewDatasetResource("db1", expectURL , "", "")
        dsunit.Expect(t, dsunit.NewExpectRequest(dsunit.FullTableDatasetCheckPolicy, expectedData))
    
    }
    
  2. With shared expected data folder
    func Test_Usecase(t *testing.T) {
        parent := toolbox.CallerDirectory(3)
        if !dsunit.InitFromURL(t, path.Join(parent, "test", "config.yaml")) {
            return
        }
    
    
        ... business test logic comes here
    
    
        baseDir := path.Join(parent, "test", "data")
        dsunit.ExpectFor(t, "db1", dsunit.FullTableDatasetCheckPolicy, baseDir, "use_case_1")
    }
    
Forcing table truncation before loading data

@table_x.json

[ {},
    {"id":1,"name":"name 1"},
    {"id":2,"name":"name 2"}
]

When pre-seeding table with data, if the first element is empty map, dsunit deletes all record from a table before inserting supplied dataset.

@table_x.json

[]

Empty array will with prepare method removes all record from a table.

Reverse engineer data setup and verification

	registerResponse := service.Register(dsunit.NewRegisterRequest("db1",
			&dsc.Config{
				DriverName: "sqlite3",
				Descriptor: "[url]",
				Parameters: map[string]interface{}{
					"url": filename,
				},
			}))
	if registerResponse.Stats != "ok" {
		log.Fatal(registerResponse.Error)
	}
    
	response := service.Freeze(&dsunit.FreezeRequest{
			Datastore:"db1",
			DestURL:"/tmp/dn1/expect/users.json",
			SQL:"SELECT * FROM users",
    })
	
Tester methods
Service Methods Description Request Response
Register(t *testing.T, request *RegisterRequest) bool register database connection RegisterRequest RegisterResponse
RegisterFromURL(t *testing.T, URL string) bool as above, where JSON request is fetched from URL/relative path RegisterRequest RegisterResponse
Recreate(t *testing.T, request *RecreateRequest) bool recreate database/datastore RecreateRequest RecreateResponse
RecreateFromURL(t *testing.T, URL string) bool as above, where JSON request is fetched from URL/relative path RecreateRequest RecreateResponse
RunSQL(t *testing.T, request *RunSQLRequest) bool run SQL commands RunSQLRequest RunSQLResponse
RunSQLFromURL(t *testing.T, URL string) bool as above, where JSON request is fetched from URL/relative path RunSQLRequest RunSQLResponse
RunScript(t *testing.T, request *RunScriptRequest) bool run SQL script RunScriptRequest RunSQLResponse
RunScriptFromURL(t *testing.T, URL string) bool as above, where JSON request is fetched from URL/relative path RunScriptRequest RunSQLResponse
AddTableMapping(t *testing.T, request *MappingRequest) bool register database table mapping (view), MappingRequest MappingResponse
AddTableMappingFromURL(t *testing.T, URL string) bool as above, where JSON request is fetched from URL/relative path MappingRequest MappingResponse
Init(t *testing.T, request *InitRequest) bool initialize datastore (register, recreate, run sql, add mapping) InitRequest MappingResponse
InitFromURL(t *testing.T, URL string) bool as above, where JSON request is fetched from URL/relative path InitRequest MappingResponse
Prepare(t *testing.T, request *PrepareRequest) bool populate databstore with provided data PrepareRequest MappingResponse
PrepareFromURL(t *testing.T, URL string) bool as above, where JSON request is fetched from URL/relative path PrepareRequest MappingResponse
PrepareDatastore(t *testing.T, datastore string) bool match to populate all data files that are in the same location as a test file, with the same test file prefix, followed by lowe camel case test name n/a n/a
PrepareFor(t *testing.T, datastore string, baseDirectory string, method string) bool match to populate all data files that are located in baseDirectory with method name n/a n/a
Expect(t *testing.T, request *ExpectRequest) bool verify databstore with provided data ExpectRequest MappingResponse
ExpectFromURL(t *testing.T, URL string) bool as above, where JSON request is fetched from URL/relative path ExpectRequest MappingResponse
ExpectDatasets(t *testing.T, datastore string, checkPolicy int) bool match to verify all data files that are in the same location as a test file, with the same test file prefix, followed by lowe camel case test name n/a n/a
ExpectFor(t *testing.T, datastore string, checkPolicy int, baseDirectory string, method string) bool match to verify all dataset files that are located in the same directory as the test file with method name n/a n/a
Freeze(request *FreezeRequest) *FreezeResponse match to verify all dataset files that are located in the same directory as the test file with method name n/a n/a
Dump(request *DumpRequest) *DumpResponse creates a database schema from existing database for supplied tables, datastore, and target Vendor DumpRequest DumpResponse
Compare(request *CompareRequest) *CompareResponse compares data based on specified SQLs from various databases CompareRequest CompareResponse

Validation

This library uses assertly as the underlying validation mechanism

Macros

The macro is an expression with parameters that expands original text value. The general format of macro: <ds:MACRO_NAME [json formated array of parameters]>

The following macro are build-in:

Name Parameters Description Example
sql SQL expression Returns value of SQL expression <ds:sql["SELECT CURRENT_DATE()"]>
seq name of sequence/table for autoicrement Returns value of Sequence <ds:seq["users"]>
Predicates

Predicate allows expected value to be evaluated with actual dataset value using custom predicate logic.

Name Parameters Description Example
between from, to values Evaluate actual value with between predicate <ds:between[1.888889, 1.88889]>
within_sec base time, delta, optional date format Evaluate if actual time is within delta of the base time <ds:within_sec["now", 6, "yyyyMMdd HH:mm:ss"]>
Directives
Data preparation

Most SQL drivers provide meta data about autoincrement, primary key, however if this is not available or partial verification with SQL is used, the following directive come handy.

@autoincrement@

Allows specifying autoincrement field

[
  {"@autoincrement@":"id"},
  {"id":1, "username":"Dudi", "active":true, "salary":12400, "comments":"abc","last_access_time": "2016-03-01 03:10:00"},
  {"id":2, "username":"Rudi", "active":true, "salary":12600, "comments":"def","last_access_time": "2016-03-01 05:10:00"}
]

@indexBy@

(see also asserly indexBy directive usage, for nested data structe validation)

Allows specifying pk fields


[
  {"@indexBy@":["id"]},
  {"id":1, "username":"Dudi", "active":true, "salary":12400, "comments":"abc","last_access_time": "2016-03-01 03:10:00"},
  {"id":2, "username":"Rudi", "active":true, "salary":12600, "comments":"def","last_access_time": "2016-03-01 05:10:00"}
]

Data validation.

@fromQuery@

Allows specified query to fetch actual dataset to be validated against expected dataset

users.json

[
  {"@fromQuery@":"SELECT *  FROM users where id <= 2 ORDER BY id"},
  {"id":1, "username":"Dudi", "active":true, "salary":12400, "comments":"abc","last_access_time": "2016-03-01 03:10:00"},
  {"id":2, "username":"Rudi", "active":true, "salary":12600, "comments":"def","last_access_time": "2016-03-01 05:10:00"}
]

API Documentation

API documentation is available in the docs directory.

GoCover

GoCover

Examples

This project provide a various datasore dsunit integration examples (some with docker vi endly).

RDBMS

NoSQL

External projects::

License

The source code is made available under the terms of the Apache License, Version 2, as stated in the file LICENSE.

Individual files may be made available under their own specific license, all compatible with Apache License, Version 2. Please see individual files for details.

Credits and Acknowledgements

Library Author: Adrian Witas

Contributors: Sudhakaran Dharmaraj

Documentation

Overview

Package dsunit - Datastore testing library.

Index

Constants

View Source
const (
	//FullTableDatasetCheckPolicy policy will drive comparison of all actual datastore data
	FullTableDatasetCheckPolicy = 0
	//SnapshotDatasetCheckPolicy policy will drive comparison of subset of  actual datastore data that is is listed in expected dataset
	SnapshotDatasetCheckPolicy = 1
)
View Source
const (
	AutoincrementDirective  = "@autoincrement@"
	FromQueryDirective      = "@fromQuery@"
	FromQueryAliasDirective = "@fromQueryAlias@"
)
View Source
const (
	ObfuscationMethodReplace    = "replace"
	ObfuscationMethodShuffle    = "shuffle"
	ObfuscationMethodDictionary = "dictionary"
	ObfuscationMethodCipher     = "cipher"
)
View Source
const StatusOk = "ok"

StatusOk represents ok status

Variables

View Source
var LogF = fmt.Printf
View Source
var SubstitutionMapKey = (*data.Map)(nil)

SubstitutionMapKey if provided in context, it will be used to substitute/expand dataset

Functions

func AddTableMapping added in v0.1.1

func AddTableMapping(t *testing.T, request *MappingRequest) bool

AddTableMapping Add table mapping

func AddTableMappingFromURL added in v0.1.1

func AddTableMappingFromURL(t *testing.T, URL string) bool

AddTableMappingFromURL Add table mapping, JSON request is fetched from URL

func Expect added in v0.1.1

func Expect(t *testing.T, request *ExpectRequest) bool

Expect Verify datastore with supplied expected datasets

func ExpectDatasets

func ExpectDatasets(t *testing.T, datastore string, checkPolicy int) bool

ExpectDatasets matches all dataset files that are located in the same directory as the test file with method name to verify that all listed dataset values are present in datastore

func ExpectFor added in v0.1.1

func ExpectFor(t *testing.T, datastore string, checkPolicy int, baseDirectory string, method string) bool

ExpectFor matches all dataset files that are located in baseDirectory with method name to verify that all listed dataset values are present in datastore Note the matchable dataset files in the base directory have the following naming:

<lower_underscore method name>_expect_<table>.[json|csv]
To prepare expected dataset table: 'users' and 'permissions' for test method ReadAll you would
have you create the following files in the baseDirectory

read_all_expect_users.json
read_all_expect_permissions.json

func ExpectFromURL added in v0.1.1

func ExpectFromURL(t *testing.T, URL string) bool

ExpectFromURL Verify datastore with supplied expected datasets, JSON request is fetched from URL

func ExpectWithURL added in v0.10.11

func ExpectWithURL(t *testing.T, checkPolicy int, URL string, datastore string, datasets ...*Dataset) bool

ExpectWithURL Verify datastore with supplied expected datasets, JSON requests are fetched from files in directory

func GetDatastoreDialect added in v0.1.1

func GetDatastoreDialect(datastore string, registry dsc.ManagerRegistry) dsc.DatastoreDialect

GetDatastoreDialect return GetDatastoreDialect for supplied datastore and registry.

func Init added in v0.1.1

func Init(t *testing.T, request *InitRequest) bool

Init datastore, (register, recreated, run sql, add mapping)

func InitFromURL added in v0.1.1

func InitFromURL(t *testing.T, URL string) bool

InitFromURL Init datastore, (register, recreated, run sql, add mapping), JSON request is fetched from URL

func Ping added in v0.7.0

func Ping(t *testing.T, datastore string, timeoutMs int) bool

Ping wait untill database is online or error

func PopulateWithURL added in v0.10.11

func PopulateWithURL(t *testing.T, URL string, datastore string, datasets ...*Dataset) bool

PopulateWithURL Populate database with datasets, JSON requests are fetched from files in directory

func Prepare added in v0.1.1

func Prepare(t *testing.T, request *PrepareRequest) bool

Prepare Populate database with datasets

func PrepareDatastore

func PrepareDatastore(t *testing.T, datastore string) bool

PrepareDatastore matches all dataset files that are in the same location as a test file, with the same test file prefix, followed by lowe camel case test name.

func PrepareFor added in v0.1.1

func PrepareFor(t *testing.T, datastore string, baseDirectory string, method string) bool

PrepareFor matches all dataset files that are located in baseDirectory with method name and populate datastore with all listed dataset Note the matchable dataset files in the base directory have the following naming:

<lower_underscore method name>_populate_<table>.[json|csv]
To prepare dataset to populate datastore table: 'users' and 'permissions' for test method ReadAll you would
have you create the following files in the baseDirectory

read_all_prepare_travelers2.json
read_all_populate_permissions.json

func PrepareFromURL added in v0.1.1

func PrepareFromURL(t *testing.T, URL string) bool

PrepareFromURL Populate database with datasets, JSON request is fetched from URL

func Recreate added in v0.1.1

func Recreate(t *testing.T, request *RecreateRequest) bool

Recreate recreates datastore

func RecreateDatastore added in v0.1.1

func RecreateDatastore(adminDatastore, targetDatastore string, registry dsc.ManagerRegistry) error

RecreateDatastore recreates target datastore from supplied admin datastore and registry

func RecreateFromURL added in v0.1.1

func RecreateFromURL(t *testing.T, URL string) bool

RecreateFromURL Recreate recreates datastore, JSON request is fetched from URL

func Register added in v0.1.1

func Register(t *testing.T, request *RegisterRequest) bool

Register registers new datastore connection

func RegisterFromURL added in v0.1.1

func RegisterFromURL(t *testing.T, URL string) bool

RegisterFromURL Register registers new datastore connection, JSON request is fetched from URL

func RunSQL added in v0.1.1

func RunSQL(t *testing.T, request *RunSQLRequest) bool

RunSQL runs supplied SQL

func RunSQLFromURL added in v0.1.1

func RunSQLFromURL(t *testing.T, URL string) bool

RunSQLFromURL RunSQL runs supplied SQL, JSON request is fetched from URL

func RunScript added in v0.1.1

func RunScript(t *testing.T, request *RunScriptRequest) bool

RunScript runs supplied SQL scripts

func RunScriptFromURL added in v0.1.1

func RunScriptFromURL(t *testing.T, URL string) bool

RunScriptFromURL RunScript runs supplied SQL scripts, JSON request is fetched from URL

func StartServer

func StartServer(port string)

StartServer start dsunit server

func UseRemoteTestServer

func UseRemoteTestServer(endpoint string)

UseRemoteTestServer enables remove testing mode

Types

type BaseResponse added in v0.1.1

type BaseResponse struct {
	Status  string
	Message string
}

BaseResponse represent base response.

func NewBaseOkResponse added in v0.1.1

func NewBaseOkResponse() *BaseResponse

func NewBaseResponse added in v0.1.1

func NewBaseResponse(status, message string) *BaseResponse

func (BaseResponse) Error added in v0.1.1

func (r BaseResponse) Error() error

func (*BaseResponse) SetError added in v0.1.1

func (r *BaseResponse) SetError(err error)

type CheckSchemaRequest added in v0.10.0

type CheckSchemaRequest struct {
	Source           *SchemaTarget
	Dest             *SchemaTarget
	Tables           []string
	CheckNullables   bool
	CheckPrimaryKeys bool
}

CheckSchemaRequest represents schema check request

type CheckSchemaResponse added in v0.10.0

type CheckSchemaResponse struct {
	*BaseResponse
	Tables []*SchemaTableCheck
	*assertly.Validation
}

CheckSchemaResponse represents schema check response

func NewCheckSchemaResponse added in v0.10.0

func NewCheckSchemaResponse() *CheckSchemaResponse

NewCheckSchemaResponse returns new check schema response

type CompareRequest added in v0.4.1

type CompareRequest struct {
	Source1           *DatastoreSQL
	Source2           *DatastoreSQL
	Directives        map[string]interface{}
	Ignore            []string // columns to ignore
	OmitEmpty         bool
	MaxRowDiscrepancy int //max discrepant rows
}

CompareRequest represent compare request

func (*CompareRequest) ApplyDirective added in v0.4.1

func (r *CompareRequest) ApplyDirective(record map[string]interface{})

func (CompareRequest) IndexBy added in v0.4.1

func (r CompareRequest) IndexBy() []string

IndexBy returns index by directive if specified

func (*CompareRequest) Init added in v0.10.0

func (r *CompareRequest) Init() error

type CompareResponse added in v0.4.1

type CompareResponse struct {
	*BaseResponse
	Dataset1Count int
	Dataset2Count int
	MatchedRows   int
	*assertly.Validation
}

CompareResponse represents compare response

type DatafileInfo added in v0.1.1

type DatafileInfo struct {
	Filename string
	Name     string
	Ext      string
	Prefix   string
	Postfix  string
}

DatafileInfo represent data file

func NewDatafileInfo added in v0.1.1

func NewDatafileInfo(filename, prefix, postfix string) *DatafileInfo

NewDatafileInfo returns new datafile info if supplied filedinfo matches prefix, postfix

type Dataset

type Dataset struct {
	Table   string  `required:"true"`
	Records Records `required:"true"`
}

Records represent data records

func NewDataset added in v0.1.1

func NewDataset(table string, records ...map[string]interface{}) *Dataset

NewDataset creates a new dataset for supplied table and records.

type DatasetResource

type DatasetResource struct {
	*url.Resource      ` description:"data file location, csv, json, ndjson formats are supported"`
	*DatastoreDatasets `required:"true" description:"datastore datasets"`
	Prefix             string ` description:"location data file prefix"`  //apply prefix
	Postfix            string ` description:"location data file postgix"` //apply suffix
	// contains filtered or unexported fields
}

DatasetResource represents a dataset resource

func NewDatasetResource added in v0.1.1

func NewDatasetResource(datastore string, URL, prefix, postfix string, datasets ...*Dataset) *DatasetResource

func (*DatasetResource) Init added in v0.1.1

func (r *DatasetResource) Init() error

func (*DatasetResource) Load added in v0.1.1

func (r *DatasetResource) Load() (err error)

Loads dataset from specified resource or data map

type DatasetValidation added in v0.1.1

type DatasetValidation struct {
	Dataset string
	*assertly.Validation
	Expected interface{}
	Actual   interface{}
}

ExpectRequest represents data validation

type DatastoreDatasets added in v0.1.1

type DatastoreDatasets struct {
	Datastore string                              `required:"true" description:"register datastore"`
	Datasets  []*Dataset                          `description:"collection of dataset per table"`
	Data      map[string][]map[string]interface{} `description:"map, where each pair represent table name and records (backwad compatiblity)"`
}

DatastoreDatasets represents a collection of datastore datasets

type DatastoreSQL added in v0.4.1

type DatastoreSQL struct {
	Datastore string
	SQL       string
}

type DumpRequest added in v0.2.0

type DumpRequest struct {
	Datastore string   `description:"registered datastore i.e. db1"`
	Tables    []string `description:"tables, all if empty"`
	DestURL   string   `description:"represent dataset destination"`
	Target    string   `description:"target vendor, use only if different than source"`
	/*
		mapping url content should represents a map between source and dest data (all data type should be upper case) type i.e

		{
			"INT": "BIGINT",
			"INTEGER":   "BIGINT",
			"NUMERIC":   "DECIMAL(7,2)",
			"FLOAT":   "DECIMAL(7,2)",
			"VARCHAR":   "VARCHAR(255)",
			"STRING":   "VARCHAR(255)",
			"CHAR":      "VARCHAR(255)",
			"DATE":      "DATE",
			"TIMESTAMP": "TIMESTAMP",
		},

	*/
	MappingURL string `description:"if target driver is used - you can provide data type mapping"`
}

DumpRequest represent a request to create a database schema

func NewDumpRequestFromURL added in v0.6.2

func NewDumpRequestFromURL(URL string) (*DumpRequest, error)

NewDumpRequestFromURL create a request from url

type DumpResponse added in v0.2.0

type DumpResponse struct {
	*BaseResponse
	Count   int
	DestURL string
}

DumpResponse represents a dump response

type ExpectRequest added in v0.1.1

type ExpectRequest struct {
	*DatasetResource
	CheckPolicy int `required:"true" description:"0 - FullTableDatasetCheckPolicy, 1 - SnapshotDatasetCheckPolicy"`
}

ExpectRequest represents verification datastore request

func NewExpectRequest added in v0.1.1

func NewExpectRequest(checkPolicy int, resource *DatasetResource) *ExpectRequest

NewExpectRequest creates a new prepare request

func NewExpectRequestFromURL added in v0.1.1

func NewExpectRequestFromURL(URL string) (*ExpectRequest, error)

NewExpectRequestFromURL create a request from URL

func (*ExpectRequest) Validate added in v0.1.1

func (r *ExpectRequest) Validate() error

Validate checks if request is valid

type ExpectResponse

type ExpectResponse struct {
	*BaseResponse
	Validation  []*DatasetValidation
	PassedCount int
	FailedCount int
}

ExpectResponse represents verification response

type FreezeRequest added in v0.1.1

type FreezeRequest struct {
	Datastore        string            `description:"registered datastore i.e. db1"`
	SQL              string            `description:"dataset SQL soruce"`
	DestURL          string            `description:"represent dataset destination"`
	OmitEmpty        bool              `description:"flag to skip empty attributes"`
	Ignore           []string          `description:"path to ignore i.e. request.postbody"`
	Replace          map[string]string `description:"key of path with corresponding replacement value"`
	LocationTimezone string            `description:"convert time to specified timezone i.e UTC"`
	Override         map[string]string `description:"overrides column with supplied values"`
	ASCII            []string          `description:"column values to be ascii sanitized"`
	RelativeDate     []string          `description:"transform date to date expr"`
	Obfuscation      []Obfuscation     `description:"obfuscation rules"`
	Reset            bool              `description:"add extra empty record to truncate before inserting"`
	TimeFormat       string            `description:"java/ios based time format"`
	TimeLayout       string            `description:"golang based time layout"`
}

FreezeRequest represent a request to create a data set from datastore for provided SQL and target path

func (*FreezeRequest) Init added in v0.2.1

func (r *FreezeRequest) Init() error

type FreezeResponse added in v0.1.1

type FreezeResponse struct {
	*BaseResponse
	Count   int
	DestURL string
}

FreezeResponse response

type InitRequest added in v0.1.1

type InitRequest struct {
	Datastore string
	Recreate  bool
	*RegisterRequest
	Admin *RegisterRequest
	*MappingRequest
	*RunScriptRequest
}

InitRequest represents datastore init request, it actual aggregates, registraction, recreation, mapping and run script request

func NewInitRequest added in v0.1.1

func NewInitRequest(datastore string, recreate bool, register, admin *RegisterRequest, mapping *MappingRequest, script *RunScriptRequest) *InitRequest

NewInitRequest creates a new database init request

func NewInitRequestFromURL added in v0.1.1

func NewInitRequestFromURL(URL string) (*InitRequest, error)

NewInitRequestFromURL create a request from URL

func (*InitRequest) Init added in v0.1.1

func (r *InitRequest) Init() (err error)

func (*InitRequest) Validate added in v0.1.1

func (r *InitRequest) Validate() error

type InitResponse added in v0.1.1

type InitResponse struct {
	*BaseResponse
	Tables []string
}

InitResponse represent init datastore response

type Mapper added in v0.1.1

type Mapper struct {
	// contains filtered or unexported fields
}

func NewMapper added in v0.1.1

func NewMapper() *Mapper

func (*Mapper) Add added in v0.1.1

func (s *Mapper) Add(mapping *Mapping)

func (*Mapper) Has added in v0.1.1

func (s *Mapper) Has(table string) bool

func (*Mapper) Map added in v0.1.1

func (s *Mapper) Map(dataset *Dataset) []*Dataset

type Mapping added in v0.1.1

type Mapping struct {
	*url.Resource
	*MappingTable
	Name string `required:"true" description:"mapping name (i.e view name)"`
}

Mapping represents mapping

func (*Mapping) Tables added in v0.1.1

func (m *Mapping) Tables() []string

Tables returns tables of this mapping

type MappingColumn added in v0.1.1

type MappingColumn struct {
	Name         string `required:"true" description:"column name"`
	DefaultValue string
	FromColumn   string `description:"if specified it defined value source for this column"`
	Required     bool   `description:"table record will be mapped if values for all required columns are present"`
	Unique       bool   `description:"flag key/s that are unique"`
}

MappingColumn represents column with its source definition.

type MappingRequest added in v0.1.1

type MappingRequest struct {
	Mappings []*Mapping `required:"true" description:"virtual table mapping"`
}

MappingRequest represnet a mapping request

func NewMappingRequest added in v0.1.1

func NewMappingRequest(mappings ...*Mapping) *MappingRequest

NewMappingRequest creates new mapping request

func NewMappingRequestFromURL added in v0.1.1

func NewMappingRequestFromURL(URL string) (*MappingRequest, error)

NewMappingRequestFromURL create a request from URL

func (*MappingRequest) Init added in v0.1.1

func (r *MappingRequest) Init() (err error)

Init init request

func (*MappingRequest) Validate added in v0.1.1

func (r *MappingRequest) Validate() error

type MappingResponse added in v0.1.1

type MappingResponse struct {
	*BaseResponse
	Tables []string
}

MappingResponse represents mapping response

type MappingTable added in v0.1.1

type MappingTable struct {
	Table   string           `required:"true"`
	Columns []*MappingColumn `required:"true"`

	Associations []*MappingTable
}

MappingTable represents a table mapping, mapping allow to route data defined in only one table to many tables.

type ModificationInfo added in v0.1.1

type ModificationInfo struct {
	Subject  string
	Method   string `description:"modification method determined by presence of primary key: load - insert, persist: insert or update"`
	Deleted  int
	Modified int
	Added    int
}

ModificationInfo represents a modification info

type Obfuscation added in v0.10.11

type Obfuscation struct {
	Columns       []string
	Method        ObfuscationMethod
	DictionaryURL string
	Dictionary    []string
	Key           *kms.Key
	IDKey         string
	Template      string
}

func (*Obfuscation) Init added in v0.10.11

func (o *Obfuscation) Init(ctx context.Context)

func (*Obfuscation) Obfuscate added in v0.10.11

func (o *Obfuscation) Obfuscate(ctx context.Context, value string, record map[string]interface{}, column string) (string, error)

type ObfuscationMethod added in v0.10.11

type ObfuscationMethod string

type PingRequest added in v0.7.0

type PingRequest struct {
	Datastore string
	TimeoutMs int
}

PingRequest represents ping request

type PingResponse added in v0.7.0

type PingResponse struct {
	*BaseResponse
}

PingResponse represents a ping response

type PrepareRequest added in v0.1.1

type PrepareRequest struct {
	Expand           bool `description:"substitute $ expression with content of context.state"`
	*DatasetResource `required:"true" description:"datasets resource"`
}

PrepareRequest represents a request to populate datastore with data resource

func NewPrepareRequest added in v0.1.1

func NewPrepareRequest(resource *DatasetResource) *PrepareRequest

NewPrepareRequest creates a new prepare request

func NewPrepareRequestFromURL added in v0.1.1

func NewPrepareRequestFromURL(URL string) (*PrepareRequest, error)

NewPrepareRequestFromURL create a request from URL

func (*PrepareRequest) Validate added in v0.1.1

func (r *PrepareRequest) Validate() error

Validate checks if request is valid

type PrepareResponse added in v0.1.1

type PrepareResponse struct {
	*BaseResponse
	Expand       bool                         `description:"substitute $ expression with content of context.state"`
	Modification map[string]*ModificationInfo `description:"modification info by subject"`
}

PrepareResponse represents a prepare response

type QueryRequest added in v0.1.1

type QueryRequest struct {
	Datastore   string
	SQL         string
	IgnoreError bool
	Expect      []map[string]interface{} `description:"if specified validation would take place"`
}

QueryRequest represents get sequences request

func NewQueryRequest added in v0.1.1

func NewQueryRequest(datastore, SQL string) *QueryRequest

type QueryResponse added in v0.1.1

type QueryResponse struct {
	*BaseResponse
	Records Records
	*assertly.Validation
}

QueryResponse represents get sequences response

type Record added in v0.1.1

type Record map[string]interface{}

func (*Record) AsMap added in v0.1.1

func (r *Record) AsMap() map[string]interface{}

AsMAp returns record as map

func (*Record) Columns added in v0.1.1

func (r *Record) Columns() []string

Columns returns column names.

func (*Record) HasColumn added in v0.1.1

func (r *Record) HasColumn(column string) bool

HasColumn returns true if this row has passed in column value.

func (*Record) IsEmpty added in v0.1.1

func (r *Record) IsEmpty() bool

IsEmpty return true if empty

func (*Record) SetValue added in v0.1.1

func (r *Record) SetValue(column string, value interface{})

SetValue sets column value on this row.

func (*Record) String added in v0.1.1

func (r *Record) String() string

String return row content as string JSON.

func (*Record) Value added in v0.1.1

func (r *Record) Value(column string) interface{}

Value returns raw column value for this row.

func (Record) ValueAsString added in v0.1.1

func (r Record) ValueAsString(column string) string

ValueAsString returns column value as string.

type Records added in v0.1.1

type Records []map[string]interface{}

Records represents table records

func (*Records) Autoincrement added in v0.1.1

func (r *Records) Autoincrement() bool

PrimaryKey returns primary key directive if matched in the following order: @Autoincrement@, @IndexBy@

func (*Records) Columns added in v0.1.1

func (r *Records) Columns() []string

Columns returns unique column names for this dataset

func (*Records) Expand added in v0.1.1

func (r *Records) Expand(context toolbox.Context, includeDirectives bool) (result []interface{}, err error)

Records returns non empty records //directive a filtered out

func (*Records) FromQuery added in v0.1.1

func (r *Records) FromQuery() (string, string)

FromQuery returns value for @FromQuery@ directive

func (*Records) ShouldDeleteAll added in v0.1.1

func (r *Records) ShouldDeleteAll() bool

ShouldDeleteAll checks if dataset contains empty record (indicator to delete all)

func (*Records) UniqueKeys added in v0.1.1

func (r *Records) UniqueKeys() []string

UniqueKeys returns value for unique key directive, it test keys in the following order: @Autoincrement@, @IndexBy@

type RecreateRequest added in v0.1.1

type RecreateRequest struct {
	Datastore      string `` /* 157-byte string literal not displayed */
	AdminDatastore string `description:"database  used to run DDL"`
}

RecreateRequest represent recreate datastore request

func NewRecreateRequest added in v0.1.1

func NewRecreateRequest(datastore, adminDatastore string) *RecreateRequest

NewRecreateRequest create new recreate request

func NewRecreateRequestFromURL added in v0.1.1

func NewRecreateRequestFromURL(URL string) (*RecreateRequest, error)

NewRecreateRequestFromURL create a request from URL

type RecreateResponse added in v0.1.1

type RecreateResponse struct {
	*BaseResponse
}

RecreateResponse represents recreate datastore response

type RegisterRequest added in v0.1.1

type RegisterRequest struct {
	Datastore   string                 `required:"true" description:"datastore name"`
	Config      *dsc.Config            `description:"datastore config"`
	ConfigURL   string                 `description:"datastore config URL"`
	Tables      []*dsc.TableDescriptor `description:"optional table descriptors"`
	PingRequest `json:",inline" yaml:",inline"`
	Ping        bool `description:"flag to wait for database get online"`
}

RegisterRequest represent register request

func NewRegisterRequest added in v0.1.1

func NewRegisterRequest(datastore string, config *dsc.Config, tables ...*dsc.TableDescriptor) *RegisterRequest

NewRegisterRequest create new register request

func NewRegisterRequestFromURL added in v0.1.1

func NewRegisterRequestFromURL(URL string) (*RegisterRequest, error)

func (*RegisterRequest) Init added in v0.1.1

func (r *RegisterRequest) Init() (err error)

func (*RegisterRequest) Validate added in v0.1.1

func (r *RegisterRequest) Validate() error

type RegisterResponse added in v0.1.1

type RegisterResponse struct {
	*BaseResponse
}

RegisterResponse represents register response

type RunSQLRequest added in v0.1.1

type RunSQLRequest struct {
	Datastore string `required:"true" description:"registered datastore name"`
	Expand    bool   `description:"substitute $ expression with content of context.state"`
	SQL       []string
}

RunSQLRequest represents run SQL request

func NewRunSQLRequest added in v0.1.1

func NewRunSQLRequest(datastore string, SQL ...string) *RunSQLRequest

NewRunSQLRequest creates new run SQL request

func NewRunSQLRequestFromURL added in v0.1.1

func NewRunSQLRequestFromURL(URL string) (*RunSQLRequest, error)

NewRunSQLRequestFromURL create a request from URL

type RunSQLResponse added in v0.1.1

type RunSQLResponse struct {
	*BaseResponse
	RowsAffected int
}

RunSQLRequest represents run SQL response

type RunScriptRequest added in v0.1.1

type RunScriptRequest struct {
	Datastore string `required:"true" description:"registered datastore name"`
	Expand    bool   `description:"substitute $ expression with content of context.state"`
	Scripts   []*dsurl.Resource
}

RunScriptRequest represents run SQL Script request

func NewRunScriptRequest added in v0.1.1

func NewRunScriptRequest(datastore string, scripts ...*dsurl.Resource) *RunScriptRequest

NewRunScriptRequest creates new run script request

func NewRunScriptRequestFromURL added in v0.1.1

func NewRunScriptRequestFromURL(URL string) (*RunScriptRequest, error)

NewRunScriptRequestFromURL create a request from URL

type SchemaTableCheck added in v0.10.0

type SchemaTableCheck struct {
	Table string
	*assertly.Validation
}

type SchemaTarget added in v0.10.0

type SchemaTarget struct {
	Datastore  string `description:"datastore"`
	Target     string `description:"target vendor, use only if different than source"`
	MappingURL string `description:"if target driver is used - you can provide data type mapping"`
}

type SequenceRequest added in v0.1.1

type SequenceRequest struct {
	Datastore string
	Tables    []string
}

SequenceRequest represents get sequences request

func NewSequenceRequest added in v0.1.1

func NewSequenceRequest(datastore string, tables ...string) *SequenceRequest

type SequenceResponse added in v0.1.1

type SequenceResponse struct {
	*BaseResponse
	Sequences map[string]int
}

SequenceResponse represents get sequences response

type Service

type Service interface {
	//registry returns registry of registered database managers
	Registry() dsc.ManagerRegistry

	//Register registers new datastore connection
	Register(request *RegisterRequest) *RegisterResponse

	//Recreate remove and creates datastore
	Recreate(request *RecreateRequest) *RecreateResponse

	//RunSQL runs supplied SQL
	RunSQL(request *RunSQLRequest) *RunSQLResponse

	//RunScript runs supplied SQL scripts
	RunScript(request *RunScriptRequest) *RunSQLResponse

	//Add table mapping
	AddTableMapping(request *MappingRequest) *MappingResponse

	//Init datastore, (register, recreated, run sql, add mapping)
	Init(request *InitRequest) *InitResponse

	//Populate database with datasets
	Prepare(request *PrepareRequest) *PrepareResponse

	//Verify datastore with supplied expected datasets
	Expect(request *ExpectRequest) *ExpectResponse

	//Query returns query from database
	Query(request *QueryRequest) *QueryResponse

	//Sequence returns sequence for supplied tables
	Sequence(request *SequenceRequest) *SequenceResponse

	//Freeze creates a dataset from existing database/datastore (reverse engineering test setup/verification)
	Freeze(request *FreezeRequest) *FreezeResponse

	//Dump creates a database schema from existing database for supplied tables, datastore
	Dump(request *DumpRequest) *DumpResponse

	//Compare compares data produces by specified SQLs
	Compare(request *CompareRequest) *CompareResponse

	//CheckSchema checks source and dest schema
	CheckSchema(request *CheckSchemaRequest) *CheckSchemaResponse

	//Ping waits until if database is online or error
	Ping(request *PingRequest) *PingResponse

	SetContext(context toolbox.Context)
}

Service represents test service

func New added in v0.1.1

func New() Service

New creates new dsunit service

func NewServiceClient

func NewServiceClient(serverURL string) Service

NewServiceClient returns a new dsunit service client

type Tester added in v0.1.1

type Tester interface {

	// Register registers new datastore connection
	Register(t *testing.T, request *RegisterRequest) bool

	// RegisterFromURL registers new datastore connection, JSON request is fetched from URL
	RegisterFromURL(t *testing.T, URL string) bool

	// Recreate recreates datastore
	Recreate(t *testing.T, request *RecreateRequest) bool

	// RecreateFromURL recreates datastore, JSON request is fetched from URL
	RecreateFromURL(t *testing.T, URL string) bool

	// RunSQL runs supplied SQL
	RunSQL(t *testing.T, request *RunSQLRequest) bool

	// RunSQLFromURL runs supplied SQL, JSON request is fetched from URL
	RunSQLFromURL(t *testing.T, URL string) bool

	// RunScript runs supplied SQL scripts
	RunScript(t *testing.T, request *RunScriptRequest) bool

	// RunScriptFromURL runs supplied SQL scripts, JSON request is fetched from URL
	RunScriptFromURL(t *testing.T, URL string) bool

	// AddTableMapping adds table mapping
	AddTableMapping(t *testing.T, request *MappingRequest) bool

	// AddTableMappingFromURL adds table mapping, JSON request is fetched from URL
	AddTableMappingFromURL(t *testing.T, URL string) bool

	//Init inits datastore, (register, recreated, run sql, add mapping)
	Init(t *testing.T, request *InitRequest) bool

	// InitFromURL inits datastore, (register, recreated, run sql, add mapping), JSON request is fetched from URL
	InitFromURL(t *testing.T, URL string) bool

	// Prepare populates database with datasets
	Prepare(t *testing.T, request *PrepareRequest) bool

	// PrepareFromURL populates database with datasets, JSON request is fetched from URL
	PrepareFromURL(t *testing.T, URL string) bool

	// PopulateWithURL populates database with datasets, JSON requests are fetched from files in directory
	PopulateWithURL(t *testing.T, URL string, datastore string, datasets ...*Dataset) bool

	//PrepareDatastore matches all dataset files that are in the same location as a test file, with the same test file prefix, followed by lowe camel case test name.
	PrepareDatastore(t *testing.T, datastore string) bool

	// PrepareFor matches all dataset files that are located in baseDirectory with method name and
	// populate datastore with all listed dataset
	// Note the matchable dataset files in the base directory have the following naming:
	//
	//  <lower_underscore method name>_populate_<table>.[json|csv]
	//  To prepare dataset to populate datastore table: 'users' and 'permissions' for test method ReadAll you would
	//  have you create the following files in the baseDirectory
	//
	//  read_all_prepare_travelers2.json
	//  read_all_populate_permissions.json
	//
	PrepareFor(t *testing.T, datastore string, baseDirectory string, method string) bool

	// Expect verifies datastore with supplied expected datasets
	Expect(t *testing.T, request *ExpectRequest) bool

	// ExpectFromURL verifies datastore with supplied expected datasets, JSON request is fetched from URL
	ExpectFromURL(t *testing.T, URL string) bool

	// ExpectWithURL verifies datastore with supplied expected datasets, JSON requests are fetched from files in directory
	ExpectWithURL(t *testing.T, checkPolicy int, URL string, datastore string, datasets ...*Dataset) bool

	// ExpectDatasets matches all dataset files that are located in the same directory as the test file with method name to
	// verify that all listed dataset values are present in datastore
	ExpectDatasets(t *testing.T, datastore string, checkPolicy int) bool

	// ExpectFor matches all dataset files that are located in baseDirectory with method name to
	// verify that all listed dataset values are present in datastore
	// Note the matchable dataset files in the base directory have the following naming:
	//
	//  <lower_underscore method name>_expect_<table>.[json|csv]
	//  To prepare expected dataset table: 'users' and 'permissions' for test method ReadAll you would
	//  have you create the following files in the baseDirectory
	//
	//  read_all_expect_users.json
	//  read_all_expect_permissions.json
	//
	ExpectFor(t *testing.T, datastore string, checkPolicy int, baseDirectory string, method string) bool

	// Ping waits until database is online or error
	Ping(t *testing.T, datastore string, timeoutMs int) bool
}

func NewRemoveTester added in v0.1.1

func NewRemoveTester(endpoint string) Tester

NewRemoveTester creates a new remove tester

func NewTester added in v0.1.1

func NewTester() Tester

NewTester creates a new local tester

Directories

Path Synopsis
Package server - Remote testing dsunit server
Package server - Remote testing dsunit server

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL