qlbridge: github.com/araddon/qlbridge/datasource Index | Files | Directories

package datasource

import "github.com/araddon/qlbridge/datasource"

Copyright 2011 The Go Authors. All rights reserved. Use of this source code is governed by a BSD-style license that can be found in the LICENSE file.

Datasource package contains database/source type related. A few datasources are implemented here (test, csv). This package also includes schema base services (datasource registry).

Index

Package Files

context.go context_wrapper.go csv.go datatypes.go doc.go introspect.go json.go key.go schemadb.go session.go

Constants

const (
    // Default Max Allowed packets for connections
    MaxAllowedPacket = 4194304
)
const (
    // SchemaDbSourceType is schemadb source type name
    SchemaDbSourceType = "schemadb"
)

Variables

var (

    // DialectWriterCols list of columns for dialectwriter.
    DialectWriterCols = []string{"mysql"}
    // DialectWriters list of differnt writers.
    DialectWriters = []schema.DialectWriter{&mysqlWriter{}}
)
var (
    // ErrNotDate an error for trying to corece/convert to Time a field that is not a time.
    ErrNotDate = errors.New("Unable to conver to time value")
)
var (
    // IntrospectCount is default number of rows to evaluate for introspection
    // based schema discovery.
    IntrospectCount = 20
)

func IntrospectSchema Uses

func IntrospectSchema(s *schema.Schema, name string, iter schema.Iterator) error

IntrospectSchema discover schema from contents of row introspection.

func IntrospectTable Uses

func IntrospectTable(tbl *schema.Table, iter schema.Iterator) error

IntrospectTable accepts a table and schema Iterator and will read a representative sample of rows, introspecting the results to create a schema. Generally used for CSV, Json files to create strongly typed schemas.

func KeyFromWhere Uses

func KeyFromWhere(wh interface{}) schema.Key

Given a Where expression, lets try to create a key which

requires form    `idenity = "value"`

func MessageConversion Uses

func MessageConversion(vals []interface{}) []schema.Message

MessageConversion convert values of type schema.Message.

func MysqlValueString Uses

func MysqlValueString(t value.ValueType) string

func NewMySqlSessionVars Uses

func NewMySqlSessionVars() expr.ContextReadWriter

func NewNamespacedContextReader Uses

func NewNamespacedContextReader(basereader expr.ContextReader, namespace string) expr.ContextReader

NewNestedContextReader provides a context reader which prefixes all keys with a name space. This is useful if you have overlapping field names between ContextReaders within a NestedContextReader.

msg.Get("foo.key")

func NewNestedContextReadWriter Uses

func NewNestedContextReadWriter(readers []expr.ContextReader, writer expr.ContextWriter, ts time.Time) expr.ContextReadWriter

NewNestedContextReader provides a context reader which is a composite of ordered child readers the first reader with a key will be used

func NewNestedContextReader Uses

func NewNestedContextReader(readers []expr.ContextReader, ts time.Time) expr.ContextReader

NewNestedContextReader provides a context reader which is a composite of ordered child readers the first reader with a key will be used

func RowsForSession Uses

func RowsForSession(ctx *plan.Context) [][]driver.Value

func SchemaDBStoreProvider Uses

func SchemaDBStoreProvider(s *schema.Schema) schema.Source

SchemaDBStoreProvider create source for schemadb

type ContextSimple Uses

type ContextSimple struct {
    Data map[string]value.Value
    // contains filtered or unexported fields
}

func NewContextMap Uses

func NewContextMap(data map[string]interface{}, namespacing bool) *ContextSimple

func NewContextMapTs Uses

func NewContextMapTs(data map[string]interface{}, namespacing bool, ts time.Time) *ContextSimple

func NewContextSimple Uses

func NewContextSimple() *ContextSimple

func NewContextSimpleData Uses

func NewContextSimpleData(data map[string]value.Value) *ContextSimple

func NewContextSimpleNative Uses

func NewContextSimpleNative(data map[string]interface{}) *ContextSimple

func NewContextSimpleTs Uses

func NewContextSimpleTs(data map[string]value.Value, ts time.Time) *ContextSimple

func NewMySqlGlobalVars Uses

func NewMySqlGlobalVars() *ContextSimple

func (*ContextSimple) All Uses

func (m *ContextSimple) All() map[string]value.Value

func (*ContextSimple) Body Uses

func (m *ContextSimple) Body() interface{}

func (*ContextSimple) Commit Uses

func (m *ContextSimple) Commit(rowInfo []expr.SchemaInfo, row expr.RowWriter) error

func (*ContextSimple) Delete Uses

func (m *ContextSimple) Delete(row map[string]value.Value) error

func (ContextSimple) Get Uses

func (m ContextSimple) Get(key string) (value.Value, bool)

func (*ContextSimple) Id Uses

func (m *ContextSimple) Id() uint64

func (*ContextSimple) Put Uses

func (m *ContextSimple) Put(col expr.SchemaInfo, rctx expr.ContextReader, v value.Value) error

func (*ContextSimple) Row Uses

func (m *ContextSimple) Row() map[string]value.Value

func (*ContextSimple) SupportNamespacing Uses

func (m *ContextSimple) SupportNamespacing()

func (*ContextSimple) Ts Uses

func (m *ContextSimple) Ts() time.Time

type ContextWrapper Uses

type ContextWrapper struct {
    // contains filtered or unexported fields
}

func NewContextWrapper Uses

func NewContextWrapper(val interface{}) *ContextWrapper

func (*ContextWrapper) Get Uses

func (m *ContextWrapper) Get(key string) (value.Value, bool)

func (*ContextWrapper) Row Uses

func (m *ContextWrapper) Row() map[string]value.Value

func (*ContextWrapper) Ts Uses

func (m *ContextWrapper) Ts() time.Time

type CsvDataSource Uses

type CsvDataSource struct {
    // contains filtered or unexported fields
}

Csv DataSource, implements qlbridge schema DataSource, SourceConn, Scanner

to allow csv files to be full featured databases.
- very, very naive scanner, forward only single pass
- can open a file with .Open()
- assumes comma delimited
- not thread-safe
- does not implement write operations

func NewCsvSource Uses

func NewCsvSource(table string, indexCol int, ior io.Reader, exit <-chan bool) (*CsvDataSource, error)

NewCsvSource reader assumes we are getting first row as headers - optionally may be gzipped

func (*CsvDataSource) Close Uses

func (m *CsvDataSource) Close() error

func (*CsvDataSource) Columns Uses

func (m *CsvDataSource) Columns() []string

func (*CsvDataSource) Init Uses

func (m *CsvDataSource) Init()

func (*CsvDataSource) Next Uses

func (m *CsvDataSource) Next() schema.Message

func (*CsvDataSource) Open Uses

func (m *CsvDataSource) Open(connInfo string) (schema.Conn, error)

func (*CsvDataSource) Setup Uses

func (m *CsvDataSource) Setup(*schema.Schema) error

func (*CsvDataSource) Table Uses

func (m *CsvDataSource) Table(tableName string) (*schema.Table, error)

func (*CsvDataSource) Tables Uses

func (m *CsvDataSource) Tables() []string

type FileLineHandler Uses

type FileLineHandler func(line []byte) (schema.Message, error)

type JsonHelperScannable Uses

type JsonHelperScannable u.JsonHelper

JsonHelperScannable expects map json's (not array) map[string]interface

func (*JsonHelperScannable) MarshalJSON Uses

func (m *JsonHelperScannable) MarshalJSON() ([]byte, error)

func (*JsonHelperScannable) Scan Uses

func (m *JsonHelperScannable) Scan(src interface{}) error

Scan the database/sql interface for scanning sql byte vals into this typed structure.

func (*JsonHelperScannable) UnmarshalJSON Uses

func (m *JsonHelperScannable) UnmarshalJSON(data []byte) error

UnmarshalJSON bytes into this typed struct

func (JsonHelperScannable) Value Uses

func (m JsonHelperScannable) Value() (driver.Value, error)

Value This is the go sql/driver interface we need to implement to allow conversion back forth

type JsonSource Uses

type JsonSource struct {
    // contains filtered or unexported fields
}

JsonSource implements qlbridge schema DataSource, SourceConn, Scanner to allow new line delimited json files to be full featured databases. - very, very naive scanner, forward only single pass - can open a file with .Open() - not thread-safe - does not implement write operations

func NewJsonSource Uses

func NewJsonSource(table string, rc io.ReadCloser, exit <-chan bool, lh FileLineHandler) (*JsonSource, error)

NewJsonSource reader assumes we are getting NEW LINE delimted json file - optionally may be gzipped

func (*JsonSource) Close Uses

func (m *JsonSource) Close() error

func (*JsonSource) Columns Uses

func (m *JsonSource) Columns() []string

func (*JsonSource) CreateIterator Uses

func (m *JsonSource) CreateIterator() schema.Iterator

func (*JsonSource) Init Uses

func (m *JsonSource) Init()

func (*JsonSource) Next Uses

func (m *JsonSource) Next() schema.Message

func (*JsonSource) Open Uses

func (m *JsonSource) Open(connInfo string) (schema.Conn, error)

func (*JsonSource) Setup Uses

func (m *JsonSource) Setup(*schema.Schema) error

func (*JsonSource) Table Uses

func (m *JsonSource) Table(tableName string) (*schema.Table, error)

func (*JsonSource) Tables Uses

func (m *JsonSource) Tables() []string

type JsonWrapper Uses

type JsonWrapper json.RawMessage

JsonWrapper json data

func (*JsonWrapper) MarshalJSON Uses

func (m *JsonWrapper) MarshalJSON() ([]byte, error)

func (*JsonWrapper) Scan Uses

func (m *JsonWrapper) Scan(src interface{}) error

func (*JsonWrapper) Unmarshal Uses

func (m *JsonWrapper) Unmarshal(v interface{}) error

func (*JsonWrapper) UnmarshalJSON Uses

func (m *JsonWrapper) UnmarshalJSON(data []byte) error

UnmarshalJSON bytes into this typed struct

func (JsonWrapper) Value Uses

func (m JsonWrapper) Value() (driver.Value, error)

Value This is the go sql/driver interface we need to implement to allow conversion back forth

type KeyCol Uses

type KeyCol struct {
    Name string
    Val  driver.Value
}

Variety of Key Types

func NewKeyCol Uses

func NewKeyCol(name string, val driver.Value) KeyCol

func (KeyCol) Key Uses

func (m KeyCol) Key() driver.Value

type KeyInt Uses

type KeyInt struct {
    Id int
}

Variety of Key Types

func NewKeyInt Uses

func NewKeyInt(key int) KeyInt

func (*KeyInt) Key Uses

func (m *KeyInt) Key() driver.Value

type KeyInt64 Uses

type KeyInt64 struct {
    Id int64
}

Variety of Key Types

func NewKeyInt64 Uses

func NewKeyInt64(key int64) KeyInt64

func (*KeyInt64) Key Uses

func (m *KeyInt64) Key() driver.Value

type NamespacedContextReader Uses

type NamespacedContextReader struct {
    // contains filtered or unexported fields
}

func (*NamespacedContextReader) Get Uses

func (n *NamespacedContextReader) Get(key string) (value.Value, bool)

func (*NamespacedContextReader) Row Uses

func (n *NamespacedContextReader) Row() map[string]value.Value

func (*NamespacedContextReader) Ts Uses

func (n *NamespacedContextReader) Ts() time.Time

type NestedContextReader Uses

type NestedContextReader struct {
    // contains filtered or unexported fields
}

func (*NestedContextReader) Delete Uses

func (n *NestedContextReader) Delete(delRow map[string]value.Value) error

func (*NestedContextReader) Get Uses

func (n *NestedContextReader) Get(key string) (value.Value, bool)

func (*NestedContextReader) Put Uses

func (n *NestedContextReader) Put(col expr.SchemaInfo, readCtx expr.ContextReader, v value.Value) error

func (*NestedContextReader) Row Uses

func (n *NestedContextReader) Row() map[string]value.Value

func (*NestedContextReader) Ts Uses

func (n *NestedContextReader) Ts() time.Time

type SchemaDb Uses

type SchemaDb struct {
    // contains filtered or unexported fields
}

SchemaDb Static Schema Source, implements qlbridge DataSource to allow in-memory native go data to have a Schema and implement and be operated on by Sql Operations.

func NewSchemaDb Uses

func NewSchemaDb(s *schema.Schema) *SchemaDb

NewSchemaDb create new db for storing schema.

func (*SchemaDb) Close Uses

func (m *SchemaDb) Close() error

Close down everything.

func (*SchemaDb) DropTable Uses

func (m *SchemaDb) DropTable(t string) error

func (*SchemaDb) Init Uses

func (m *SchemaDb) Init()

Init initialize

func (*SchemaDb) Open Uses

func (m *SchemaDb) Open(schemaObjectName string) (schema.Conn, error)

Open Create a SchemaSource specific to schema object (table, database)

func (*SchemaDb) Setup Uses

func (m *SchemaDb) Setup(*schema.Schema) error

Setup the schemadb

func (*SchemaDb) Table Uses

func (m *SchemaDb) Table(table string) (*schema.Table, error)

Table get schema Table

func (*SchemaDb) Tables Uses

func (m *SchemaDb) Tables() []string

Tables list of table names.

type SchemaSource Uses

type SchemaSource struct {
    // contains filtered or unexported fields
}

SchemaSource type for the schemadb connection (thread-safe).

func (*SchemaSource) Close Uses

func (m *SchemaSource) Close() error

func (*SchemaSource) Columns Uses

func (m *SchemaSource) Columns() []string

func (*SchemaSource) Get Uses

func (m *SchemaSource) Get(key driver.Value) (schema.Message, error)

func (*SchemaSource) Next Uses

func (m *SchemaSource) Next() schema.Message

func (*SchemaSource) SetContext Uses

func (m *SchemaSource) SetContext(ctx *plan.Context)

SetContext set the plan context

func (*SchemaSource) SetRows Uses

func (m *SchemaSource) SetRows(rows [][]driver.Value)

type SqlDriverMessage Uses

type SqlDriverMessage struct {
    Vals  []driver.Value
    IdVal uint64
}

SqlDriverMessage context message of values.

func NewSqlDriverMessage Uses

func NewSqlDriverMessage(id uint64, row []driver.Value) *SqlDriverMessage

func (*SqlDriverMessage) Body Uses

func (m *SqlDriverMessage) Body() interface{}

func (*SqlDriverMessage) Id Uses

func (m *SqlDriverMessage) Id() uint64

func (*SqlDriverMessage) ToMsgMap Uses

func (m *SqlDriverMessage) ToMsgMap(colidx map[string]int) *SqlDriverMessageMap

type SqlDriverMessageMap Uses

type SqlDriverMessageMap struct {
    Vals     []driver.Value // Values
    ColIndex map[string]int // Map of column names to ordinal position in vals
    IdVal    uint64         // id()
    // contains filtered or unexported fields
}

SqlDriverMessageMap Context message with column/position info.

func NewSqlDriverMessageMap Uses

func NewSqlDriverMessageMap(id uint64, row []driver.Value, colindex map[string]int) *SqlDriverMessageMap

func NewSqlDriverMessageMapCtx Uses

func NewSqlDriverMessageMapCtx(id uint64, ctx expr.ContextReader, colindex map[string]int) *SqlDriverMessageMap

func NewSqlDriverMessageMapEmpty Uses

func NewSqlDriverMessageMapEmpty() *SqlDriverMessageMap

func NewSqlDriverMessageMapVals Uses

func NewSqlDriverMessageMapVals(id uint64, row []driver.Value, cols []string) *SqlDriverMessageMap

func (*SqlDriverMessageMap) Body Uses

func (m *SqlDriverMessageMap) Body() interface{}

func (*SqlDriverMessageMap) Copy Uses

func (m *SqlDriverMessageMap) Copy() *SqlDriverMessageMap

func (*SqlDriverMessageMap) Get Uses

func (m *SqlDriverMessageMap) Get(key string) (value.Value, bool)

func (*SqlDriverMessageMap) Id Uses

func (m *SqlDriverMessageMap) Id() uint64

func (*SqlDriverMessageMap) Key Uses

func (m *SqlDriverMessageMap) Key() driver.Value

func (*SqlDriverMessageMap) Row Uses

func (m *SqlDriverMessageMap) Row() map[string]value.Value

func (*SqlDriverMessageMap) SetKey Uses

func (m *SqlDriverMessageMap) SetKey(key string)

func (*SqlDriverMessageMap) SetKeyHashed Uses

func (m *SqlDriverMessageMap) SetKeyHashed(key string)

func (*SqlDriverMessageMap) SetRow Uses

func (m *SqlDriverMessageMap) SetRow(row []driver.Value)

func (*SqlDriverMessageMap) Ts Uses

func (m *SqlDriverMessageMap) Ts() time.Time

func (*SqlDriverMessageMap) Values Uses

func (m *SqlDriverMessageMap) Values() []driver.Value

type StringArray Uses

type StringArray []string

StringArray Convert json to array of strings

func (*StringArray) MarshalJSON Uses

func (m *StringArray) MarshalJSON() ([]byte, error)

func (*StringArray) Scan Uses

func (m *StringArray) Scan(src interface{}) error

Scan the database/sql interface for scanning sql byte vals into this typed structure.

func (*StringArray) UnmarshalJSON Uses

func (m *StringArray) UnmarshalJSON(data []byte) error

func (StringArray) Value Uses

func (m StringArray) Value() (driver.Value, error)

Value convert string to json values

type TimeValue Uses

type TimeValue time.Time

TimeValue Convert a string/bytes to time.Time by parsing the string with a wide variety of different date formats that are supported in http://godoc.org/github.com/araddon/dateparse

func (*TimeValue) MarshalJSON Uses

func (m *TimeValue) MarshalJSON() ([]byte, error)

func (*TimeValue) Scan Uses

func (m *TimeValue) Scan(src interface{}) error

func (*TimeValue) Time Uses

func (m *TimeValue) Time() time.Time

func (*TimeValue) UnmarshalJSON Uses

func (m *TimeValue) UnmarshalJSON(data []byte) error

func (TimeValue) Value Uses

func (m TimeValue) Value() (driver.Value, error)

Directories

PathSynopsis
filesPackage files is a cloud (gcs, s3) and local file datasource that translates json, csv, files into appropriate interface for qlbridge DataSource so we can run queries.
membtreeMembtree implements a Datasource in-memory implemenation using the google btree.
memdbMemdb package implements a Qlbridge Datasource in-memory implemenation using the hashicorp go-memdb (immuteable radix tree's).
mockcsvPackage mockcsv implements an in-memory csv data source for testing usage implemented by wrapping the mem-b-tree, loading csv data into it.
mockcsvtestdataPackage mockcsvtestdata is csv test data only used for tests.
sqlitePackage sqlite implements a Qlbridge Datasource interface around sqlite that translates mysql syntax to sqlite.

Package datasource imports 22 packages (graph) and is imported by 73 packages. Updated 2019-10-28. Refresh now. Tools for package owners.