README
¶
Source
import "github.com/coralproject/sponge/pkg/source"
Package source has the drivers to connect to the external source and retrieve data. The credentials for the external source are setup in the strategy file.
Variables
var strategy str.Strategy
Holds the credentials for the external source as well as all the entities that needs to be extracted.
var uuid string
Universally Unique Identifier used for the logs.
var credential str.Credential
Credential for the external source (database or web service).
Interface Sourcer
This is the interface that needs to be implemented by any driver to databases that wants to be included.
func GetData
`GetData(string, *Options) ([]map[string]interface{}, error)`
Returns all the data (query by options in Options) in the format []map[string]interface{}
func IsWebService
`IsWebService() bool`
Returns true if the imlementation of sourcer is a web service.
func New
`func New(d string) (Sourcer, error)`
Depending on the parameter it returns a structure with the connection to the external source that implements the interface Sourcer.
func GetEntities
func GetEntities() ([]string, error)
Gets all the entities names from the source
func GetforeignEntity
`func GetForeignEntity(name string) string`
Gets the foreign entinty's name for the coral collection.
Driver mysql
Structure mysql that implements interface Sourcer. It gets data from a Mysql database.
Driver postgresql
Structure postgresql that implements interface Sourcer. It gets data from a Postgresql database.
Driver mongodb
Structure to hold a connection to a mongoDB that implements interface Sourcer. It gets data from the mongo database.
API
Structure the way to get data from a web service that implements interface Sourcer.
How to add a new source
If you need to add a new external source implements the interface Sourcer over an structure with the connection to the database.
Documentation
¶
Overview ¶
Package source implements a way to get data from external MongoDB sources.
Package source implements a way to get data from external MySQL sources.
Package source implements a way to get data from external PostgreSQL sources.
Package source implements a way to get data from external sources.
External possible sources: * MySQL * PostgreSQL * MongoDB * Webservice
Index ¶
- func GetEntities() ([]string, error)
- func GetForeignEntity(name string) string
- func Init(u string) (string, error)
- type API
- func (a API) GetData(entity string, options *Options) ([]map[string]interface{}, error)
- func (a API) GetFireHoseData(pageAfter string) ([]map[string]interface{}, string, error)
- func (a API) GetQueryData(entityname string, options *Options, ids []string) ([]map[string]interface{}, error)
- func (a API) GetWebServiceData() ([]map[string]interface{}, bool, error)
- func (a API) IsWebService() bool
- type MongoDB
- type MySQL
- type Options
- type PostgreSQL
- type QueryData
- type Sourcer
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func GetEntities ¶
GetEntities gets all the entities names from this data source
func GetForeignEntity ¶
GetForeignEntity returns the foreign's source of the entity
Types ¶
type API ¶
type API struct {
Connection string
}
API is the struct that has the connection string to the external mysql database
func (API) GetData ¶
GetData does the request to the webservice once and get back the data based on the parameters
func (API) GetFireHoseData ¶
GetFireHoseData use the firehose to constantly GET data from the web service
func (API) GetQueryData ¶
func (a API) GetQueryData(entityname string, options *Options, ids []string) ([]map[string]interface{}, error)
GetQueryData will return all the data based on a specific list of IDs
func (API) GetWebServiceData ¶
GetWebServiceData does the request to the webservice once and get back the data
func (API) IsWebService ¶
IsWebService is a func from the Sourcer interface. It tell us if the external source is a database or API
type MongoDB ¶
type MongoDB struct {
Connection string
Database *mgo.Session
}
MongoDB is the struct that has the connection string to the external mysql database
func (MongoDB) GetQueryData ¶
func (m MongoDB) GetQueryData(entity string, options *Options, ids []string) ([]map[string]interface{}, error)
GetQueryData needs to be implemented for mongodb to implement the sourcer interface
func (MongoDB) IsWebService ¶
IsWebService is used to check what is that sourcerer interface
type MySQL ¶
MySQL is the struct that has the connection string to the external mysql database
func (MySQL) GetQueryData ¶
func (m MySQL) GetQueryData(entityname string, options *Options, ids []string) ([]map[string]interface{}, error)
GetQueryData returns the raw data from the table based on the ids
func (MySQL) IsWebService ¶
IsWebService returns true only if the implementation of Sourcer is an API
type Options ¶
type Options struct {
Limit int
Offset int
Orderby string
Query string
Types string
Importonlyfailed bool
ReportOnFailedRecords bool
Reportdbfile string
TimeWaiting int
}
Options will hold all the options that came from flags
type PostgreSQL ¶
PostgreSQL is the struct that has the connection string to the external postgresql database
func (PostgreSQL) GetData ¶
func (p PostgreSQL) GetData(entityname string, options *Options) ([]map[string]interface{}, error)
GetData returns the raw data from the tableName
func (PostgreSQL) GetQueryData ¶
func (p PostgreSQL) GetQueryData(entityname string, options *Options, ids []string) ([]map[string]interface{}, error)
GetQueryData returns the raw data from the tableName
func (PostgreSQL) IsWebService ¶
func (p PostgreSQL) IsWebService() bool
IsWebService is a func from the Sourcer interface to check if the external source is api or database
type Sourcer ¶
type Sourcer interface {
// GetData returns data for a specific entity filter by all the options
GetData(string, *Options) ([]map[string]interface{}, error) //int, int, string, string // args ...interface{}) ([]map[string]interface{}, error) //
GetQueryData(string, *Options, []string) ([]map[string]interface{}, error) //int, int, string
IsWebService() bool
}
Sourcer is where the data is coming from (webservice or database)
Source Files