Documentation ¶
Overview ¶
Package pgranger implements a sharding library to be run in the application processes.
Examples:
For initiation: pgRanger, err := pgranger.NewPGRanger(shardingConfigStr, &dbSecret, dbFactory) if err != nil { log.Fataf("server cannot init PGRanger : %s", err) } defer pgRanger.Close()
For pointwise routing to a db instance/shard: At top of request handler.
dbCtrler, err := reqHandler.pgRanger.FindDBControllerByUUID(shardingFieldName, shardingFieldVal, false) if err != nil { return nil, err } tx, err := dbCtrler.DB().Begin() err = tx.Exec(...) err = tx.Commit()
Index ¶
- func ConvertUUIDsToBytes(shardingFieldUUIDs []uuid.UUID) ([][]byte, error)
- func CreateShardingData(shardingConfigStr *string, dbSecret *DBSecret, dbFactory SqlxDBFactory) (*shardingData, error)
- func GetDbURI(dbName string, dbHost string, dbPort int) string
- func LoadLocalConfigFile(localConfigFilePath string) (*string, error)
- type DBConfig
- type DBConnConfig
- type DBController
- type DBControllerConfig
- type DBControllerImpl
- type DBRunnable
- type DBSecret
- type FanOutContext
- type FanOutResult
- type Namespace
- type NamespaceConfig
- type PGRanger
- func (m *PGRanger) Close() error
- func (m *PGRanger) ExecuteFanOut(fanOutProcessor PointFanOutProcessor, isReplica bool) (*FanOutResult, error)
- func (m *PGRanger) FindAllDBControllers(isReplica bool) (map[string]DBController, error)
- func (m *PGRanger) FindDBController(shardingFieldName string, shardingFieldVal []byte, isReplica bool) (DBController, error)
- func (m *PGRanger) FindDBControllerByUUID(shardingFieldName string, shardingFieldVal uuid.UUID, isReplica bool) (DBController, error)
- func (m *PGRanger) FindDBControllers(shardingFieldName string, shardingFieldValues [][]byte, isReplica bool) ([]string, map[string][][]byte, map[string]DBController, error)
- type PointFanOutProcessor
- type Range
- type RangeConfig
- type ShardingConfig
- type SqlxDBFactory
- type SqlxDBFactoryImpl
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func ConvertUUIDsToBytes ¶
ConvertUUIDsToBytes converts uuids to [][]byte
func CreateShardingData ¶
func CreateShardingData( shardingConfigStr *string, dbSecret *DBSecret, dbFactory SqlxDBFactory, ) (*shardingData, error)
CreateShardingData creates the sharding data object.
func LoadLocalConfigFile ¶
Types ¶
type DBConnConfig ¶
type DBController ¶
type DBControllerConfig ¶
type DBControllerConfig struct {
DB *DBConfig `json:"db"`
}
type DBControllerImpl ¶
type DBControllerImpl struct {
// contains filtered or unexported fields
}
DBControllerImpl implements DBController.
func (*DBControllerImpl) DB ¶
func (d *DBControllerImpl) DB() *sqlx.DB
DB returns the *sqlx.DB client.
func (*DBControllerImpl) URI ¶
func (d *DBControllerImpl) URI() string
URI returns the URI of the db client.
type DBRunnable ¶
type DBRunnable interface {
Run(resChan chan interface{}, errChan chan error)
}
type FanOutContext ¶
type FanOutContext struct { // DBURIToDBRunnable contains mapping from DB URIs to DBRunnables. DBURIToDBRunnable map[string]DBRunnable }
FanOutContext is the context/input for the fan out execution.
type FanOutResult ¶
type FanOutResult struct { // DBResults contains result per DB. DBResults map[string]interface{} // ErrorDetails contains error per DB. ErrorDetails map[string]error }
FanOutResult is the result from the fan out execution.
func Execute ¶
func Execute(fanOutContext *FanOutContext) (*FanOutResult, error)
Execute executes fanout runnables in parallel.
type Namespace ¶
type Namespace struct {
// contains filtered or unexported fields
}
Namespace provides the ranges and corresponding db connections for a sharding key.
type NamespaceConfig ¶
type NamespaceConfig struct {
Ranges []*RangeConfig `json:"ranges"`
}
type PGRanger ¶
type PGRanger struct {
// contains filtered or unexported fields
}
PGRanger manages all sharding information and behaviors around sharding.
func NewPGRanger ¶
func NewPGRanger( shardingConfigStr *string, dbSecret *DBSecret, dbFactory SqlxDBFactory, ) (*PGRanger, error)
NewPGRanger creates the PGRanger.
func (*PGRanger) Close ¶
Close will close the db connections in the struct. If there are multiple errors in closing we keep the last one.
func (*PGRanger) ExecuteFanOut ¶
func (m *PGRanger) ExecuteFanOut(fanOutProcessor PointFanOutProcessor, isReplica bool) (*FanOutResult, error)
ExecuteFanOut is the entry point for fanning out to a bunch of dbs from application logic. fanOutProcessor is the callback provided by the application to be called during the processing. The application will need to implement and pass in this PointFanOutProcessor interface. PointFanOutProcessor is named so because the fan out input are point values. In the future we could have range values as fan out input.
func (*PGRanger) FindAllDBControllers ¶
func (m *PGRanger) FindAllDBControllers(isReplica bool) (map[string]DBController, error)
FindAllDBControllers returns all db controllers. Make a copy so users can not accidentally modify shardingData.dbURIToDBController.
func (*PGRanger) FindDBController ¶
func (m *PGRanger) FindDBController( shardingFieldName string, shardingFieldVal []byte, isReplica bool, ) (DBController, error)
FindDBController finds a db controller by []byte.
func (*PGRanger) FindDBControllerByUUID ¶
func (m *PGRanger) FindDBControllerByUUID( shardingFieldName string, shardingFieldVal uuid.UUID, isReplica bool, ) (DBController, error)
FindDBControllerByUUID finds a db controller by uuid.
type PointFanOutProcessor ¶
type PointFanOutProcessor interface { // Name of the PointFanOutProcessor instance. Name() string // ProvideShardingFieldNameAndValues provides the sharding field name and values for PGRanger to look up. ProvideShardingFieldNameAndValues() (*string, [][]byte) // CreateFanOutContextForAllDBs is a callback that will provide runnables for all db shards. Implement this if your // use case needs requests to all db shards. CreateFanOutContextForAllDBs(dbURIToDBCtrler map[string]DBController) (*FanOutContext, error) // CreateFanOutContext is a callback to provide the runnables for specific db shards with input which are decided // by pgranger based on output of FetchShardingFieldNameAndValues. CreateFanOutContext( dbURIs []string, dbURIToShardingFieldVals map[string][][]byte, dbURIToDBCtrler map[string]DBController, ) (*FanOutContext, error) }
PointFanOutProcessor is to be implemented by users who needs fan out to db instances. It is to be provided as an input to ExecuteFanOut(). There are two key steps:
- The instance of implementation class of PointFanOutProcessor should provide the sharding field name and values for the fan out. For this, implement ProvideShardingFieldNameAndValues(). The PGRanger will then find the correct db instances hosting the sharding field values. If fan out to all db instances are needed, return nil from ProvideShardingFieldNameAndValues().
- Now that PGRanger has decided the sharding field values and the corresponding db instances, it needs the runnables for "what to do" on each db instance. For this purpose, PGRanger will feed the sharding field values and db instances into the CreateFanOutContext or CreateFanOutContextForAllDBs for db runnable creation.
PGRanger will then run all the runnables returned from step 2 in parallel in ExecuteFanOut.
type Range ¶
type Range struct {
// contains filtered or unexported fields
}
Range provides begin and end of a range and the db stores. Each range represents a micro-partition that can be moved across shards as one unit. It would make sense to split the sharding key space into a reasonably large amount of ranges.
type RangeConfig ¶
type ShardingConfig ¶
type ShardingConfig struct { Version *int32 `json:"version"` Namespaces map[string]*NamespaceConfig `json:"namespaces"` DBControllers []*DBControllerConfig `json:"dbControllers"` }
type SqlxDBFactory ¶
Source Files ¶
Directories ¶
Path | Synopsis |
---|---|
example
|
|
userprofile
In this example, we have a user profile application which has its sharding key user_id spreaded over 4 ranges on 2 host shards.
|
In this example, we have a user profile application which has its sharding key user_id spreaded over 4 ranges on 2 host shards. |