pgranger

package module
v0.0.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 24, 2022 License: Apache-2.0 Imports: 11 Imported by: 0

README

PGRanger

Client side sharding library for Postgresql in Golang. Application processes can load sharding information in memory and talk to a group of postgres instances with no middleman. Application can assume there is one single db instance in the transaction code. The sharding key space can be custom divided by user into micro-partition which will be moved as a unit, e.g. we can divide the sharding key of user_id into 256 ranges upfront, located on 16 db hosts, and if we need to scale up, we can move some ranges to new db hosts.

Full example in example/userprofile.

Advantages:

  1. Compared with Vitess, which does not have Postgres support, pgranger eliminates the middle man like VTGate, VTTablet, and thus reduces latency and failure points. It is also simpler to understand and maintain.
  2. Compared with CockroachDB, pgranger is faster on what it can do, namely the single shard data access. This is because pgranger is essential postgres underhood, which is faster than CockroachDB when not overwhelmed.
  3. Compared with application level sharding, pgranger is strictly better because it does not leak sharding information outside of application.

In summary, if your application needs a simple and scalable postgres solution, and the traffic pattern can be formulated into single shard data access, then pgranger could potentially help.

Downside:

  1. No data migration support. Will add as a next step.
  2. No distributed transactions. Investigating as a next step. Because sharding information is on application process, a custom version could potentially be built.

Would love any discussion, contribution or help on this!! Can be reached at olddaves at gmail.com

Documentation

Overview

Package pgranger implements a sharding library to be run in the application processes.

Examples:

For initiation:
pgRanger, err := pgranger.NewPGRanger(shardingConfigStr, &dbSecret, dbFactory)
if err != nil {
  log.Fataf("server cannot init PGRanger : %s", err)
}
defer pgRanger.Close()

For pointwise routing to a db instance/shard: At top of request handler.

dbCtrler, err := reqHandler.pgRanger.FindDBControllerByUUID(shardingFieldName, shardingFieldVal, false)
if err != nil {
	return nil, err
}
tx, err := dbCtrler.DB().Begin()
err = tx.Exec(...)
err = tx.Commit()

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func ConvertUUIDsToBytes

func ConvertUUIDsToBytes(shardingFieldUUIDs []uuid.UUID) ([][]byte, error)

ConvertUUIDsToBytes converts uuids to [][]byte

func CreateShardingData

func CreateShardingData(
	shardingConfigStr *string,
	dbSecret *DBSecret,
	dbFactory SqlxDBFactory,
) (*shardingData, error)

CreateShardingData creates the sharding data object.

func GetDbURI

func GetDbURI(dbName string, dbHost string, dbPort int) string

GetDbURI returns the identifier for a DB instance.

func LoadLocalConfigFile

func LoadLocalConfigFile(localConfigFilePath string) (*string, error)

Types

type DBConfig

type DBConfig struct {
	DBName  *string `json:"dbName"`
	Host    *string `json:"host"`
	Port    *int    `json:"port"`
	SSLMode *string `json:"sslMode"`
}

type DBConnConfig

type DBConnConfig struct {
	DBName   string
	Host     string
	Port     int
	Username string
	Password string
	SSLMode  string
}

type DBController

type DBController interface {
	DB() *sqlx.DB
	URI() string
}

type DBControllerConfig

type DBControllerConfig struct {
	DB *DBConfig `json:"db"`
}

type DBControllerImpl

type DBControllerImpl struct {
	// contains filtered or unexported fields
}

DBControllerImpl implements DBController.

func (*DBControllerImpl) DB

func (d *DBControllerImpl) DB() *sqlx.DB

DB returns the *sqlx.DB client.

func (*DBControllerImpl) URI

func (d *DBControllerImpl) URI() string

URI returns the URI of the db client.

type DBRunnable

type DBRunnable interface {
	Run(resChan chan interface{}, errChan chan error)
}

type DBSecret

type DBSecret struct {
	Username string `json:"username"`
	Password string `json:"password"`
}

type FanOutContext

type FanOutContext struct {
	// DBURIToDBRunnable contains mapping from DB URIs to DBRunnables.
	DBURIToDBRunnable map[string]DBRunnable
}

FanOutContext is the context/input for the fan out execution.

type FanOutResult

type FanOutResult struct {
	// DBResults contains result per DB.
	DBResults map[string]interface{}
	// ErrorDetails contains error per DB.
	ErrorDetails map[string]error
}

FanOutResult is the result from the fan out execution.

func Execute

func Execute(fanOutContext *FanOutContext) (*FanOutResult, error)

Execute executes fanout runnables in parallel.

type Namespace

type Namespace struct {
	// contains filtered or unexported fields
}

Namespace provides the ranges and corresponding db connections for a sharding key.

type NamespaceConfig

type NamespaceConfig struct {
	Ranges []*RangeConfig `json:"ranges"`
}

type PGRanger

type PGRanger struct {
	// contains filtered or unexported fields
}

PGRanger manages all sharding information and behaviors around sharding.

func NewPGRanger

func NewPGRanger(
	shardingConfigStr *string,
	dbSecret *DBSecret,
	dbFactory SqlxDBFactory,
) (*PGRanger, error)

NewPGRanger creates the PGRanger.

func (*PGRanger) Close

func (m *PGRanger) Close() error

Close will close the db connections in the struct. If there are multiple errors in closing we keep the last one.

func (*PGRanger) ExecuteFanOut

func (m *PGRanger) ExecuteFanOut(fanOutProcessor PointFanOutProcessor, isReplica bool) (*FanOutResult, error)

ExecuteFanOut is the entry point for fanning out to a bunch of dbs from application logic. fanOutProcessor is the callback provided by the application to be called during the processing. The application will need to implement and pass in this PointFanOutProcessor interface. PointFanOutProcessor is named so because the fan out input are point values. In the future we could have range values as fan out input.

func (*PGRanger) FindAllDBControllers

func (m *PGRanger) FindAllDBControllers(isReplica bool) (map[string]DBController, error)

FindAllDBControllers returns all db controllers. Make a copy so users can not accidentally modify shardingData.dbURIToDBController.

func (*PGRanger) FindDBController

func (m *PGRanger) FindDBController(
	shardingFieldName string, shardingFieldVal []byte, isReplica bool,
) (DBController, error)

FindDBController finds a db controller by []byte.

func (*PGRanger) FindDBControllerByUUID

func (m *PGRanger) FindDBControllerByUUID(
	shardingFieldName string, shardingFieldVal uuid.UUID, isReplica bool,
) (DBController, error)

FindDBControllerByUUID finds a db controller by uuid.

func (*PGRanger) FindDBControllers

func (m *PGRanger) FindDBControllers(
	shardingFieldName string, shardingFieldValues [][]byte, isReplica bool,
) ([]string, map[string][][]byte, map[string]DBController, error)

FindDBControllers finds the db controllers covering the sharding field values.

type PointFanOutProcessor

type PointFanOutProcessor interface {
	// Name of the PointFanOutProcessor instance.
	Name() string
	// ProvideShardingFieldNameAndValues provides the sharding field name and values for PGRanger to look up.
	ProvideShardingFieldNameAndValues() (*string, [][]byte)
	// CreateFanOutContextForAllDBs is a callback that will provide runnables for all db shards. Implement this if your
	// use case needs requests to all db shards.
	CreateFanOutContextForAllDBs(dbURIToDBCtrler map[string]DBController) (*FanOutContext, error)
	// CreateFanOutContext is a callback to provide the runnables for specific db shards with input which are decided
	// by pgranger based on output of FetchShardingFieldNameAndValues.
	CreateFanOutContext(
		dbURIs []string,
		dbURIToShardingFieldVals map[string][][]byte,
		dbURIToDBCtrler map[string]DBController,
	) (*FanOutContext, error)
}

PointFanOutProcessor is to be implemented by users who needs fan out to db instances. It is to be provided as an input to ExecuteFanOut(). There are two key steps:

  1. The instance of implementation class of PointFanOutProcessor should provide the sharding field name and values for the fan out. For this, implement ProvideShardingFieldNameAndValues(). The PGRanger will then find the correct db instances hosting the sharding field values. If fan out to all db instances are needed, return nil from ProvideShardingFieldNameAndValues().
  2. Now that PGRanger has decided the sharding field values and the corresponding db instances, it needs the runnables for "what to do" on each db instance. For this purpose, PGRanger will feed the sharding field values and db instances into the CreateFanOutContext or CreateFanOutContextForAllDBs for db runnable creation.

PGRanger will then run all the runnables returned from step 2 in parallel in ExecuteFanOut.

type Range

type Range struct {
	// contains filtered or unexported fields
}

Range provides begin and end of a range and the db stores. Each range represents a micro-partition that can be moved across shards as one unit. It would make sense to split the sharding key space into a reasonably large amount of ranges.

func (*Range) String

func (r *Range) String() string

type RangeConfig

type RangeConfig struct {
	Begin      *string     `json:"begin"`
	End        *string     `json:"end"`
	PrimaryDB  *DBConfig   `json:"primaryDB"`
	ReplicaDBs []*DBConfig `json:"replicaDBs"`
}

type ShardingConfig

type ShardingConfig struct {
	Version       *int32                      `json:"version"`
	Namespaces    map[string]*NamespaceConfig `json:"namespaces"`
	DBControllers []*DBControllerConfig       `json:"dbControllers"`
}

type SqlxDBFactory

type SqlxDBFactory interface {
	CreateSqlxDB(dbConfig *DBConfig, dbSecret *DBSecret) (*string, *sqlx.DB, error)
}

type SqlxDBFactoryImpl

type SqlxDBFactoryImpl struct{}

func (*SqlxDBFactoryImpl) CreateSqlxDB

func (f *SqlxDBFactoryImpl) CreateSqlxDB(dbConfig *DBConfig, dbSecret *DBSecret) (*string, *sqlx.DB, error)

Directories

Path Synopsis
example
userprofile
In this example, we have a user profile application which has its sharding key user_id spreaded over 4 ranges on 2 host shards.
In this example, we have a user profile application which has its sharding key user_id spreaded over 4 ranges on 2 host shards.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL