README
¶
go-dqlite 
This repository provides the dqlite Go package, which can be used to
replicate a SQLite database across a cluster, using the Raft
algorithm.
Design higlights
- No external processes needed: dqlite is just a Go library, you link it it to your application exactly like you would with SQLite.
- Replication needs a SQLite patch which is not yet included upstream.
- The Go Raft package from Hashicorp is used internally for replicating the write-ahead log frames of SQLite across all nodes.
How does it compare to rqlite?
The main differences from rqlite are:
- Full support for transactions
- No need for statements to be deterministic (e.g. you can use
time()) - Frame-based replication instead of statement-based replication, this means in dqlite there's more data flowing between nodes, so expect lower performance. Should not really matter for most use cases.
Status
This is beta software for now, but we'll get to rc/release soon.
Demo
To see dqlite in action, make sure you have the following dependencies installed:
- Go (tested on 1.8)
- gcc
- any dependency/header that SQLite needs to build from source
- Python 3
Then run:
go get -d github.com/CanonicalLtd/dqlite
cd $GOPATH/src/github.com/CanonicalLtd/dqlite
make dependencies
./run-demo
This should spawn three dqlite-based nodes, each of one running the code in the demo Go source.
Each node inserts data in a test table and then dies abruptly after a random timeout. Leftover transactions and failover to other nodes should be handled gracefully.
While the demo is running, to get more details about what's going on behind the scenes you can also open another terminal and run a command like:
watch ls -l /tmp/dqlite-demo-*/ /tmp/dqlite-demo-*/snapshots/
and see how the data directories of the three nodes evolve in terms
SQLite databases (test.db), write-ahead log files (test.db-wal),
raft logs store (raft.db), and raft snapshots.
Documentation
The documentation for this package can be found on Godoc.
FAQ
Q: How does dqlite behave during conflict situations? Does Raft select a winning WAL write and any others in flight are aborted?
A: There can't be a conflict situation. Raft's model is that only the leader can append new log entries, which translated to dqlite means that only the leader can write new WAL frames. So this means that any attempt to perform a write transaction on a non-leader node will fail with a sqlite3x.ErrNotLeader error (and in this case clients are supposed to retry against whoever is the new leader).
Q: When not enough nodes are available, are writes hung until consensus?
A: Yes, however there's a (configurable) timeout. This is a consequence of Raft sitting in the CP spectrum of the CAP theorem: in case of a network partition it chooses consistency and sacrifices availability.
Documentation
¶
Overview ¶
Package dqlite implements a database/sql/driver with raft-based SQLite replication.
Index ¶
- Constants
- Variables
- func NewFSM(r *Registry) raft.FSM
- type Conn
- func (c *Conn) Begin() (driver.Tx, error)deprecated
- func (c *Conn) BeginTx(ctx context.Context, opts driver.TxOptions) (driver.Tx, error)
- func (c *Conn) Close() error
- func (c *Conn) Exec(query string, args []driver.Value) (driver.Result, error)
- func (c *Conn) ExecContext(ctx context.Context, query string, args []driver.NamedValue) (driver.Result, error)
- func (c *Conn) Prepare(query string) (driver.Stmt, error)
- func (c *Conn) PrepareContext(ctx context.Context, query string) (driver.Stmt, error)
- func (c *Conn) Query(query string, args []driver.Value) (driver.Rows, error)
- func (c *Conn) QueryContext(ctx context.Context, query string, args []driver.NamedValue) (driver.Rows, error)
- type DatabaseServerStore
- type DialFunc
- type Driver
- type DriverError
- type DriverOption
- func WithConnectionBackoffCap(cap time.Duration) DriverOption
- func WithConnectionBackoffFactor(factor time.Duration) DriverOption
- func WithConnectionTimeout(timeout time.Duration) DriverOption
- func WithContext(context context.Context) DriverOption
- func WithContextTimeout(timeout time.Duration) DriverOption
- func WithDialFunc(dial DialFunc) DriverOption
- func WithLogFunc(log LogFunc) DriverOption
- type InmemServerStore
- type LogFunc
- type LogLevel
- type Registry
- type Result
- type Rows
- type Server
- type ServerInfo
- type ServerOption
- type ServerStore
- type Stmt
- func (s *Stmt) Close() error
- func (s *Stmt) Exec(args []driver.Value) (driver.Result, error)
- func (s *Stmt) ExecContext(ctx context.Context, args []driver.NamedValue) (driver.Result, error)
- func (s *Stmt) NumInput() int
- func (s *Stmt) Query(args []driver.Value) (driver.Rows, error)
- func (s *Stmt) QueryContext(ctx context.Context, args []driver.NamedValue) (driver.Rows, error)
- type Tx
Constants ¶
const ( LogDebug = logging.Debug LogInfo = logging.Info LogWarn = logging.Warn LogError = logging.Error )
Available logging levels.
Variables ¶
var ErrNoAvailableLeader = client.ErrNoAvailableLeader
ErrNoAvailableLeader is returned as root cause of Open() if there's no leader available in the cluster.
var NewInmemServerStore = client.NewInmemServerStore
NewInmemServerStore creates ServerStore which stores its data in-memory.
Functions ¶
func NewFSM ¶
NewFSM creates a new dqlite FSM, suitable to be passed to raft.NewRaft.
It will handle replication of the SQLite write-ahead log.
This is mostly an internal implementation detail of dqlite, but it needs to be exposed since the raft.Raft parameter that NewDriver accepts doesn't allow access to the FSM that it was passed when created with raft.NewRaft().
Types ¶
type Conn ¶
type Conn struct {
// contains filtered or unexported fields
}
Conn implements the sql.Conn interface.
func (*Conn) BeginTx ¶
BeginTx starts and returns a new transaction. If the context is canceled by the user the sql package will call Tx.Rollback before discarding and closing the connection.
This must check opts.Isolation to determine if there is a set isolation level. If the driver does not support a non-default level and one is set or if there is a non-default isolation level that is not supported, an error must be returned.
This must also check opts.ReadOnly to determine if the read-only value is true to either set the read-only transaction property if supported or return an error if it is not supported.
func (*Conn) Close ¶
Close invalidates and potentially stops any current prepared statements and transactions, marking this connection as no longer in use.
Because the sql package maintains a free pool of connections and only calls Close when there's a surplus of idle connections, it shouldn't be necessary for drivers to do their own connection caching.
func (*Conn) ExecContext ¶
func (c *Conn) ExecContext(ctx context.Context, query string, args []driver.NamedValue) (driver.Result, error)
ExecContext is an optional interface that may be implemented by a Conn.
func (*Conn) PrepareContext ¶
PrepareContext returns a prepared statement, bound to this connection. context is for the preparation of the statement, it must not store the context within the statement itself.
type DatabaseServerStore ¶
type DatabaseServerStore struct {
// contains filtered or unexported fields
}
DatabaseServerStore persists a list addresses of dqlite servers in a SQL table.
func DefaultServerStore ¶
func DefaultServerStore(filename string) (*DatabaseServerStore, error)
DefaultServerStore creates a new ServerStore using the given filename to open a SQLite database, with default names for the schema, table and column parameters.
It also creates the table if it doesn't exist yet.
func NewServerStore ¶
func NewServerStore(db *sql.DB, schema, table, column string) *DatabaseServerStore
NewServerStore creates a new ServerStore.
func (*DatabaseServerStore) Get ¶
func (d *DatabaseServerStore) Get(ctx context.Context) ([]ServerInfo, error)
Get the current servers.
func (*DatabaseServerStore) Set ¶
func (d *DatabaseServerStore) Set(ctx context.Context, servers []ServerInfo) error
Set the servers addresses.
type Driver ¶
type Driver struct {
// contains filtered or unexported fields
}
Driver perform queries against a dqlite server.
func NewDriver ¶
func NewDriver(store ServerStore, options ...DriverOption) (*Driver, error)
NewDriver creates a new dqlite driver, which also implements the driver.Driver interface.
func (*Driver) Open ¶
Open establishes a new connection to a SQLite database on the dqlite server.
The given name must be a pure file name without any directory segment, dqlite will connect to a database with that name in its data directory.
Query parameters are always valid except for "mode=memory".
If this node is not the leader, or the leader is unknown an ErrNotLeader error is returned.
func (*Driver) SetContextTimeout ¶ added in v0.2.2
SetContextTimeout sets the default client timeout when no context deadline is provided.
type DriverError ¶
DriverError is returned in case of database errors.
type DriverOption ¶
type DriverOption func(*driverOptions)
DriverOption can be used to tweak driver parameters.
func WithConnectionBackoffCap ¶
func WithConnectionBackoffCap(cap time.Duration) DriverOption
WithConnectionBackoffCap sets the maximum connection retry backoff value, (regardless of the backoff factor) for retrying failed connection attempts.
If not used, the default is 1 second.
func WithConnectionBackoffFactor ¶
func WithConnectionBackoffFactor(factor time.Duration) DriverOption
WithConnectionBackoffFactor sets the exponential backoff factor for retrying failed connection attempts.
If not used, the default is 50 milliseconds.
func WithConnectionTimeout ¶
func WithConnectionTimeout(timeout time.Duration) DriverOption
WithConnectionTimeout sets the connection timeout.
If not used, the default is 5 seconds.
func WithContext ¶
func WithContext(context context.Context) DriverOption
WithContext sets a global cancellation context.
func WithContextTimeout ¶ added in v0.2.2
func WithContextTimeout(timeout time.Duration) DriverOption
WithContextTimeout sets the default client context timeout when no context deadline is provided.
If not used, the default is 5 seconds.
func WithDialFunc ¶
func WithDialFunc(dial DialFunc) DriverOption
WithDialFunc sets a custom dial function.
func WithLogFunc ¶
func WithLogFunc(log LogFunc) DriverOption
WithLogFunc sets a custom logging function.
type InmemServerStore ¶
type InmemServerStore = client.InmemServerStore
InmemServerStore keeps the list of target gRPC SQL servers in memory.
type Registry ¶
type Registry struct {
// contains filtered or unexported fields
}
Registry tracks internal data shared by the dqlite Driver and FSM.
func NewRegistry ¶
NewRegistry creates a new Registry, which is expected to be passed to both NewFSM and NewDriver.
The ID parameter is a string identifying the local node.
func NewRegistryWithLogger ¶
NewRegistryWithLogger returns a registry configured with the given logger.
type Result ¶
type Result struct {
// contains filtered or unexported fields
}
Result is the result of a query execution.
func (*Result) LastInsertId ¶
LastInsertId returns the database's auto-generated ID after, for example, an INSERT into a table with primary key.
func (*Result) RowsAffected ¶
RowsAffected returns the number of rows affected by the query.
type Rows ¶
type Rows struct {
// contains filtered or unexported fields
}
Rows is an iterator over an executed query's results.
func (*Rows) ColumnTypeDatabaseTypeName ¶
ColumnTypeDatabaseTypeName implements RowsColumnTypeDatabaseTypeName.
func (*Rows) ColumnTypeScanType ¶
ColumnTypeScanType implements RowsColumnTypeScanType.
type Server ¶
type Server struct {
// contains filtered or unexported fields
}
Server implements the dqlite network protocol.
func NewServer ¶
func NewServer(raft *raft.Raft, registry *Registry, listener net.Listener, options ...ServerOption) (*Server, error)
NewServer creates a new Server instance.
type ServerInfo ¶
type ServerInfo = client.ServerInfo
ServerInfo holds information about a single server.
type ServerOption ¶
type ServerOption func(*serverOptions)
ServerOption can be used to tweak server parameters.
func WithServerAddressProvider ¶
func WithServerAddressProvider(provider raft.ServerAddressProvider) ServerOption
WithServerAddressProvider sets a custom resolver for server addresses.
func WithServerLogFunc ¶
func WithServerLogFunc(log LogFunc) ServerOption
WithServerLogFunc sets a custom log function for the server.
type ServerStore ¶
type ServerStore = client.ServerStore
ServerStore is used by a dqlite client to get an initial list of candidate dqlite server addresses that it can dial in order to find a leader dqlite server to use.
Once connected, the client periodically updates the addresses in the store by querying the leader server about changes in the cluster (such as servers being added or removed).
type Stmt ¶
type Stmt struct {
// contains filtered or unexported fields
}
Stmt is a prepared statement. It is bound to a Conn and not used by multiple goroutines concurrently.
func (*Stmt) ExecContext ¶
ExecContext executes a query that doesn't return rows, such as an INSERT or UPDATE.
ExecContext must honor the context timeout and return when it is canceled.
func (*Stmt) QueryContext ¶
QueryContext executes a query that may return rows, such as a SELECT.
QueryContext must honor the context timeout and return when it is canceled.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
internal
|
|
|
protocol
Package protocol is a generated protocol buffer package.
|
Package protocol is a generated protocol buffer package. |
|
replication
Package replication implements the core part of dqlite, setting up raft-based replication of the SQLite WAL.
|
Package replication implements the core part of dqlite, setting up raft-based replication of the SQLite WAL. |
|
trace
Package trace implements a tracing system that can handle emitting large amounts of entries with minimal performance overhead.
|
Package trace implements a tracing system that can handle emitting large amounts of entries with minimal performance overhead. |