README
¶
dqlite 
This repository provides the dqlite Go package, which can be used to
replicate a SQLite database across a cluster, using the Raft
algorithm.
Design higlights
- No external processes needed: dqlite is just a Go library, you link it it to your application exactly like you would with SQLite.
- Replication needs a SQLite patch which is not yet included upstream.
- The Go Raft package from Hashicorp is used internally for replicating the write-ahead log frames of SQLite across all nodes.
How does it compare to rqlite?
The main differences from rqlite are:
- Full support for transactions
- No need for statements to be deterministic (e.g. you can use
time()) - Frame-based replication instead of statement-based replication, this means in dqlite there's more data flowing between nodes, so expect lower performance. Should not really matter for most use cases.
Status
This is beta software for now, but we'll get to rc/release soon.
Demo
To see dqlite in action, make sure you have the following dependencies installed:
- Go (tested on 1.8)
- gcc
- any dependency/header that SQLite needs to build from source
- Python 3
Then run:
go get -d github.com/CanonicalLtd/dqlite
cd $GOPATH/src/github.com/CanonicalLtd/dqlite
make dependencies
./run-demo
This should spawn three dqlite-based nodes, each of one running the code in the demo Go source.
Each node inserts data in a test table and then dies abruptly after a random timeout. Leftover transactions and failover to other nodes should be handled gracefully.
While the demo is running, to get more details about what's going on behind the scenes you can also open another terminal and run a command like:
watch ls -l /tmp/dqlite-demo-*/ /tmp/dqlite-demo-*/snapshots/
and see how the data directories of the three nodes evolve in terms
SQLite databases (test.db), write-ahead log files (test.db-wal),
raft logs store (raft.db), and raft snapshots.
Documentation
The documentation for this package can be found on Godoc.
FAQ
Q: How does dqlite behave during conflict situations? Does Raft select a winning WAL write and any others in flight are aborted?
A: There can't be a conflict situation. Raft's model is that only the leader can append new log entries, which translated to dqlite means that only the leader can write new WAL frames. So this means that any attempt to perform a write transaction on a non-leader node will fail with a sqlite3x.ErrNotLeader error (and in this case clients are supposed to retry against whoever is the new leader).
Q: When not enough nodes are available, are writes hung until consensus?
A: Yes, however there's a (configurable) timeout. This is a consequence of Raft sitting in the CP spectrum of the CAP theorem: in case of a network partition it chooses consistency and sacrifices availability.
Documentation
¶
Overview ¶
Package dqlite implements a database/sql/driver with raft-based SQLite replication.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func NewFSM ¶
NewFSM creates a new dqlite FSM, suitable to be passed to raft.NewRaft.
It will handle replication of the SQLite write-ahead log.
This is mostly an internal implementation detail of dqlite, but it needs to be exposed since the raft.Raft parameter that NewDriver accepts doesn't allow access to the FSM that it was passed when created with raft.NewRaft().
Types ¶
type Conn ¶
type Conn struct {
// contains filtered or unexported fields
}
Conn implements the sql.Conn interface.
func (*Conn) Close ¶
Close invalidates and potentially stops any current prepared statements and transactions, marking this connection as no longer in use.
Because the sql package maintains a free pool of connections and only calls Close when there's a surplus of idle connections, it shouldn't be necessary for drivers to do their own connection caching.
type Driver ¶
type Driver struct {
// contains filtered or unexported fields
}
Driver manages a node partecipating to a dqlite replicated cluster.
func NewDriver ¶
NewDriver creates a new node of a dqlite cluster, which also implements the driver.Driver interface.
The Registry instance must be the same one that was passed to NewFSM to build the raft.FSM which in turn got passed to raft.NewRaft for creating the raft instance.
func (*Driver) Open ¶
Open starts a new connection to a SQLite database.
The given name must be a pure file name without any directory segment, dqlite will connect to a database with that name in its data directory.
Query parameters are always valid except for "mode=memory".
If this node is not the leader, or the leader is unknown an ErrNotLeader error is returned.
type DriverConfig ¶
type DriverConfig struct {
Logger *log.Logger // Logger to use to emit messages.
BarrierTimeout time.Duration // Maximum amount of time to wait for the FSM to catch up with logs.
ApplyTimeout time.Duration // Maximum amount of time to wait for a raft FSM command to be applied.
CheckpointThreshold uint64 // Minimum number of WAL frames before performing a checkpoint.
}
DriverConfig holds configuration options for a dqlite SQL Driver.
type Registry ¶
Registry tracks internal data shared by the dqlite Driver and FSM.
func NewRegistry ¶
NewRegistry creates a new Registry, which is expected to be passed to both NewFSM and NewDriver.
The dir parameter is the directory where dqlite will store the underlying SQLite database files.
type Stmt ¶
type Stmt struct {
// contains filtered or unexported fields
}
Stmt is a prepared statement. It is bound to a Conn and not used by multiple goroutines concurrently.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
internal
|
|
|
protocol
Package protocol is a generated protocol buffer package.
|
Package protocol is a generated protocol buffer package. |
|
replication
Package replication implements the core part of dqlite, setting up raft-based replication of the SQLite WAL.
|
Package replication implements the core part of dqlite, setting up raft-based replication of the SQLite WAL. |
|
trace
Package trace implements a tracing system that can handle emitting large amounts of entries with minimal performance overhead.
|
Package trace implements a tracing system that can handle emitting large amounts of entries with minimal performance overhead. |
|
recover
|
|