README
¶
flockd 0.3.1
flockd provides a simple file system-based key/value database that uses file locking for concurrency safety. Keys correspond to files, values to their contents, and tables to directories. Files are share-locked on read (Get and ForEach) and exclusive-locked on write (Set, Create, Update, and Delete).
This may be overkill if you have only one application using a set of files in a directory. But if you need to sync files between multiple systems, like a distributed database, assuming your sync software respects file system locks, flockd might be a great way to go. This is especially true for modestly-sized databases and databases with a single primary instance and multiple read-only secondary instances.
In any event, your file system must support proper file locking for this to work. If your file system does not, it might still work if file renaming and unlinking is atomic and flockd is used exclusively to access files. If not, then all bets are off, and you can expect occasional bad reads.
All of this may turn out to be a bad idea. YMMV. Warranty not included.
Inspirations
-
diskv: Similar use of one file per key/value pair. Uses a sync.RWMutex for concurrency protection. Lots of features, including path transformation, caching, and compression.
-
fskv: Use of buckets similar to Tables here (I stole the idea, really). Relies on temporary files for locking. Depends on afero for file management.
-
Scribble: Uses a single JSON file for the database. Relies on sync.Mutex for concurrency control.
Documentation
¶
Overview ¶
Package flockd provides a simple file system-based key/value database that uses file locking for concurrency safety. Keys correspond to files, values to their contents, and tables to directories. Files are share-locked on read (Get and ForEach) and exclusive-locked on write (Set, Create, Update, and Delete).
This may be overkill if you have only one application using a set of files in a directory. But if you need to sync files between multiple systems, like a distributed database, assuming your sync software respects file system locks, flockd might be a great way to go. This is especially true for modestly-sized databases and databases with a single primary instance and multiple read-only secondary instances.
In any event, your file system must support proper file locking for this to work. If your file system does not, it might still work if file renaming and unlinking is atomic and flockd is used exclusively to access files. If not, then all bets are off, and you can expect occasional bad reads.
All of this may turn out to be a bad idea. YMMV. Warranty not included.
Example ¶
package main
import (
"fmt"
"log"
"os"
"time"
"github.com/iovation/flockd"
)
var timeout = 10 * time.Millisecond
func main() {
// Set up a new database.
db, err := flockd.New("flockd.db", timeout)
if err != nil {
log.Fatal("flockd.New", err)
}
defer os.RemoveAll("flockd.db")
// Create a table.
tbl, err := db.Table("foo")
if err != nil {
log.Fatal("flockd.Table", err)
}
// Add a key/value pair to the table.
key := "greeting"
if err := tbl.Create(key, []byte("Hello world!")); err != nil {
log.Fatal("flockd.Create", err)
}
// Fetch the value.
val, err := tbl.Get(key)
if err != nil {
log.Fatal("flockd.Get", err)
}
fmt.Println(string(val))
// Set the value.
if err := tbl.Set(key, []byte("Goodbye world!")); err != nil {
log.Fatal("flockd.Set", err)
}
// Fetch the new value.
val, err = tbl.Get(key)
if err != nil {
log.Fatal("flockd.Get", err)
}
fmt.Println(string(val))
// Delete the value.
if err := tbl.Delete(key); err != nil {
log.Fatal("flockd.Delete", err)
}
// No more value.
val, err = tbl.Get(key)
if err != os.ErrNotExist {
log.Fatal("flockd.Get", err)
}
}
Output: Hello world! Goodbye world!
Index ¶
- type DB
- func (db *DB) Create(key string, val []byte) error
- func (db *DB) Delete(key string) error
- func (db *DB) ForEach(feFunc ForEachFunc) error
- func (db *DB) Get(key string) ([]byte, error)
- func (db *DB) Path() string
- func (db *DB) Set(key string, val []byte) error
- func (db *DB) Table(name string) (*Table, error)
- func (db *DB) Tables() ([]*Table, error)
- func (db *DB) Update(key string, val []byte) error
- type ForEachFunc
- type Table
- func (table *Table) Create(key string, value []byte) error
- func (table *Table) Delete(key string) error
- func (table *Table) ForEach(feFunc ForEachFunc) error
- func (table *Table) Get(key string) ([]byte, error)
- func (table *Table) Name() string
- func (table *Table) Set(key string, value []byte) error
- func (table *Table) Update(key string, value []byte) error
Examples ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type DB ¶
type DB struct {
// contains filtered or unexported fields
}
DB defines a file system directory as the root for a simple key/value database.
func New ¶
New creates a new key/value database, with the specified directory as the root table. If the directory does not exist, it will be created. The timeout sets the maximum time flockd will wait for a file lock when attempting to read, write, or delete a file, in nanoseconds. Returns an error if the directory creation fails or if the timeout is less than or equal to zero.
Example ¶
package main
import (
"log"
"os"
"time"
"github.com/iovation/flockd"
)
func main() {
// Open the database with an access timeout of 10 ms.
db, err := flockd.New("my.db", 10*time.Millisecond)
if err != nil {
log.Fatal("flockd.New", err)
}
// Optionally remove the database.
defer os.RemoveAll(db.Path())
}
Output:
func (*DB) Create ¶
Create creates the key/value pair by writing it to a file named for the key, plus the extension ".kv", in the root directory, but only if the file does not already exist.
func (*DB) Delete ¶
Delete deletes the key and its value by deleting the file named for the key, plus the extension ".kv", in the root directory.
func (*DB) ForEach ¶
func (db *DB) ForEach(feFunc ForEachFunc) error
ForEach finds each file with the extension ".kv" in the root directory and calls the specified function, passing the file's key and value (file basename and contents).
Example ¶
package main
import (
"fmt"
"io/ioutil"
"log"
"os"
"time"
"github.com/iovation/flockd"
)
func tmpdir(name string) string {
dir, err := ioutil.TempDir("", name)
if err != nil {
log.Fatal("TempDir", err)
}
return dir
}
func main() {
// Open the database
path := tmpdir("foreach.db")
db, err := flockd.New(path, time.Millisecond)
if err != nil {
log.Fatal("flockd.New", err)
}
defer os.RemoveAll(path)
// Load some data into the root table.
for k, v := range map[string]string{
"thing 1": "thing 2",
"wilma": "fred",
"michelle": "barrack",
} {
if err := db.Set(k, []byte(v)); err != nil {
log.Fatal("flockd.Set", err)
}
}
// Iterate over the records in the root table.
db.ForEach(func(key string, val []byte) error {
fmt.Printf("%v: %s\n", key, val)
return nil
})
}
Output: thing 1: thing 2 wilma: fred michelle: barrack
func (*DB) Get ¶
Get returns the value for the key by reading the file named for the key, plus the extension ".kv", from the root directory.
func (*DB) Set ¶
Set sets the value for the key by writing it to the file named for the key, plus the extension ".kv", in the root directory.
func (*DB) Table ¶
Table creates a table in the database. The table corresponds to a subdirectory of the database root directory. Its name will be the table name plus the extension ".tbl". Keys and values can be written directly to the table. Pass a path created by filepath.Join to create a deeper subdirectory. If the directory does not exist, it will be created. Returns an error if the directory creation fails. If the table has been created previously for the instance of the database, it will be returned immediately without checking for the existence of the directory on the file system.
type ForEachFunc ¶
ForEachFunc is the type of the function called for each record fetched by ForEach. The arguments consist of the key and value to process. Returning an error halts the execution of ForEach.
type Table ¶
type Table struct {
// contains filtered or unexported fields
}
Table represents a diretory into which keys and values can be written.
func (*Table) Create ¶
Create creates the key/value pair by writing it to the file named for key, plus the extension ".kv", in the table directory, but only if the file does not already exist. The key must not contain a path separator character; if it does, os.ErrInvalid will be returned. Returns os.ErrExist if the file already exists.
To create the file, Create first opens it with the key name, but only if it doesn't already exist. It then tries to acquire an exclusive lock on the file. It tries only once, and doesn't wait for a lock, so that if any other process first got a lock, the file would be considered to already exist.
Create then creates a temporary file in the table directory and tries to acquire an exclusive lock. If the temporary file already has exclusive lock, Create will wait up to the timeout set for the database to acquire the lock before returning a context.DeadlineExceeded error. Once it has the lock, it writes the value to the temporary file, then moves the temporary file to the new file.
func (*Table) Delete ¶
Delete deletes the key and its value by deleting the file named for key, plus the extension ".kv", from the table directory. The key must not contain a path separator character; if it does, os.ErrInvalid will be returned. Before deleting the file, Delete tries to acquire an exclusive lock. If the file already has exclusive lock, Delete will wait up to the timeout set for the database to acquire the lock before returning a context.DeadlineExceeded error. Once it has acquired the lock, it deletes the file.
func (*Table) ForEach ¶
func (table *Table) ForEach(feFunc ForEachFunc) error
ForEach executes a function for each key/value pair in the table. Internally, ForEach reads the table directory to find record files, fetches its contents via Get(), and passes the key and retrieved value to feFunc. An error returned by any of these steps, including from the feFunc function, causes ForEach to halt the search and return the error. The feFunc function must not modify the table; doing so results in undefined behavior.
Example ¶
package main
import (
"fmt"
"io/ioutil"
"log"
"os"
"time"
"github.com/iovation/flockd"
)
func tmpdir(name string) string {
dir, err := ioutil.TempDir("", name)
if err != nil {
log.Fatal("TempDir", err)
}
return dir
}
func main() {
// Open the database
path := tmpdir("foreach.db")
db, err := flockd.New(path, time.Millisecond)
if err != nil {
log.Fatal("flockd.New", err)
}
defer os.RemoveAll(path)
// Create a table.
tbl, err := db.Table("simpsons.db")
if err != nil {
log.Fatal("flockd.DB.Table", err)
}
// Add some data.
for k, v := range map[string]string{
"Marge": "Homer",
"Selma": "Sideshow Bob",
"Manjula": "Apu",
"Maude": "Ned",
} {
if err := tbl.Set(k, []byte(v)); err != nil {
log.Fatal("flockd.Table.Set", err)
}
}
// Iterate over the records in the root table.
tbl.ForEach(func(key string, val []byte) error {
fmt.Printf("%v ❤️ %s\n", key, val)
return nil
})
}
Output: Marge ❤️ Homer Maude ❤️ Ned Manjula ❤️ Apu Selma ❤️ Sideshow Bob
func (*Table) Get ¶
Get returns the value for the key by reading the file named for key, plus the extension ".kv", from the table directory. The key must not contain a path separator character; if it does, os.ErrInvalid will be returned. If the file does not exist, os.ErrNotExist will be returned. For concurrency safety, Get acquires a shared file system lock on the file before reading its contents. If the file has an exclusive lock on it, Get will wait up to the timeout set for the database for the shared lock before returning a context.DeadlineExceeded error.
func (*Table) Name ¶
Name returns the name of the table, which corresponds to the name of the subdirectory without the extension ".tbl".
func (*Table) Set ¶
Set sets the value for the key by writing it to the file named for key, plus the extension ".kv", in the table directory. The key must not contain a path separator character; if it does, os.ErrInvalid will be returned.
To set the value, Set first creates a temporary file in the table directory and tries to acquire an exclusive lock. If the temporary file already has exclusive lock, Set will wait up to the timeout set for the database to acquire the lock before returning a context.DeadlineExceeded error. Once it has the lock, it writes the value to the temporary file.
Next, it tries to acquire an exclusive lock on the file with the key name, again waiting up to the database timeout before returning a context.DeadlineExceeded error. Once it has the lock, it moves the temporary file to the new file.
func (*Table) Update ¶
Update updates the value for the key by writing it to an existing file named for key, plus the extension ".kv", in the table directory. The key must not contain a path separator character; if it does, os.ErrInvalid will be returned. If the file does not already exist, os.ErrNotExist will be returned.
To update the file, Update first opens the file with the key name for writing. If the file does not exist, os.ErrNotExist will be returned.
Next, Update creates a temporary file in the table directory and tries to acquire an exclusive lock. If the temporary file already has exclusive lock, Update will wait up to the timeout set for the database to acquire the lock before returning a context.DeadlineExceeded error. Once it has the lock, it writes the value to the temporary file.
Next, it tries to acquire an exclusive lock on the opened file, again waiting up to the database timeout before returning a context.DeadlineExceeded error. Once it has the lock, it moves the temporary file to the new file.
Source Files