README
¶
Key-Value File (kvfile)
Introduction
go-kvfile stores key/value pairs to a file with O(log N) (binary search) lookup.
The values are concatenated together at the beginning of the file, followed by a set of length-suffixed entries containing each key and the offset of the associated value, followed by a list of positions of index entries.
The compress package supports seekable-zstd compressed kvfiles.
CLI
The kvfile CLI can be used to read/write a kvfile on the command line:
NAME:
kvfile - A CLI tool for working with key-value files
COMMANDS:
count Print the number of keys in a k/v file.
keys Print all keys in a k/v file in sorted order.
values Print all key-value pairs in a k/v file.
get Get the value for a specific key.
write Write a new kvfile from JSON input.
GLOBAL OPTIONS:
--binary-keys read and log keys as binary (base58) (default: false)
--binary-values read and log values as binary (base58) (default: true)
--file value, -f value path to the kvfile to read
--compress use kvfile compression (default: false)
Usage
The Reader reads values from the kvfile and can search for specific keys using a binary search on the key index:
// Read keys from the file.
Get(): Looks up the value for the given key.
ReadTo(): Reads the value for the given key to the writer.
Exists(): Checks if the given key exists in the store.
// Iterate over keys in the file.
ScanPrefix(): iterates over key/value pairs with a prefix.
ScanPrefixKeys(): iterates over key/value pairs with a prefix, returning keys only.
// Utilities for reading the file structure.
ReadIndexEntry(): Reads the index entry at the given index.
SearchIndexEntry(): Looks up an index entry for the given key.
GetValuePosition(): Determines the position and length of the value for the key.
Write() writes the given key-value pairs to the file with the writer.
var buf bytes.Buffer
keys := [][]byte{
[]byte("test-1"),
[]byte("test-2"),
[]byte("test-3"),
}
vals := [][]byte{
[]byte("val-1"),
[]byte("val-2"),
[]byte("val-3"),
}
var index int
err := Write(&buf, keys, func(wr io.Writer, key []byte) (uint64, error) {
nw, err := wr.Write(vals[index])
if err != nil {
return 0, err
}
index++
return uint64(nw), nil
})
The Writer can be used to incrementally write keys and values to a file.
var buf bytes.Buffer
keys := [][]byte{
[]byte("test-2"),
[]byte("test-3"),
[]byte("test-1"),
}
vals := [][]byte{
[]byte("val-2"),
[]byte("val-3"),
[]byte("val-1"),
}
wr := NewWriter(&buf)
for i, key := range keys {
if err := wr.WriteValue(key, bytes.NewReader(vals[i])); err != nil {
t.Fatal(err.Error())
}
}
if err := wr.Close(); err != nil {
t.Fatal(err.Error())
}
Support
Please open a GitHub issue with any questions / issues.
... or feel free to reach out on Matrix Chat or Discord.
License
MIT
Documentation
¶
Index ¶
- func Write(writer io.Writer, keys [][]byte, writeValue WriteValueFunc) error
- func WriteIndex(writer io.Writer, index []*IndexEntry, pos uint64) (uint64, error)
- func WriteIterator(writer io.Writer, keyIterator KeyIteratorFunc, writeValueFunc WriteValueFunc) error
- type FileReaderAt
- type IndexEntry
- func (m *IndexEntry) CloneMessageVT() protobuf_go_lite.CloneMessage
- func (m *IndexEntry) CloneVT() *IndexEntry
- func (this *IndexEntry) EqualMessageVT(thatMsg any) bool
- func (this *IndexEntry) EqualVT(that *IndexEntry) bool
- func (x *IndexEntry) GetKey() []byte
- func (x *IndexEntry) GetOffset() uint64
- func (x *IndexEntry) GetSize() uint64
- func (x *IndexEntry) MarshalJSON() ([]byte, error)
- func (x *IndexEntry) MarshalProtoJSON(s *json.MarshalState)
- func (x *IndexEntry) MarshalProtoText() string
- func (m *IndexEntry) MarshalToSizedBufferVT(dAtA []byte) (int, error)
- func (m *IndexEntry) MarshalToVT(dAtA []byte) (int, error)
- func (m *IndexEntry) MarshalVT() (dAtA []byte, err error)
- func (*IndexEntry) ProtoMessage()
- func (x *IndexEntry) Reset()
- func (m *IndexEntry) SizeVT() (n int)
- func (x *IndexEntry) String() string
- func (x *IndexEntry) UnmarshalJSON(b []byte) error
- func (x *IndexEntry) UnmarshalProtoJSON(s *json.UnmarshalState)
- func (m *IndexEntry) UnmarshalVT(dAtA []byte) error
- type KeyIteratorFunc
- type Reader
- func (r *Reader) Exists(key []byte) (bool, error)
- func (r *Reader) Get(key []byte) ([]byte, bool, error)
- func (r *Reader) GetValuePosition(key []byte) (idx, length int64, indexEntry *IndexEntry, indexEntryIdx int, err error)
- func (r *Reader) GetValuePositionWithEntry(indexEntry *IndexEntry, indexEntryIdx int) (idx, length int64, err error)
- func (r *Reader) GetValueSize(key []byte) (int64, error)
- func (r *Reader) GetWithEntry(indexEntry *IndexEntry, indexEntryIdx int) ([]byte, error)
- func (r *Reader) ReadIndexEntry(indexEntryIdx uint64) (*IndexEntry, error)
- func (r *Reader) ReadTo(key []byte, to io.Writer) (int, bool, error)
- func (r *Reader) ScanPrefix(prefix []byte, cb func(key, value []byte) error) error
- func (r *Reader) ScanPrefixEntries(prefix []byte, cb func(indexEntry *IndexEntry, indexEntryIdx int) error) error
- func (r *Reader) ScanPrefixKeys(prefix []byte, cb func(key []byte) error) error
- func (r *Reader) SearchIndexEntryWithKey(key []byte) (*IndexEntry, int, error)
- func (r *Reader) SearchIndexEntryWithPrefix(prefix []byte, last bool) (*IndexEntry, int, error)
- func (r *Reader) Size() uint64
- type ReaderAtSeeker
- type WriteValueFunc
- type Writer
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Write ¶
func Write(writer io.Writer, keys [][]byte, writeValue WriteValueFunc) error
Write writes the given key/value pairs to the store in writer. Serializes and writes the key/value pairs. Note: keys must not contain duplicates or an error will be returned. The values will be stored in the order of the original keys slice. writeValue should write the given value to the writer returning the number of bytes written.
func WriteIndex ¶ added in v0.5.0
WriteIndex sorts and checks the index entries and writes them to a file.
pos is the position the writer is located at in the file. returns the number of bytes written (end pos - pos).
func WriteIterator ¶
func WriteIterator(writer io.Writer, keyIterator KeyIteratorFunc, writeValueFunc WriteValueFunc) error
WriteIterator writes the key/value pairs using the given iterators.
WriteValueFunc writes a value and returns number of bytes written and any error. WriteIteratorFunc is a function that returns key/value pairs to write.
Note: keys must not contain duplicates or an error will be returned.
Types ¶
type FileReaderAt ¶
FileReaderAt is a fs.File that implements ReaderAt.
type IndexEntry ¶
type IndexEntry struct {
// Key is the key of the entry.
Key []byte `protobuf:"bytes,1,opt,name=key,proto3" json:"key,omitempty"`
// Offset is the position of the value in bytes.
Offset uint64 `protobuf:"varint,2,opt,name=offset,proto3" json:"offset,omitempty"`
// Size is the size of the value in bytes.
Size uint64 `protobuf:"varint,3,opt,name=size,proto3" json:"size,omitempty"`
// contains filtered or unexported fields
}
IndexEntry is an entry in the index. The index is sorted by key.
func (*IndexEntry) CloneMessageVT ¶
func (m *IndexEntry) CloneMessageVT() protobuf_go_lite.CloneMessage
func (*IndexEntry) CloneVT ¶
func (m *IndexEntry) CloneVT() *IndexEntry
func (*IndexEntry) EqualMessageVT ¶
func (this *IndexEntry) EqualMessageVT(thatMsg any) bool
func (*IndexEntry) EqualVT ¶
func (this *IndexEntry) EqualVT(that *IndexEntry) bool
func (*IndexEntry) GetKey ¶
func (x *IndexEntry) GetKey() []byte
func (*IndexEntry) GetOffset ¶
func (x *IndexEntry) GetOffset() uint64
func (*IndexEntry) GetSize ¶ added in v0.5.0
func (x *IndexEntry) GetSize() uint64
func (*IndexEntry) MarshalJSON ¶ added in v0.6.0
func (x *IndexEntry) MarshalJSON() ([]byte, error)
MarshalJSON marshals the IndexEntry to JSON.
func (*IndexEntry) MarshalProtoJSON ¶ added in v0.6.0
func (x *IndexEntry) MarshalProtoJSON(s *json.MarshalState)
MarshalProtoJSON marshals the IndexEntry message to JSON.
func (*IndexEntry) MarshalProtoText ¶ added in v0.6.2
func (x *IndexEntry) MarshalProtoText() string
func (*IndexEntry) MarshalToSizedBufferVT ¶
func (m *IndexEntry) MarshalToSizedBufferVT(dAtA []byte) (int, error)
func (*IndexEntry) MarshalToVT ¶
func (m *IndexEntry) MarshalToVT(dAtA []byte) (int, error)
func (*IndexEntry) MarshalVT ¶
func (m *IndexEntry) MarshalVT() (dAtA []byte, err error)
func (*IndexEntry) ProtoMessage ¶
func (*IndexEntry) ProtoMessage()
func (*IndexEntry) Reset ¶
func (x *IndexEntry) Reset()
func (*IndexEntry) SizeVT ¶
func (m *IndexEntry) SizeVT() (n int)
func (*IndexEntry) String ¶
func (x *IndexEntry) String() string
func (*IndexEntry) UnmarshalJSON ¶ added in v0.6.0
func (x *IndexEntry) UnmarshalJSON(b []byte) error
UnmarshalJSON unmarshals the IndexEntry from JSON.
func (*IndexEntry) UnmarshalProtoJSON ¶ added in v0.6.0
func (x *IndexEntry) UnmarshalProtoJSON(s *json.UnmarshalState)
UnmarshalProtoJSON unmarshals the IndexEntry message from JSON.
func (*IndexEntry) UnmarshalVT ¶
func (m *IndexEntry) UnmarshalVT(dAtA []byte) error
type KeyIteratorFunc ¶
WriteIteratorFunc is a function that returns key/value pairs to write. The callback should return one key at a time in the order they should be written to the file. Return nil, nil or nil, io.EOF if no keys remain.
type Reader ¶
type Reader struct {
// contains filtered or unexported fields
}
Reader is a key/value file reader.
func BuildReader ¶
BuildReader constructs a new Reader, reading the number of index entries.
func BuildReaderWithFile ¶
func BuildReaderWithFile(f FileReaderAt) (*Reader, error)
BuildReaderWithFile constructs a new Reader with an fs.File.
func BuildReaderWithSeeker ¶
func BuildReaderWithSeeker(rd ReaderAtSeeker) (*Reader, error)
BuildReaderWithSeeker constructs a new Reader with a io.ReaderSeeker reading the file size.
func (*Reader) GetValuePosition ¶
func (r *Reader) GetValuePosition(key []byte) (idx, length int64, indexEntry *IndexEntry, indexEntryIdx int, err error)
GetValuePosition determines the position and length of the value for the key.
Returns -1, 1, nil, -1, nil if not found.
func (*Reader) GetValuePositionWithEntry ¶
func (r *Reader) GetValuePositionWithEntry(indexEntry *IndexEntry, indexEntryIdx int) (idx, length int64, err error)
GetValuePositionWithEntry determines the position and length of the value with an entry.
Returns -1, 1, nil, -1, nil if not found.
func (*Reader) GetValueSize ¶ added in v0.4.1
GetValueSize looks up the size of the value for the given key without reading the value. Returns -1, nil if not found.
func (*Reader) GetWithEntry ¶
func (r *Reader) GetWithEntry(indexEntry *IndexEntry, indexEntryIdx int) ([]byte, error)
GetWithEntry returns the value for the given index entry.
func (*Reader) ReadIndexEntry ¶
func (r *Reader) ReadIndexEntry(indexEntryIdx uint64) (*IndexEntry, error)
ReadIndexEntry reads the index entry at the given index.
func (*Reader) ReadTo ¶
ReadTo reads the value for the given key to the writer.
Returns number of bytes read, found, and any error. Returns 0, false, nil if not found.
func (*Reader) ScanPrefix ¶
ScanPrefix iterates over key/value pairs with a prefix.
func (*Reader) ScanPrefixEntries ¶
func (r *Reader) ScanPrefixEntries(prefix []byte, cb func(indexEntry *IndexEntry, indexEntryIdx int) error) error
ScanPrefixEntries iterates over entries with the given key prefix.
func (*Reader) ScanPrefixKeys ¶
ScanPrefixKeys iterates over keys with a prefix.
func (*Reader) SearchIndexEntryWithKey ¶
func (r *Reader) SearchIndexEntryWithKey(key []byte) (*IndexEntry, int, error)
SearchIndexEntryWithKey looks up an index entry for the given key.
If not found, returns nil, idx, err and idx is the index where the searched element would appear if inserted into the list.
func (*Reader) SearchIndexEntryWithPrefix ¶
SearchIndexEntryWithPrefix returns the entry of the first key with the prefix.
If last is true returns the last element that matches the prefix.
If the key or prefix is not found, returns nil, idx, err, where idx is the element where an element with the given prefix would appear if inserted.
type ReaderAtSeeker ¶
type ReaderAtSeeker interface {
io.ReaderAt
io.ReadSeeker
}
ReaderAtSeeker is a ReaderAt and a ReadSeeker.
type WriteValueFunc ¶
WriteValueFunc is a function that writes the value for a key to a writer. Return the number of bytes written and any error.
type Writer ¶ added in v0.5.0
type Writer struct {
// contains filtered or unexported fields
}
Writer allows progressively writing values to a kvfile. The index will be written once the writer is closed (flushed). Note: keys must not contain duplicates or an error will be returned. Concurrency safe.
Source Files
Directories