README
¶
go-generic-object-store
Introduction
go-generic-object-store is a small, fast, light-weight, in-memory, off-heap (off of the Go heap) library designed specifically for use with string interning and other similar redundancy reduction concepts.
Design
Object Store
At the top level of the library is the Object Store. You can have multiple instances of different ObjectStores. Each ObjectStore has a pool of slabs and a lookup table. The following diagram should help illuminate these concepts.
Slab Pools
slabPools is a map[uint8]*slabPool. The map index indicates the size (in bytes) of the objects stored in a particular pool. When attempting to add a new object if there are no available slabs in a pool a new one will be created. When a slab is completely empty it will be deleted.
Fragmentation is a concern if objects are frequently added and deleted.
Lookup Table
lookupTable is a []SlabAddr. SlabAddr is a uintptr which stores the memory address of a slab. The lookupTable is sorted in descending order to speed up searches.
Slab
slab is a struct which contains a single field: objSize uint8. All of the data used by slabs is MMapped memory which is ignored by the Go GC. We don't actually hold references to any slab structs. When we need to access the data contained in a slab we allocate an empty []byte and point its Data field to a memory address in the store, or adjust the memory address by known offsets and convert the underlying data into a different type.
- The 1st byte in a
slabis the object size of all stored objects inside theslab(uint8). - The 2nd through 9th (or 5th if running on 32-bit architecture) bytes in a
slabis the number of objects stored inside theslab(uint). - The next part of the
[]byteholds the slice header and data from the[]uint64ofbitset.BitSet.set. - Finally, the rest of the space in a
slabis dedicated storage for objects. The required space is calculated by multiplying object size by objects per slab.
Notes
- The object store is not safe for concurrent operations. You need to implement necessary locking/unlocking at the next higher level.
- It has not been extensively tested on 32-bit architecture.
Limitations
- 255 maximum bytes per object stored in a slab
See Also
- We use https://github.com/willf/bitset for keeping track of free object slots in slabs.
- For an example of the object store in action check out https://github.com/robert-milan/go-object-interning
Documentation
¶
Index ¶
- Variables
- func NewSlabPool(objSize uint8) *slabPool
- type FragStat
- type MemStat
- type ObjAddr
- type ObjectStore
- func (o *ObjectStore) Add(obj []byte) (ObjAddr, error)
- func (o *ObjectStore) Delete(obj ObjAddr) error
- func (o *ObjectStore) FragStatsByObjSize(size uint8) (float32, error)
- func (o *ObjectStore) FragStatsPerPool() (fragStats []FragStat)
- func (o *ObjectStore) FragStatsTotal() (float32, error)
- func (o *ObjectStore) Get(obj ObjAddr) ([]byte, error)
- func (o *ObjectStore) MemStatsByObjSize(size uint8) (uint64, error)
- func (o *ObjectStore) MemStatsPerPool() (memStats []MemStat)
- func (o *ObjectStore) MemStatsTotal() (uint64, error)
- func (o *ObjectStore) Search(searching []byte) (ObjAddr, bool)
- type ObjectStoreConfig
- type SlabAddr
Constants ¶
This section is empty.
Variables ¶
var Config = NewConfig()
Config provides an ObjectStoreConfig with default settings.
Functions ¶
func NewSlabPool ¶
func NewSlabPool(objSize uint8) *slabPool
NewSlabPool initializes a new slab pool and returns a pointer to it
Types ¶
type ObjAddr ¶
type ObjAddr = uintptr
ObjAddr is a uintptr used for storing the addresses of objects in slabs
type ObjectStore ¶
type ObjectStore struct {
// contains filtered or unexported fields
}
ObjectStore contains a map of slabPools indexed by the size of the objects stored in each pool It also contains a lookup table which is a slice of SlabAddr lookupTable is kept sorted in descending order and updated whenever a slab is created or deleted
func NewObjectStore ¶
func NewObjectStore(c ObjectStoreConfig) ObjectStore
NewObjectStore initializes a new object store with the given configuration Once an object store has been initialized its configuration cannot be changed
func (*ObjectStore) Add ¶
func (o *ObjectStore) Add(obj []byte) (ObjAddr, error)
Add takes an object and adds it to the slab pool of the correct size On success it returns the memory address of the added object as an ObjAddr On failure it returns an error as the second value
func (*ObjectStore) Delete ¶
func (o *ObjectStore) Delete(obj ObjAddr) error
Delete deletes an object by object address On success it returns nil, otherwise it returns an error message
func (*ObjectStore) FragStatsByObjSize ¶
func (o *ObjectStore) FragStatsByObjSize(size uint8) (float32, error)
FragStatsByObjSize returns the fragmentation percent of the requested pool as specified by size
func (*ObjectStore) FragStatsPerPool ¶
func (o *ObjectStore) FragStatsPerPool() (fragStats []FragStat)
FragStatsPerPool returns a slice containing a FragStat for each non-empty slab pool
func (*ObjectStore) FragStatsTotal ¶
func (o *ObjectStore) FragStatsTotal() (float32, error)
FragStatsTotal returns the total fragmentation percent across the object store
func (*ObjectStore) Get ¶
func (o *ObjectStore) Get(obj ObjAddr) ([]byte, error)
Get retrieves a value by object address On success it returns a byte slice of appropriate length, containing the requested object data On failure the second returned value is the error
func (*ObjectStore) MemStatsByObjSize ¶
func (o *ObjectStore) MemStatsByObjSize(size uint8) (uint64, error)
MemStatsByObjSize returns the size of a slab pool in bytes. It only looks at MMapped memory
func (*ObjectStore) MemStatsPerPool ¶
func (o *ObjectStore) MemStatsPerPool() (memStats []MemStat)
MemStatsPerPool returns a slice containing a MemStat for each non-empty slab pool
func (*ObjectStore) MemStatsTotal ¶
func (o *ObjectStore) MemStatsTotal() (uint64, error)
MemStatsTotal returns the estimated total MMapped memory used across the object store
type ObjectStoreConfig ¶
type ObjectStoreConfig struct {
BaseObjectsPerSlab uint8
GrowthFactor float64 // for use with math.Pow this is easier
}
ObjectStoreConfig is used by the object store when creating a new instance. Please see the documentation at https://github.com/replay/go-generic-object-store for more information
func NewConfig ¶
func NewConfig() ObjectStoreConfig
NewConfig returns a new object store configuration with default settings. Please see the documentation at https://github.com/replay/go-generic-object-store for more information.
Source Files