README
¶
zstorage
A ZK-based object storage library
Overview
This library facilitates the storage of data in Zookeeper.
At a high level, the Store supports the following API:
Put(item Item) (Ident, error)
Get(ident Ident) (item Item, err error)
List(category string) (locations []Location, err error)
Variants(location Location) (variants []string, err error)
Delete(ident Ident) error
Close() error
Nomenclature
An Item fully represents an item in the datastore. Clients will provide an Item when Putting data and will receive an Item when Getting data.
An Item is fully identified by a composed Ident. The Ident points to a Location and also an optional Variant and an optional Version.
Variants
An item may have any number of variants. Note that this is different from the Version which is described below. If a client Puts an item with a variant, it will live as a child of the current node for that item.
Additionally, when deleting an item, the user may specify a variant. If no variant is specified when deleting an item, that item and all of its variant will be deleted.
Paths
Here is an example of how a typical path might look like in the system:
/[basePath]/[category]/buckets/[bucket]/[name]/[variant]
The basePath can be configured on the Store, and will be prepended to any znode path that is generated for an operation.
The category, name, and variant parts are set on the item's Ident property.
The bucket is generated by the Store using a configurable hashing function. It is derived by hashing the item name. The number of buckets to which a name might possibly hash can be configured when constructing the store.
Optimistic Locking
Part of an item's Ident is a Version field called Version.
When performing a Get, the Store will set the Version on the returned Item. This allows the client to specify that version when performing a subsequent mutating operation.
If this is set on an item's Ident property when performing a mutating operation, it will ensure that the item that is updated is specifically that version when setting it. If another client happened to set the item before the first client was able to do so, the library will return the ErrVersionConflict error. At this point, the client may choose to do another read and try again.
If the Item.Ident.Version is set to NoPriorVersion when passing an Item to Put() it is assumed that this Put() must create the item and it will return ErrVersionConflict if the node already exists. If no Version is specified, Put will create the node if it doesn't already exist or ignore and overwrite the existing Item with the new one if it does.
Documentation
¶
Index ¶
- Constants
- Variables
- func LocationsByName(locations []Location) (interface{}, func(_, _ int) bool)
- func ValidateCategory(name string) error
- func ValidateNamed(name string, required bool) error
- type ConnectionOpts
- type Connector
- type HashProviderFunc
- type IStore
- type Ident
- type Item
- type Location
- type Store
- func (s *Store) Close() error
- func (s *Store) Delete(ident Ident) (err error)
- func (s *Store) Get(ident Ident) (item Item, err error)
- func (s *Store) List(category string) (locations []Location, err error)
- func (s *Store) Put(item Item) (Ident, error)
- func (s *Store) Variants(location Location) (variants []string, err error)
- type StoreOpt
- type Version
Constants ¶
const ( // ErrIllegalOption is returned when a StoreOpt configuration is set w/ an illegal value. ErrIllegalOption = internalError("illegal option configuration") // ErrVersionConflict is returned when a specified ZKVersion is rejected by // ZK when performing a mutating operation on a znode. Clients that receive // this can retry by re-reading the Item and then trying again. ErrVersionConflict = internalError("zk version conflict") // ErrNotFound is returned when an attempting to read a znode that does not exist. ErrNotFound = internalError("znode not found") )
const ( // DefaultNumHashBuckets is the number of buckets that will be used to // spread out items living within a category by placing children of the // category into numerically named buckets. Clients may choose to override // this by specifying OptNumHashBuckets when building the Store. DefaultNumHashBuckets = 256 // DefaultBucketsZnodeName is the default name of parent znode that will // store the buckets for a particular category. This is necessary to // allow categories like "foo" and "foo/bar", since we enforce that // category names cannot end in this name. Clients may choose to override // this default by specifying OptBucketsZnodeName when building the // Store. DefaultBucketsZnodeName = "buckets" )
const MaxDataSize = 1024 * 1024
MaxDataSize represents the size of the largest data blob that a caller can store.
const NoPriorVersion = -1
NoPriorVersion tells Put that we're expecting to create a new znode for a particular item, not to update an existing item. If znode already exists, then ErrVersionConflict may be returned.
Variables ¶
var DefaultHashProviderFunc = HashProvider(md5.New)
DefaultHashProviderFunc is the default provider of hash functions unless overriden with OptHashProviderFunc when building the Store.
var ( // DefaultZKACL is the default ACL set that will be used when creating // nodes in ZK unless overridden with OptACL when building the Store. DefaultZKACL = zk.WorldACL(zk.PermAll) )
Functions ¶
func LocationsByName ¶
LocationsByName returns a sort function helper that may be passed to sort.Slice in order to sort a slice of Location structs.
func ValidateCategory ¶
ValidateCategory checks that a category is required, and can look like a path or not.
func ValidateNamed ¶
ValidateNamed validates items that have a "name". Like an actual Name or perhaps a Version. Since some names can be blank, we use the 'required' parameter to signify whether or not a name can be blank, and then after that we check against the regexp.
Types ¶
type ConnectionOpts ¶
type ConnectionOpts struct {
// ConnectTimeout is the timeout to make the initial connection to ZK.
ConnectTimeout time.Duration
// InitialSessionTimeout is how long to wait for a valid session to
// be established once the connection happens.
InitialSessionTimeout time.Duration
// Auth represents authentication details. If left alone, no auth will
// be performed
Auth struct {
Schema string
Secret []byte
}
}
ConnectionOpts are used when creating a new Zk connection
type Connector ¶
type Connector interface {
// Connect returns a ZK connection
Connect() (*zk.Conn, error)
// Close should ensure the ZK connection is closed.
Close() error
}
Connector specifies a way to connect to ZK
func ExistingConnection ¶
ExistingConnection returns an existing connection.
The existing connection should have already established a session before calling this method
func NewConnection ¶
func NewConnection(addrs []string, opts ConnectionOpts) Connector
NewConnection returns a Connector that creates a new ZK connection
type HashProviderFunc ¶
HashProviderFunc is a factory for hashers.
func HashProvider ¶
func HashProvider(f func() hash.Hash) HashProviderFunc
HashProvider adapts a golang stdlib hasher into a hash provider func for use by this package.
type IStore ¶
type IStore interface {
Put(item Item) (Ident, error)
Get(ident Ident) (item Item, err error)
List(category string) (locations []Location, err error)
Variants(location Location) (variants []string, err error)
Delete(ident Ident) error
Close() error
}
IStore is the interface to which Store confirms. Documentation for these methods lives on the concrete Store type, as the NewStore method returns a concrete type, not this interface. Clients may choose to use the IStore interface if they wish.
type Ident ¶
type Ident struct {
// Location points to where an item lives in the store.
Location
// Variant, if specified, specifies a named version of the data.
Variant string
// Version specifies the ZK version of the data. This will be
// used to prevent accidental overwrites. If Version is nil,
// then the version will not be considered for operations.
Version Version
}
Ident specifies the location of a stored item
type Item ¶
type Item struct {
// Ident identifies an Item in the ZK backend.
Ident
// Data represents the bytes to be stored within the znode.
Data []byte
}
Item represents the data of a particular item in the store
type Location ¶
type Location struct {
// Category is a base path for a particular category of data. For example,
// this might be something like "widgets" or "widgets/2017" etc. This
// field is required.
Category string
// Name is the name of the stored data. This field is required.
Name string
}
Location points to a particular item in the store
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store exposes an API for performing CRUD operations against a backing ZK cluster.
func (*Store) Delete ¶
Delete deletes the identified item. An error is NOT returned in the case where the item does not already exist in the store.
func (*Store) Get ¶
Get fetches the data for a particuar item. If a particluar version is desired, it must be set on the ident. Returns ErrNotFound if no such item exists.
func (*Store) List ¶
List lists all of the known, latest version Locations that exist under the specified category. Returns ErrNotFound if the category cannot be found within the store.
func (*Store) Put ¶
Put stores the specified item. If successful, an Ident will be returned that reflects the updated metadata for that item (specifically, the Version)
If a client attempts to set an Item with a Variant, and the current item (with no Variant) does not exist yet, the current item will be created as well, with the same data as the specified Item.
Returns ErrVersionConflict if there is a Version mismatch between the item given and the version of the data currently stored. This check is not performed if there is no Version set for the given item.
type StoreOpt ¶
StoreOpt allows a Store to be configured. Returns ErrIllegalOption if the option configuration cannot be applied to the store.
func OptACL ¶
OptACL configures the store to use a particular ACL when creating nodes. A nil or empty ACL list does not alter the store configuration.
func OptBasePath ¶
OptBasePath specifies a root path that will be prepended to all paths written to or read from. An empty path will not change the store configuration. The specified path must begin with "/" and be 'clean' (see path.Clean) otherwise ErrIllegalOption is returned.
func OptBucketsZnodeName ¶
OptBucketsZnodeName allows the client to configure the znode name that will contain the numerically-named bucket nodes. Returns ErrIllegalOption when the specifeid znode name is invalid.
func OptHashProviderFunc ¶
func OptHashProviderFunc(hashProviderFunc HashProviderFunc) StoreOpt
OptHashProviderFunc allows the client to configure which hasher to use to map item names to buckets. A nil hash func does not alter the store configuration.
func OptNumHashBuckets ¶
OptNumHashBuckets specifies the number of hash buckets that will be created under a store path for each content type when data is being written or read.
If this value is changed after data is written, previously written data may not be able to be found later. If the bucket count is zero then the store configuration is not altered. If the bucket count is negative then ErrIllegalOption is returned.
type Version ¶
type Version struct {
// contains filtered or unexported fields
}
Version represents an optional ZK version number.
func NewVersion ¶
NewVersion returns a ZK version w/ the given value. Future calls to Version will return true.
Source Files