store

package
v0.0.0-...-b093a1f Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: May 14, 2016 License: BSD-2-Clause Imports: 11 Imported by: 0

Documentation

Overview

Store is a package for managing persistent local server data, including data blobs, metadata, user data, and server configerations.

Large files are divided into blocks for internal server deduplication, this is invisble to the outside. Lower block size gives better deduplication, and higher block size gives faster sequential access and less metadata. (use 4MB for now)

Store is designed to store files with the following possibilities:

Importing a local file: data is read, hashed, and saved a block at a time. The hash and length is reported at the end.

Requested to be saved locally: hash and length are given (along with locations to the server). Inner hash are downloaded to see what blocks are needed, then the blocks are downloaded.

As such hashes maybe stored bottom up or top down, and when a tree is traversed, it may find missing nodes. An error condition that the server must handle or report to client.

Index

Constants

View Source
const BlobSize = 4 << 20 //4MByte blocks
View Source
const (
	TTLLongAgo = TTL(-1)
)

Variables

This section is empty.

Functions

This section is empty.

Types

type Blob

type Blob interface {
	bitset.Blob
}

staticly import Blob interface from bitset

func NewBlobFromFile

func NewBlobFromFile(file *os.File, size int64) Blob

staticly import NewBlobFromFile function from bitset

type Database

type Database interface {
	MetaStore
	//GetState checks if we have a file or not, or in progress
	GetState(key HLKey) FileState
	//GetAt reads len(b) bytes of file key from offset off.
	//returns error if b can't be read completely
	GetAt(key HLKey, b []byte, off int64) error
	//PutAt writes b to file from offset off.
	//
	//has returns the number of leaf nodes completed. This can report the progress.
	//Iff has is full, complete is true
	//
	//If hash checking fails, an err is reported. The server can us this information
	//to demote the source.
	PutAt(key HLKey, b []byte, off int64) (has ht.Nodes, complete bool, err error)

	//Import a file from reader
	ImportFromReader(r io.Reader) HLKey
}

Database is the full interface for all the data of a server, WIP

type FileState

type FileState int
const (
	//FileNone means we don't have this file
	FileNone FileState = iota
	//FilePart means we have parts of this file
	FilePart
	//FileComplete means we have all of this file
	FileComplete
	//FileExpired means the ttl is outdated, gc maybe removing this file
	FileExpired
)

type HLKey

type HLKey interface {
	//Hash returns the hash of refereced. Do not modify the returned contests
	GetHash() []byte
	//Length returns the length of refereced file in bytes
	GetLength() int64
	//FullBytes is used when a key need to include length, as an attacker might
	//claim the existence of a file of the same hash but different size.
	//Do not modify the returned contests
	FullBytes() []byte
}

HLKey is the hash and length of some data. Used to look up data in a high level database. Even though length is unnecessary for uniqueness, it is an usefull meta to keep around without significant overhead, for code path conditions, sanity checking, and debuging.

Before a file can be fully checked, the length can not be conformed, it is to possible to have two downloads in progress, with the same hash but different length, where at least one can never be competed. So for all network communications, the length is nessary to go with the hash.

The length is always included in very link and network request with the hash, or for child nodes inferred from the size of parent and it's position, so we can always know the length when we know the hash, without using any additional lookup.

func NewHLKey

func NewHLKey(hash []byte, length int64) HLKey

NewHLKey create a new HLKey, the hash is deep copied

type KV

type KV interface {
	//Get gets the value for the given key. It returns nil if KV does not contain the key.
	//
	//Warning: Get might not see what was Set or Deleted until after sync().
	//This is a temperary work around to easly warp some apis.
	Get(key []byte) []byte
	//Set sets the value for the given key. It overwrites any previous value for that key.
	//Do not set with len(v) == 0, use Delete instead.
	Set(key []byte, v []byte)
	//Delete deletes the value for the given key, it is a no-op if KV does not contain the key.
	Delete(key []byte)
	//Sync commits all the changes to stable storage. It blocks until done.
	Sync()
	//Close closes KV.
	Close() error
	GC(startAfterKey []byte, f func(key []byte, v []byte) (delete bool, stop bool))
}

KV is an interface for a []byte based key value store. For storing small pieces of metadata that are often updated.

func OpenLeveldb

func OpenLeveldb(path string) (KV, error)

type LV

type LV interface {
	//New creates a new block in LV, or nil if the key already exist.
	//size > 0
	New(key []byte, size int64) Blob
	//Get returns the Block referenced by key, or nil if key is not found.
	Get(key []byte, size int64) Blob
	//Move changes the key from from to to, and resizes to newSize.
	//If from does not exist, or to already exist, then an error is reported.
	//The Blob keyed by from should not be open.
	Move(oldKey []byte, oldSize int64, newKey []byte, newSize int64) error

	//Delete removes the Block repersented by key from LV.
	//Returns true if blob is not in LV after Delete.
	Delete(key []byte, size int64) (bool, error)

	Close() error
}

LV (Large Values) is an interface for a Block based data store. For storing large values with length known in advance and as part of key. The same key with two sizes represent two different Blobs.

func OpenFolderLV

func OpenFolderLV(root string) LV

type MetaStore

type MetaStore interface {
	//InnerHashMinLevel reports the amount of inner hash saved, everything at or
	//above the Level can be retrieved
	InnerHashMinLevel() ht.Level
	//GetInnerHashes reads inner hashes at level and offset, len(hs) should be a
	//multiple of hash length (32).
	//
	//An error is reported if parts of hs is unknown, parts known is still filled
	//with unknown parts as all zeros.
	//
	//If parts of hs are impossible by index range, it panics
	GetInnerHashes(key HLKey, hs []byte, level ht.Level, off ht.Nodes) error

	PutInnerHashes(key HLKey, hs []byte, level ht.Level, off ht.Nodes) (has ht.Nodes, complete bool, err error)

	//TTLGet gets the TTL of a file
	//TTLGet does not imply having a file. Only that the file is desired to be keeped.
	TTLGet(key HLKey) TTL

	//TTLSetAtleast updates the TTL to be at least coved to until, the total
	//increase is multiplied by key.Length() to return costs in storage time by
	//byteMonth.
	//Savings in deduplication can be reflected in byteMonth.
	//
	//The time increased form freeFrom is unaccounted, put in now to not charge before now.
	//byteMonth = 0 if freeFrom >= until
	TTLSetAtleast(key HLKey, freeFrom, until TTL) (byteMonth int64)

	//Close closes MetaStore
	Close() error
}

MetaStore is a store for hash meta data on top of KV. Such as: Child hashes, data length/hash level, and TTL.

func OpenMetaStore

func OpenMetaStore(path string) (MetaStore, error)

type TTL

type TTL int16 //January, 2000 is 0.

A TTL encodes the number of periods from a base time. For ease of local usage, it is always per month, from January, 2000.

For more information, see fensan/docs

func TTLFromBytes

func TTLFromBytes(b []byte) TTL

func TTLNow

func TTLNow() TTL

Return a TTL for the current period (month), cached. This is a TTL of 0.

func (TTL) Bytes

func (t TTL) Bytes() []byte

Bytes returns the 2 bytes of the TTL for serialization

func (TTL) End

func (t TTL) End() time.Time

End returns the time at the end of TTL, just before the 1st of next month.

func (TTL) MonthUntil

func (t TTL) MonthUntil(to TTL) int16

MonthUntil is the number of months until to, ie: 3.MonthUntil(5) = 2

func (TTL) YearMonth

func (t TTL) YearMonth() (year int, month time.Month)

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL