filestore

type FileStore ¶

type FileStore struct {
	// contains filtered or unexported fields
}

FileStore implements ObjectStore and stores objects in files on disk. It uses one file per bucket named <bucketId>.dat and in each file it stores data with the following format: <objId> <obj len in bytes> <obj data>\n In order to retrieve data faster, FileStore holds some metadata about buckets and objects in memory. In particular, it retains each object offset in its bucket file and its size in bytes.

To allow concurrent access to multiple buckets, it is used an array of `numMutexes` mutexes. Instead of having a mutex per bucket, these mutexes are in a fixed number to avoid: 1. to have too many mutexes which can use a lot of memory and decrease performances 2. to have maximum 2*numMutexes open files at the same time and avoid errors related to too many open files

Every change to a bucket file is performed in three steps 1. copy old data to a temporary file writing new data if needed (like add or replace and object) 2. move the temp file and overwrite the actual bucket file 3. update bucket's objects metadata if needed This mitigates the possibility of data corruption or mismatch between data on file and metadata in memory Because of this Store and deletion times depend on the size of the bucket (the whole bucket is copied). The retrieving time is bucket size independent because of the metadata stored in maps in memory.

func NewStore(storePath string) (*FileStore, error)

func (f *FileStore) Delete(objId, bucketId string) (bool, error)

func (f *FileStore) Retrieve(objId, bucketId string) ([]byte, bool, error)

func (f *FileStore) Store(obj []byte, objId, bucketId string) (bool, error)