dump

package module
v0.0.0-...-e5b324a Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 22, 2018 License: MIT Imports: 12 Imported by: 0

README

Dump

Go Report Card Build Status

Dump is a simple append-key-value store that uses the filesystem as an index.

It has some interesting features that may or may not work for you:

  1. All operations are either an append or a read. There is no delete or overwrite operation.
  2. There is an existence operation Exists() that will tell you if a key may exist or definitely does not.
  3. At the current time, there may only be one appender and/or one reader at a time per key. This may change in the future.
  4. The value of each key is stored in a separate file. If you have a lot of keys, you will need to increase the max number of open files in your system. There is an eviction policy for file handles that may solve this problem for you.
  5. Values are not compressed until the TimeToCompaction is reached. At that point, the value becomes immutable. The meaning of TimeToCompaction is that the file has not been accessed (for reading or writing) for this duration. This may change to the last modified time.

Example

i := New(&IndexOpts{})

i.Append("key", []byte("hello world"))
fmt.Println(i.Exists("key")) // true
i.Read("key", func(r io.Reader) {
				b, _ := ioutil.ReadAll(r)
				fmt.Println(string(b)) // hello world
})

Documentation

Index

Constants

View Source
const (
	CompressedExt = ".gz"
	NormalExt     = ".dat"
)

Variables

View Source
var (
	ErrImmutableSegment = errors.New("the segment has been compressed and is now immutable")
)

Functions

This section is empty.

Types

type Index

type Index interface {
	Append(string, []byte) error
	Read(string, func(io.Reader)) error
	Exists(string) bool
	Stats() IndexStats
	Compact() error
}

Index is an append-only key-value

func New

func New(opts *IndexOpts) Index

NewIndex creates a new index from the given options

type IndexOpts

type IndexOpts struct {
	Dir                    string
	IndexCleanupInterval   time.Duration
	CompactionInterval     time.Duration
	MaxSegmentIdlePeriod   time.Duration
	TimeToCompaction       time.Duration
	RetentionPeriod        time.Duration
	RetentionCheckInterval time.Duration
	Logger                 *log.Logger
	BloomfilterSize        float64
	BloomError             float64
	// contains filtered or unexported fields
}

IndexOpts contains options for creating the index. Defaults will be set so it is perfectly fine to pass an empty struct.

type IndexStats

type IndexStats struct {
	OpenSegments int64 `json:"open_segments"`
}

type Segment

type Segment interface {
	Append([]byte) error
	Read(func(io.Reader)) error
	Compress() error
	Close()
}

Segment is essentially an append-only file. Not safe for concurrent use by multiple goroutines.

func NewSegment

func NewSegment(path string) (Segment, error)

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL