blobs

package module
v0.13.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 13, 2023 License: MIT Imports: 14 Imported by: 0

README

fdb-blobs

PkgGoDev CI Go Report Card

A blob store for FoundationDB that handles arbitrary size blobs by splitting the data into chunks.

API documentation

Tests

Start all the services:

docker compose up -d

Run all tests:

docker compose exec blobs go test

Run a specific test:

docker compose exec blobs go test -run TestCreate

Sending arguments to go test:

docker compose exec blobs go test -cover

Running fuzz tests:

docker compose exec blobs go test -fuzz FuzzChunkSizes

MIT License

Copyright (c) 2023 Sune Simonsen mailto:sune@we-knowhow.dk

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the 'Software'), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Documentation

Index

Examples

Constants

This section is empty.

Variables

View Source 
var BlobNotFoundError = errors.New("blob not found")

Error for when a blob can't be found.

Functions

This section is empty.

Types

type Blob added in v0.7.0 

type Blob struct {
	// contains filtered or unexported fields
}

The blob type.

func (*Blob) CreatedAt added in v0.7.0 

func (blob *Blob) CreatedAt() (time.Time, error)

Returns the time the blob was created at.

Example 
blob := createTestBlob()

createdAt, err := blob.CreatedAt()
if err != nil {
	log.Fatal("Could not get created at time")
}

fmt.Printf("Blob was created at: %s", createdAt)

Output:

Blob was created at: 2023-01-01 00:00:00 +0000 UTC

func (*Blob) Id added in v0.7.0 

func (blob *Blob) Id() Id

Returns the id of the blob.

Example 
blob := createTestBlob()

id := blob.Id()

fmt.Printf("Blob id: %s", id)

Output:

func (*Blob) Len added in v0.7.0 

func (blob *Blob) Len() (uint64, error)

Returns the length of the content of the blob.

Example 
blob := createTestBlob()

len, err := blob.Len()
if err != nil {
	log.Fatal("Could not get blob content length")
}

fmt.Printf("Blob content length: %d", len)

Output:

Blob content length: 15

func (*Blob) Reader added in v0.7.0 

func (blob *Blob) Reader() io.Reader

Returns a new reader for the content of the blob.

New chunks are fetched on demand based on the chunk size and number of chunks per transaction configured for the store.

Example 
blob := createTestBlob()

r := blob.Reader()

lr := io.LimitReader(r, 10)

content, err := io.ReadAll(lr)
if err != nil {
	log.Fatal("Could read content of limited reader")
}

fmt.Printf("Start of blob content: %s", content)

Output:

Start of blob content: My blob co

type Id added in v0.2.0 

type Id string

The type of blob ids.

func (Id) FDBKey added in v0.2.0 

func (id Id) FDBKey() fdb.Key

Converts the id into a FoundationDB compatible key.

type IdGenerator added in v0.12.0 

type IdGenerator interface {
	NextId() Id
}

Interface for blob id generators.

type Option 

type Option func(store *Store) error

Store option type.

func WithChunkSize 

func WithChunkSize(chunkSize int) Option

Sets the chunk size used to store blobs.

Defaults to 10000 bytes.

Notice it is always possible to read blobs no matter what chunk size they were stored with, as that information is saved with the blob.

The chunk size needs to be greater than zero and honour the limits of FoundationDB key value sizes.

Example 
db := fdbConnect()

store, err := NewStore(db, testNamespace(), WithChunkSize(256))
if err != nil {
	log.Fatalln("Could not create store")
}

blob, err := store.Create(strings.NewReader("Blob content"))
if err != nil {
	log.Fatal("Could not create blob")
}

content, err := io.ReadAll(blob.Reader())
if err != nil {
	log.Fatal("Could not read blob content")
}

fmt.Printf("Blob content: %s", content)

Output:

Blob content: Blob content

func WithChunksPerTransaction 

func WithChunksPerTransaction(chunksPerTransaction int) Option

Set the number of chunks commited per transaction.

Defaults to 100 chunks per transaction.

The chunks per transaction needs to honour the FoundationDB transction size.

Example 
db := fdbConnect()

store, err := NewStore(db, testNamespace(), WithChunksPerTransaction(10))
if err != nil {
	log.Fatalln("Could not create store")
}

blob, err := store.Create(strings.NewReader("Blob content"))
if err != nil {
	log.Fatal("Could not create blob")
}

content, err := io.ReadAll(blob.Reader())
if err != nil {
	log.Fatal("Could not read blob content")
}

fmt.Printf("Blob content: %s", content)

Output:

Blob content: Blob content

func WithIdGenerator added in v0.12.0 

func WithIdGenerator(idGenerator IdGenerator) Option

Set the id generator used to create new ids.

Defaults to the blobs.UlidIdGenerator.

Notice that ids generated by the generator needs to be globally unique across the entire system.

Example 
db := fdbConnect()

idGenerator := &TestIdgenerator{}
store, err := NewStore(db, testNamespace(), WithIdGenerator(idGenerator))
if err != nil {
	log.Fatalln("Could not create store")
}

for i := 0; i < 3; i++ {
	blob, err := store.Create(strings.NewReader("Blob content"))
	if err != nil {
		log.Fatal("Could not create blob")
	}
	fmt.Println(blob.Id())
}

Output:

blob:0
blob:1
blob:2

func WithSystemTime added in v0.8.0 

func WithSystemTime(systemTime SystemTime) Option

Provide a system time instance, to override how timestamps are calculated.

This is useful for custom timestamp calculation and for mocking.

Example 
db := fdbConnect()

now, _ := time.Parse(time.RFC3339, "2023-01-01T00:00:00Z")
st := &SystemTimeMock{Time: now}

store, err := NewStore(db, testNamespace(), WithSystemTime(st))
if err != nil {
	log.Fatalln("Could not create store")
}

blob, err := store.Create(strings.NewReader("Blob content"))
if err != nil {
	log.Fatal("Could not create blob")
}

createdAt, err := blob.CreatedAt()
if err != nil {
	log.Fatal("Could not get created at time")
}

fmt.Printf("Blob was created at: %s", createdAt)

Output:

Blob was created at: 2023-01-01 00:00:00 +0000 UTC

type Store added in v0.10.0 

type Store struct {
	// contains filtered or unexported fields
}

The store type.

func NewStore added in v0.10.0 

func NewStore(db fdb.Database, ns string, opts ...Option) (*Store, error)

NewStore constructs a new blob store with the given FoundationDB instance, a namespace ns the blobs are stored under and a list of options.

Example 
db := fdbConnect()

store, err := NewStore(db, "blob-store-namespace")
if err != nil {
	log.Fatalln("Could not create store")
}

blob, err := store.Create(strings.NewReader("Blob content"))
if err != nil {
	log.Fatal("Could create blob")
}

content, err := io.ReadAll(blob.Reader())
if err != nil {
	log.Fatal("Could not read blob content")
}

fmt.Printf("Blob content: %s", content)

Output:

Blob content: Blob content

func (*Store) Blob added in v0.10.0 

func (store *Store) Blob(id Id) (*Blob, error)

Returns a blob instance for the given id.

Example 
store := createTestStore(WithIdGenerator(&TestIdgenerator{}))

r := strings.NewReader("My blob content")
_, err := store.Create(r)
if err != nil {
	log.Fatal("Could not create blob")
}

blob, err := store.Blob("blob:0")
if err != nil {
	log.Fatal("Could not retrieve blob")
}

content, err := io.ReadAll(blob.Reader())
if err != nil {
	log.Fatal("Could not read blob content")
}

fmt.Printf("Blob content: %s", content)

Output:

Blob content: My blob content

func (*Store) CommitUpload added in v0.10.0 

func (store *Store) CommitUpload(tr fdb.Transaction, token UploadToken) (Id, error)

Commits an upload with the given token on a transaction. This creates a blob from the upload and returns its id.

Example 
db := fdbConnect()
store, err := NewStore(db, testNamespace())

if err != nil {
	log.Fatalln("Could not create store")
}

r := strings.NewReader("My blob content")
token, err := store.Upload(r)
if err != nil {
	log.Fatal("Could not upload blob")
}

id, err := transact(db, func(tr fdb.Transaction) (Id, error) {
	return store.CommitUpload(tr, token)
})
if err != nil {
	log.Fatal("Could not commit upload")
}

blob, err := store.Blob(id)
if err != nil {
	log.Fatal("Could not retrieve blob")
}

content, err := io.ReadAll(blob.Reader())
if err != nil {
	log.Fatal("Could not read blob content")
}

fmt.Printf("Blob content: %s", content)

Output:

Blob content: My blob content

func (*Store) Create added in v0.10.0 

func (store *Store) Create(r io.Reader) (*Blob, error)

Creates and returns a new blob with the content of the given reader r.

Example 
store := createTestStore()

r := strings.NewReader("My blob content")
blob, err := store.Create(r)
if err != nil {
	log.Fatal("Could not create blob")
}

content, err := io.ReadAll(blob.Reader())
if err != nil {
	log.Fatal("Could not read blob content")
}

fmt.Printf("Blob content: %s", content)

Output:

Blob content: My blob content

func (*Store) DeleteRemovedBlobsBefore added in v0.10.0 

func (store *Store) DeleteRemovedBlobsBefore(date time.Time) ([]Id, error)

Deletes blobs that was marked as removed before a given date.

This is useful to make a periodical cleaning job.

Example 
db := fdbConnect()

idGenerator := &TestIdgenerator{}
store, err := NewStore(db, testNamespace(), WithIdGenerator(idGenerator))
if err != nil {
	log.Fatalln("Could not create store")
}

for i := 0; i < 3; i++ {
	blob, err := store.Create(strings.NewReader("Blob content"))
	if err != nil {
		log.Fatal("Could not create blob")
	}

	err = store.RemoveBlob(blob.Id())
	if err != nil {
		log.Fatal("Could not remove blob")
	}
}

blobIds, err := store.DeleteRemovedBlobsBefore(time.Now())
if err != nil {
	log.Fatal("Could not delete removed blobs")
}

for _, id := range blobIds {
	fmt.Println(id)
}

Output:

blob:0
blob:1
blob:2

func (*Store) DeleteUploadsStartedBefore added in v0.10.0 

func (store *Store) DeleteUploadsStartedBefore(date time.Time) ([]Id, error)

Deletes uploads that was started before a given time.

This is useful to make a periodical cleaning job.

Example 
db := fdbConnect()

idGenerator := &TestIdgenerator{}
store, err := NewStore(db, testNamespace(), WithIdGenerator(idGenerator))
if err != nil {
	log.Fatalln("Could not create store")
}

for i := 0; i < 3; i++ {
	// Upload without committing the upload.
	_, err := store.Upload(strings.NewReader("Blob content"))
	if err != nil {
		log.Fatal("Could not create blob")
	}
}

// Deleted uploads that was started but not committed.
blobIds, err := store.DeleteUploadsStartedBefore(time.Now())
if err != nil {
	log.Fatal("Could not delete removed blobs")
}

for _, id := range blobIds {
	fmt.Println(id)
}

Output:

blob:0
blob:1
blob:2

func (*Store) RemoveBlob added in v0.10.0 

func (store *Store) RemoveBlob(id Id) error

Marks the blob with the given id as removed.

After a blob is removed it can't be retrieved anymore, but any active readers can still access the removed blob. The removed blobs can be fully deleted using the Store.DeleteRemovedBlobsBefore method.

Example 
store := createTestStore()

r := strings.NewReader("My blob content")
createdBlob, err := store.Create(r)
if err != nil {
	log.Fatal("Could not create blob")
}

err = store.RemoveBlob(createdBlob.Id())
if err != nil {
	log.Fatal("Could not remove blob")
}

_, err = store.Blob(createdBlob.Id())

if errors.Is(err, BlobNotFoundError) {
	fmt.Println("Blob not found")
}

Output:

Blob not found

func (*Store) Upload added in v0.10.0 

func (store *Store) Upload(r io.Reader) (UploadToken, error)

Uploads the content of the given reader r into a temporary location and returns a token for commiting the upload on a transaction later.

Example 
db := fdbConnect()
store, err := NewStore(db, testNamespace())

if err != nil {
	log.Fatalln("Could not create store")
}

r := strings.NewReader("My blob content")
token, err := store.Upload(r)
if err != nil {
	log.Fatal("Could not upload blob")
}

id, err := transact(db, func(tr fdb.Transaction) (Id, error) {
	return store.CommitUpload(tr, token)
})
if err != nil {
	log.Fatal("Could not commit upload")
}

blob, err := store.Blob(id)
if err != nil {
	log.Fatal("Could not retrieve blob")
}

content, err := io.ReadAll(blob.Reader())
if err != nil {
	log.Fatal("Could not read blob content")
}

fmt.Printf("Blob content: %s", content)

Output:

Blob content: My blob content

type SystemTime added in v0.8.0 

type SystemTime interface {
	Now() time.Time
}

Interface to specify the time of the system time right now.

type SystemTimeMock added in v0.12.0 

type SystemTimeMock struct {
	Time time.Time
}

System time mock, useful for mocking the system time in tests.

func (*SystemTimeMock) Now added in v0.12.0 

func (c *SystemTimeMock) Now() time.Time

Returns the mocked time.

type TestIdgenerator added in v0.12.0 

type TestIdgenerator struct {
	// contains filtered or unexported fields
}

Id generator that returns incremental blob ids.

This is only useful for test scenarios.

func (*TestIdgenerator) NextId added in v0.12.0 

func (idGen *TestIdgenerator) NextId() Id

Returns ids of the following form: 

blob:0
blob:1
blob:2
...

type UlidIdGenerator added in v0.12.0 

type UlidIdGenerator struct{}

Id generator that returns ULID ids.

func (UlidIdGenerator) NextId added in v0.12.0 

func (_ UlidIdGenerator) NextId() Id

Returns a new ULID id every time it is called.

type UploadToken added in v0.4.0 

type UploadToken struct {
	// contains filtered or unexported fields
}

Upload token returned when uploading and used when commiting an upload.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.