ddbds

package module
v0.0.0-...-adc9fbc Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 11, 2022 License: MIT Imports: 16 Imported by: 0

README

NOTE: This has been moved to github.com/ipfs/go-ds-dynamodb

This is an implementation of go-datastore that is backed by DynamoDB.

ddbds includes support for optimized prefix queries. When you setup your table's key schema correctly and register it with ddbds, then incoming queries that match the schema will be converted into DynamoDB queries instead of table scans, enabling high performance, ordered, high-cardinality prefix queries.

Note that ddbds currently only stores values up to 400 kb (the DynamoDB maximum item size). This makes ddbds inappropriate for block storage. It could be extended to fall back to S3, but that is not yet implemented. Within the InterPlanetary ecosystem, it's designed for storing DHT records, IPNS records, peerstore records, etc.

Setup

Simple Setup with Unoptimized Queries

ddbds can be used as a simple key-value store, without optimized queries.

In this case, all datastore queries will result in full table scans using the ParallelScan API, and filtering/ordering/etc. will be performed client-side.

This is a good option if your table is small or your data and access patterns would not significantly benefit from optimized queries.

var ddbClient *dynamodb.DynamoDB = ...
tableName := "datastore-table"
ddbDS := ddbds.New(ddbClient, tableName)

By default the expected partition key is DSKey of type string. The name can be customized with the WithPartitionKey() option.

Optimized Queries

To use optimized prefix queries, you must specify a sort key.

Also, elements written into the datastore should have at least 2 parts, such as /a/b and not /a.

ddbds splits the key into partition and sort keys. Examples:

  • /a -> error (not enough parts)
  • /a/b -> [a, b]
  • /a/b/c -> [a, b/c]
  • etc.

To use optimized queries, simply specify the sort key name using the WithSortKey() option:

var ddbClient *dynamodb.DynamoDB = ...
tableName := "datastore-table"
ddbDS := ddbds.New(
	ddbClient, 
	tableName,
	ddbds.WithPartitionKey("PartitionKey"),
	ddbds.WithSortKey("SortKey"),
)

Composing Datastores

This datastore can be composed using mount datastores for optimized prefix queries under different namespaces and DynamoDB tables.

Example:

ddbDS := mount.New([]mount.Mount{
	{
		Prefix: ds.NewKey("/peers/metadata"),
		Datastore: ddbds.New(
			ddbClient,
			"datastore-peers-metadata",
			ddbds.WithPartitionkey("PeerID"),
			ddbds.WithSortKey("MetadataKey"),
		),
	},
	{
		Prefix: ds.NewKey("/peers/addrs"),
		Datastore: ddbds.New(
			ddbClient,
			"datastore-peers-addrs",
			ddbds.WithPartitionkey("PeerID"),
		),
	},
	{
		Prefix: ds.NewKey("/peers/keys"),
		Datastore: ddbds.New(
			ddbClient,
			"datastore-peers-keys",
			ddbds.WithPartitionkey("PeerID"),
			ddbds.WithSortKey("KeyType"),
		),
	},
	{
		Prefix: ds.NewKey("/providers"),
		Datastore: ddbds.New(
			ddbClient,
			"datastore-providers",
			ddbds.WithPartitionkey("ContentHash"),
			ddbds.WithSortKey("PeerID"),
		),
	},
	{
		Prefix: ds.NewKey("/"),
		Datastore: ddbds.New(
			ddbClient,
			"datastore-all",
			ddbds.WithPartitionkey("DSRootKey"),
		),
	},
})

IAM Permissions

The following describes the IAM actions and the datastore methods that use them:

  • dynamodb:GetItem
    • Get()
    • GetExpiration()
    • GetSize()
    • Has()
  • dynamodb:PutItem
    • Put()
    • PutWithTTL()
  • dynamodb:DeleteItem
    • Delete()
  • dynamodb:Scan
    • Scan() (if there is no sort key defined)
  • dynamodb:Query
    • Query() (if there is a sort key defined)
  • dynamodb:DescribeTable
    • DiskUsage()
  • dynamodb:UpdateItem
    • SetTTL()
  • dynamodb:BatchWriteItem
    • Batch.Commit()

Datastore Features

  • Batching
  • TTL
  • Disk Usage
  • Transactions
  • Checked (not applicable)
  • Scrubbed (not applicable)
  • GC (not applicable)

Documentation

Index

Constants

This section is empty.

Variables

View Source
var (
	ErrInvalidKey = errors.New("invalid key for DynamoDB datastore")
)

Functions

func WithPartitionkey

func WithPartitionkey(partitionKey string) func(o *Options)

func WithScanParallelism

func WithScanParallelism(n int) func(o *Options)

func WithSortKey

func WithSortKey(sortKey string) func(o *Options)

func WithStronglyConsistentReads

func WithStronglyConsistentReads() func(o *Options)

Types

type DDBDatastore

type DDBDatastore struct {
	// contains filtered or unexported fields
}

func New

func New(ddbClient *dynamodb.DynamoDB, table string, optFns ...func(o *Options)) *DDBDatastore

func (*DDBDatastore) Batch

func (d *DDBDatastore) Batch(_ context.Context) (datastore.Batch, error)

func (*DDBDatastore) Close

func (d *DDBDatastore) Close() error

func (*DDBDatastore) Delete

func (d *DDBDatastore) Delete(ctx context.Context, key ds.Key) error

func (*DDBDatastore) DiskUsage

func (d *DDBDatastore) DiskUsage(ctx context.Context) (uint64, error)

DiskUsage returns the size of the DynamoDB table. Note that DynamoDB only updates this size once every few hours. The underlying call is heavily throttled so this should only be called occasionally.

func (*DDBDatastore) EntryCount

func (d *DDBDatastore) EntryCount(ctx context.Context) (uint64, error)

func (*DDBDatastore) Get

func (d *DDBDatastore) Get(ctx context.Context, key ds.Key) ([]byte, error)

func (*DDBDatastore) GetExpiration

func (d *DDBDatastore) GetExpiration(ctx context.Context, key ds.Key) (time.Time, error)

func (*DDBDatastore) GetSize

func (d *DDBDatastore) GetSize(ctx context.Context, key ds.Key) (size int, err error)

func (*DDBDatastore) Has

func (d *DDBDatastore) Has(ctx context.Context, key ds.Key) (bool, error)

func (*DDBDatastore) Put

func (d *DDBDatastore) Put(ctx context.Context, key ds.Key, value []byte) error

func (*DDBDatastore) PutWithTTL

func (d *DDBDatastore) PutWithTTL(ctx context.Context, key ds.Key, value []byte, ttl time.Duration) error

func (*DDBDatastore) Query

func (d *DDBDatastore) Query(ctx context.Context, q query.Query) (query.Results, error)

func (*DDBDatastore) SetTTL

func (d *DDBDatastore) SetTTL(ctx context.Context, key ds.Key, ttl time.Duration) error

func (*DDBDatastore) Sync

func (d *DDBDatastore) Sync(ctx context.Context, prefix ds.Key) error

type Options

type Options struct {
	UseStronglyConsistentReads bool
	ScanParallelism            int
	PartitionKey               string
	SortKey                    string
	// contains filtered or unexported fields
}

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL