blobserver

package
v0.0.0-...-6608488 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 31, 2013 License: Apache-2.0 Imports: 11 Imported by: 0

Documentation

Overview

Package blobserver defines how raw blobs are stored and accessed.

Index

Constants

View Source
const MaxBlobSize = 16 << 20

MaxBlobSize is the size of a single blob in Camlistore.

TODO: formalize this in the specs. This value of 16 MB is less than App Engine's 32 MB request limit, much more than Venti's limit, and much more than the ~64 KB & 256 KB chunks that the FileWriter make

Variables

View Source
var ErrCorruptBlob = errors.New("corrupt blob; digest doesn't match")
View Source
var ErrHandlerTypeNotFound = errors.New("requested handler type not loaded")

Functions

func CreateHandler

func CreateHandler(typ string, loader Loader, config jsonconfig.Obj) (http.Handler, error)

CreateHandler instantiates an http Handler of type 'typ' from the provided JSON configuration, and finding peer handlers and configuration from the environment in 'loader'.

The handler 'typ' must have bee previously registered with RegisterHandlerConstructor.

func EnumerateAll

func EnumerateAll(src BlobEnumerator, fn func(blob.SizedRef) error) error

EnumerateAll runs fn for each blob in src. If fn returns an error, iteration stops and fn isn't called again. EnumerateAll will not return concurrently with fn.

func MergedEnumerate

func MergedEnumerate(dest chan<- blob.SizedRef, sources []Storage, after string, limit int) error

TODO: it'd be nice to make sources be []BlobEnumerator, but that makes callers more complex since assignable interfaces' slice forms aren't assignable.

func Receive

func Receive(dst BlobReceiver, br blob.Ref, src io.Reader) (blob.SizedRef, error)

Receive wraps calling a BlobReceiver's ReceiveBlob method, additionally providing verification of the src digest, and also notifying the blob hub on success.

func ReceiveNoHash

func ReceiveNoHash(dst BlobReceiver, br blob.Ref, src io.Reader) (blob.SizedRef, error)

func RegisterHandlerConstructor

func RegisterHandlerConstructor(typ string, ctor HandlerConstructor)

RegisterHandlerConstructor registers an http Handler constructor function for a given handler type.

It is an error to register the same handler type twice.

func RegisterStorageConstructor

func RegisterStorageConstructor(typ string, ctor StorageConstructor)

func StatBlob

func StatBlob(bs BlobStatter, br blob.Ref) (sb blob.SizedRef, err error)

func WaitForBlob

func WaitForBlob(storage interface{}, deadline time.Time, blobs []blob.Ref)

WaitForBlob waits until deadline for blobs to arrive. If blobs is empty, any blobs are waited on. Otherwise, those specific blobs are waited on. When WaitForBlob returns, nothing may have happened.

Types

type BlobEnumerator

type BlobEnumerator interface {
	// EnumerateBobs sends at most limit SizedBlobRef into dest,
	// sorted, as long as they are lexigraphically greater than
	// after (if provided).
	// limit will be supplied and sanity checked by caller.
	// EnumerateBlobs must close the channel.  (even if limit
	// was hit and more blobs remain)
	EnumerateBlobs(dest chan<- blob.SizedRef,
		after string,
		limit int) error
}

type BlobHub

type BlobHub interface {
	// For new blobs to notify
	NotifyBlobReceived(blob blob.Ref)

	RegisterListener(ch chan<- blob.Ref)
	UnregisterListener(ch chan<- blob.Ref)

	RegisterBlobListener(blob blob.Ref, ch chan<- blob.Ref)
	UnregisterBlobListener(blob blob.Ref, ch chan<- blob.Ref)
}

func GetHub

func GetHub(storage interface{}) BlobHub

GetHub return a BlobHub for the give storage implementation.

type BlobReceiveConfiger

type BlobReceiveConfiger interface {
	BlobReceiver
	Configer
}

type BlobReceiver

type BlobReceiver interface {
	// ReceiveBlob accepts a newly uploaded blob and writes it to
	// disk.
	ReceiveBlob(br blob.Ref, source io.Reader) (blob.SizedRef, error)
}

type BlobRemover

type BlobRemover interface {
	// RemoveBlobs removes 0 or more blobs.  Removal of
	// non-existent items isn't an error.  Returns failure if any
	// items existed but failed to be deleted.
	RemoveBlobs(blobs []blob.Ref) error
}

type BlobStatter

type BlobStatter interface {
	// Stat checks for the existence of blobs, writing their sizes
	// (if found back to the dest channel), and returning an error
	// or nil.  Stat() should NOT close the channel.
	// TODO(bradfitz): redefine this to close the channel? Or document
	// better what the synchronization rules are.
	StatBlobs(dest chan<- blob.SizedRef, blobs []blob.Ref) error
}

type Cache

type Cache interface {
	blob.SeekFetcher
	BlobReceiver
	BlobStatter
}

Cache is the minimal interface expected of a blob cache.

type Config

type Config struct {
	Writable, Readable bool
	IsQueue            bool // supports deletes
	CanLongPoll        bool

	// the "http://host:port" and optional path (but without trailing slash) to have "/camli/*" appended
	URLBase       string
	HandlerFinder FindHandlerByTyper
}

type Configer

type Configer interface {
	Config() *Config
}

type FindHandlerByTyper

type FindHandlerByTyper interface {
	// FindHandlerByType finds a handler by its handlerType and
	// returns its prefix and handler if it's loaded.  If it's not
	// loaded, the error will be ErrHandlerTypeNotFound.
	//
	// This is used by handlers to find siblings (such as the "ui" type handler)
	// which might have more knowledge about the configuration for discovery, etc.
	//
	// Note that if this is called during handler construction
	// time, only the prefix may be returned with a nil handler
	// and nil err.  Unlike GetHandler and GetStorage, this does
	// not cause the prefix to load immediately. At runtime (after
	// construction of all handlers), then prefix and handler will
	// both be non-nil when err is nil.
	FindHandlerByType(handlerType string) (prefix string, handler interface{}, err error)
}

type GenerationNotSupportedError

type GenerationNotSupportedError string

A GenerationNotSupportedError explains why a Storage value implemented the Generationer interface but failed due to a wrapped Storage value not implementing the interface.

func (GenerationNotSupportedError) Error

type Generationer

type Generationer interface {
	// Generation returns a Storage's initialization time and
	// and unique random string (or UUID).  Implementations
	// should call ResetStorageGeneration on demand if no
	// information is known.
	// The error will be of type GenerationNotSupportedError if an underlying
	// storage target doesn't support the Generationer interface.
	StorageGeneration() (initTime time.Time, random string, err error)

	// ResetGeneration deletes the information returned by Generation
	// and re-generates it.
	ResetStorageGeneration() error
}

The optional Generationer interface is an optimization and paranoia facility for clients which can be implemented by Storage implementations.

If the client sees the same random string in multiple upload sessions, it assumes that the blobserver still has all the same blobs, and also it's the same server. This mechanism is not fundamental to Camlistore's operation: the client could also check each blob before uploading, or enumerate all blobs from the server too. This is purely an optimization so clients can mix this value into their "is this file uploaded?" local cache keys.

type HandlerConstructor

type HandlerConstructor func(Loader, jsonconfig.Obj) (http.Handler, error)

A HandlerConstructor returns an http.Handler from a Loader environment and a configuration.

type Loader

type Loader interface {
	FindHandlerByTyper

	// MyPrefix returns the prefix of the handler currently being constructed.
	MyPrefix() string

	// GetHandlerType returns the handler's configured type, but does
	// not force it to start being loaded yet.
	GetHandlerType(prefix string) string // returns "" if unknown

	// GetHandler returns either a Storage or an http.Handler.
	// It forces the handler to be loaded and returns an error if
	// a cycle is created.
	GetHandler(prefix string) (interface{}, error)

	// GetStorage is like GetHandler but requires that the Handler be
	// a storage Handler.
	GetStorage(prefix string) (Storage, error)
}

type MaxEnumerateConfig

type MaxEnumerateConfig interface {
	Storage

	// MaxEnumerate returns the max that this storage interface is
	// capable of enumerating at once.
	MaxEnumerate() int
}

MaxEnumerateConfig is an optional interface implemented by Storage interfaces to advertise their max value for how many items can be enumerated at once.

type NoImplStorage

type NoImplStorage struct{}

NoImplStorage is an implementation of Storage that return a not implemented error for all operations.

func (*NoImplStorage) EnumerateBlobs

func (nis *NoImplStorage) EnumerateBlobs(dest chan<- blob.SizedRef, after string, limit int) error

func (*NoImplStorage) Fetch

func (nis *NoImplStorage) Fetch(blob.Ref) (file types.ReadSeekCloser, size int64, err error)

func (*NoImplStorage) FetchStreaming

func (nis *NoImplStorage) FetchStreaming(blob.Ref) (file io.ReadCloser, size int64, err error)

func (*NoImplStorage) ReceiveBlob

func (nis *NoImplStorage) ReceiveBlob(blob.Ref, io.Reader) (sb blob.SizedRef, err error)

func (*NoImplStorage) RemoveBlobs

func (nis *NoImplStorage) RemoveBlobs(blobs []blob.Ref) error

func (*NoImplStorage) StatBlobs

func (nis *NoImplStorage) StatBlobs(dest chan<- blob.SizedRef, blobs []blob.Ref) error

type ShutdownStorage

type ShutdownStorage interface {
	Storage
	io.Closer
}

Optional interface for storage implementations which can be asked to shut down cleanly. Regardless, all implementations should be able to survive crashes without data loss.

type StatReceiver

type StatReceiver interface {
	BlobReceiver
	BlobStatter
}

type Storage

Storage is the interface that must be implemented by a blobserver storage type. (e.g. localdisk, s3, encrypt, shard, replica, remote)

func CreateStorage

func CreateStorage(typ string, loader Loader, config jsonconfig.Obj) (Storage, error)

type StorageConfiger

type StorageConfiger interface {
	Storage
	Configer
}

type StorageConstructor

type StorageConstructor func(Loader, jsonconfig.Obj) (Storage, error)

A StorageConstructor returns a Storage implementation from a Loader environment and a configuration.

type StorageQueueCreator

type StorageQueueCreator interface {
	Storage

	CreateQueue(name string) (Storage, error)
}

StorageQueueCreator is implemented by Storage interfaces which support creating queues in which all new uploads go to both the root storage as well as the named queue, which is then returned. This is used by replication.

Directories

Path Synopsis
Package cond registers the "cond" conditional blobserver storage type to select routing of get/put operations on blobs to other storage targets as a function of their content.
Package cond registers the "cond" conditional blobserver storage type to select routing of get/put operations on blobs to other storage targets as a function of their content.
Package encrypt registers the "encrypt" blobserver storage type which stores all blobs and metadata with AES encryption into other wrapped storage targets (e.g.
Package encrypt registers the "encrypt" blobserver storage type which stores all blobs and metadata with AES encryption into other wrapped storage targets (e.g.
Package gethandler implements the HTTP handler for fetching blobs.
Package gethandler implements the HTTP handler for fetching blobs.
google
cloudstorage
Package cloudstorage registers the "googlecloudstorage" blob storage type, storing blobs on Google Cloud Storage (not Google Drive).
Package cloudstorage registers the "googlecloudstorage" blob storage type, storing blobs on Google Cloud Storage (not Google Drive).
drive
Package drive registers the "googledrive" blobserver storage type, storing blobs in a Google Drive folder.
Package drive registers the "googledrive" blobserver storage type, storing blobs in a Google Drive folder.
drive/service
DriveService translates blobserver.Storage methods into Google Drive API methods.
DriveService translates blobserver.Storage methods into Google Drive API methods.
Package handlers implements the HTTP interface to the Camlistore blob server.
Package handlers implements the HTTP interface to the Camlistore blob server.
Package localdisk registers the "filesystem" blobserver storage type, storing blobs in a forest of sharded directories at the specified root.
Package localdisk registers the "filesystem" blobserver storage type, storing blobs in a forest of sharded directories at the specified root.
Package remote registers the "remote" blobserver storage type, storing and fetching blobs from a remote Camlistore server, speaking the HTTP protocol.
Package remote registers the "remote" blobserver storage type, storing and fetching blobs from a remote Camlistore server, speaking the HTTP protocol.
Package replica registers the "replica" blobserver storage type, providing synchronous replication to one more backends.
Package replica registers the "replica" blobserver storage type, providing synchronous replication to one more backends.
Package s3 registers the "s3" blobserver storage type, storing blobs in an Amazon Web Services' S3 storage bucket.
Package s3 registers the "s3" blobserver storage type, storing blobs in an Amazon Web Services' S3 storage bucket.
Package shard registers the "shard" blobserver storage type, predictably spraying out blobs out over the provided backends based on their blobref.
Package shard registers the "shard" blobserver storage type, predictably spraying out blobs out over the provided backends based on their blobref.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL