go-cloud: github.com/google/go-cloud/blob Index | Examples | Files | Directories

package blob

import "github.com/google/go-cloud/blob"

Package blob provides an easy way to interact with Blob objects within a bucket. It utilizes standard io packages to handle reads and writes.

Index

Examples

Package Files

blob.go

Constants

const DefaultSignedURLExpiry = 1 * time.Hour

DefaultSignedURLExpiry is the default duration for SignedURLOptions.Expiry.

const MaxPageSize = 1000

MaxPageSize is the maximum value for ListOptions.PageSize.

func IsNotExist Uses

func IsNotExist(err error) bool

IsNotExist returns true iff err indicates that the referenced blob does not exist.

func IsNotImplemented Uses

func IsNotImplemented(err error) bool

IsNotImplemented returns true iff err indicates that the provider does not support the given operation.

type Attributes Uses

type Attributes struct {
    // ContentType is the MIME type of the blob object. It will not be empty.
    ContentType string
    // Metadata holds key/value pairs associated with the blob.
    // Keys are guaranteed to be in lowercase, even if the backend provider
    // has case-sensitive keys (although note that Metadata written via
    // this package will always be lowercased). If there are duplicate
    // case-insensitive keys (e.g., "foo" and "FOO"), only one value
    // will be kept, and it is undefined which one.
    Metadata map[string]string
    // ModTime is the time the blob object was last modified.
    ModTime time.Time
    // Size is the size of the object in bytes.
    Size int64
    // contains filtered or unexported fields
}

Attributes contains attributes about a blob.

func (*Attributes) As Uses

func (a *Attributes) As(i interface{}) bool

As converts i to provider-specific types. See Bucket.As for more details.

type Bucket Uses

type Bucket struct {
    // contains filtered or unexported fields
}

Bucket manages the underlying blob service and provides read, write and delete operations on given object within it.

func NewBucket Uses

func NewBucket(b driver.Bucket) *Bucket

NewBucket creates a new Bucket for a group of objects for a blob service.

func (*Bucket) As Uses

func (b *Bucket) As(i interface{}) bool

As converts i to provider-specific types. See provider documentation for which type(s) are supported.

Usage: 1. Declare a variable of the provider-specific type you want to access. 2. Pass a pointer to it to As. 3. As will return true iff the type is supported, and copy the

provider-specific type into your variable.

Provider-specific types that are intended to be mutable will be exposed as a pointer to the underlying type.

See https://github.com/google/go-cloud/blob/master/internal/docs/design.md#as for more background.

Code:

// Connect to a bucket when your program starts up.
// This example uses the file-based implementation.
dir, cleanup := newTempDir()
defer cleanup()

// Create the file-based bucket.
bucket, err := fileblob.NewBucket(dir)
if err != nil {
    log.Fatal(err)
}
// This example uses As to try to fill in a string variable. As will return
// false because fileblob doesn't support any types for Bucket.As.
// See the package documentation for your provider (e.g., gcsblob or s3blob)
// to see what type(s) it supports.
var providerSpecific string
if bucket.As(&providerSpecific) {
    fmt.Println("fileblob supports the `string` type for Bucket.As")
    // Use providerSpecific.
} else {
    fmt.Println("fileblob does not support the `string` type for Bucket.As")
}

// This example sets WriterOptions.BeforeWrite to be called before the
// provider starts writing. In the callback, it uses asFunc to try to fill in
// a *string. Again, asFunc will return false because fileblob doesn't support
// any types for Writer.
fn := func(asFunc func(i interface{}) bool) error {
    var mutableProviderSpecific *string
    if asFunc(&mutableProviderSpecific) {
        fmt.Println("fileblob supports the `*string` type for WriterOptions.BeforeWrite")
        // Use mutableProviderSpecific.
    } else {
        fmt.Println("fileblob does not support the `*string` type for WriterOptions.BeforeWrite")
    }
    return nil
}
ctx := context.Background()
if err := bucket.WriteAll(ctx, "foo.txt", []byte("Go Cloud"), &blob.WriterOptions{BeforeWrite: fn}); err != nil {
    log.Fatal(err)
}

Output:

fileblob does not support the `string` type for Bucket.As
fileblob does not support the `*string` type for WriterOptions.BeforeWrite

func (*Bucket) Attributes Uses

func (b *Bucket) Attributes(ctx context.Context, key string) (Attributes, error)

Attributes reads attributes for the given key.

func (*Bucket) Delete Uses

func (b *Bucket) Delete(ctx context.Context, key string) error

Delete deletes the object associated with key. It returns an error if that object does not exist, which can be checked by calling IsNotExist.

func (*Bucket) List Uses

func (b *Bucket) List(ctx context.Context, opt *ListOptions) (*ListIterator, error)

List returns an object that can be used to iterate over objects in a bucket, in lexicographical order of UTF-8 encoded keys. The underlying implementation fetches results in pages. Use ListOptions to control the page size and filtering.

List is not guaranteed to include all recently-written objects; some providers are only eventually consistent.

Code:

// Connect to a bucket when your program starts up.
// This example uses the file-based implementation.
dir, cleanup := newTempDir()
defer cleanup()

// Create the file-based bucket.
bucket, err := fileblob.NewBucket(dir)
if err != nil {
    log.Fatal(err)
}

// Create some blob objects for listing: "foo[0..4].txt".
ctx := context.Background()
createListableFiles(ctx, bucket)

// Iterate over them.
// This will list the blobs created above because fileblob is strongly
// consistent, but is not guaranteed to work on all providers.
iter, err := bucket.List(ctx, nil)
if err != nil {
    log.Fatal(err)
}
for {
    obj, err := iter.Next(ctx)
    if err != nil {
        log.Fatal(err)
    }
    if obj == nil {
        break
    }
    fmt.Println(obj.Key)
}

Output:

foo0.txt
foo1.txt
foo2.txt
foo3.txt
foo4.txt

func (*Bucket) NewRangeReader Uses

func (b *Bucket) NewRangeReader(ctx context.Context, key string, offset, length int64) (*Reader, error)

NewRangeReader returns a Reader that reads part of an object, reading at most length bytes starting at the given offset. If length is negative, it will read till the end of the object. offset must be >= 0, and length cannot be 0.

NewRangeReader returns an error if the object does not exist, which can be checked by calling IsNotExist. Attributes() is a lighter-weight way to check for existence.

The caller must call Close on the returned Reader when done reading.

Code:

// Connect to a bucket when your program starts up.
// This example uses the file-based implementation.
dir, cleanup := newTempDir()
defer cleanup()
// Write a file to read using the bucket.
err := ioutil.WriteFile(filepath.Join(dir, "foo.txt"), []byte("Hello, World!\n"), 0666)
if err != nil {
    log.Fatal(err)
}
// Create the file-based bucket.
bucket, err := fileblob.NewBucket(dir)
if err != nil {
    log.Fatal(err)
}

// Open a reader using the blob's key at a specific offset at length.
ctx := context.Background()
r, err := bucket.NewRangeReader(ctx, "foo.txt", 1, 4)
if err != nil {
    log.Fatal(err)
}
defer r.Close()
// The blob reader implements io.Reader, so we can use any function that
// accepts an io.Reader.
if _, err := io.Copy(os.Stdout, r); err != nil {
    log.Fatal(err)
}

Output:

ello

func (*Bucket) NewReader Uses

func (b *Bucket) NewReader(ctx context.Context, key string) (*Reader, error)

NewReader returns a Reader to read from an object, or an error when the object is not found by the given key, which can be checked by calling IsNotExist.

The caller must call Close on the returned Reader when done reading.

Code:

// Connect to a bucket when your program starts up.
// This example uses the file-based implementation.
dir, cleanup := newTempDir()
defer cleanup()
// Write a file to read using the bucket.
err := ioutil.WriteFile(filepath.Join(dir, "foo.txt"), []byte("Hello, World!\n"), 0666)
if err != nil {
    log.Fatal(err)
}
// Create the file-based bucket.
bucket, err := fileblob.NewBucket(dir)
if err != nil {
    log.Fatal(err)
}

// Open a reader using the blob's key.
ctx := context.Background()
r, err := bucket.NewReader(ctx, "foo.txt")
if err != nil {
    log.Fatal(err)
}
defer r.Close()
// The blob reader implements io.Reader, so we can use any function that
// accepts an io.Reader.
if _, err := io.Copy(os.Stdout, r); err != nil {
    log.Fatal(err)
}

Output:

Hello, World!

func (*Bucket) NewWriter Uses

func (b *Bucket) NewWriter(ctx context.Context, key string, opt *WriterOptions) (*Writer, error)

NewWriter returns a Writer that writes to an object associated with key.

A new object will be created unless an object with this key already exists. Otherwise any previous object with the same key will be replaced. The object is not guaranteed to be available until Close has been called.

The returned Writer will store the ctx for later use in Write and/or Close. To abort a write, cancel the provided ctx; othewise it must remain open until Close is called.

The caller must call Close on the returned Writer, even if the write is aborted.

Code:

// Connect to a bucket when your program starts up.
// This example uses the file-based implementation.
dir, cleanup := newTempDir()
defer cleanup()
bucket, err := fileblob.NewBucket(dir)
if err != nil {
    log.Fatal(err)
}

// Open a writer using the key "foo.txt" and the default options.
ctx := context.Background()
// fileblob doesn't support custom content-type yet, see
// https://github.com/google/go-cloud/issues/111.
w, err := bucket.NewWriter(ctx, "foo.txt", &blob.WriterOptions{
    ContentType: "application/octet-stream",
})
if err != nil {
    log.Fatal(err)
}
// The blob writer implements io.Writer, so we can use any function that
// accepts an io.Writer. A writer must always be closed.
_, printErr := fmt.Fprintln(w, "Hello, World!")
closeErr := w.Close()
if printErr != nil {
    log.Fatal(printErr)
}
if closeErr != nil {
    log.Fatal(closeErr)
}
// Copy the written blob to stdout.
r, err := bucket.NewReader(ctx, "foo.txt")
if err != nil {
    log.Fatal(err)
}
defer r.Close()
if _, err := io.Copy(os.Stdout, r); err != nil {
    log.Fatal(err)
}

Output:

Hello, World!

func (*Bucket) ReadAll Uses

func (b *Bucket) ReadAll(ctx context.Context, key string) ([]byte, error)

ReadAll is a shortcut for creating a Reader via NewReader and reading the entire blob.

Code:

// Connect to a bucket when your program starts up.
// This example uses the file-based implementation.
dir, cleanup := newTempDir()
defer cleanup()

// Create the file-based bucket.
bucket, err := fileblob.NewBucket(dir)
if err != nil {
    log.Fatal(err)
}

// Write a blob using WriteAll.
ctx := context.Background()
if err := bucket.WriteAll(ctx, "foo.txt", []byte("Go Cloud"), nil); err != nil {
    log.Fatal(err)
}

// Read it back using ReadAll.
b, err := bucket.ReadAll(ctx, "foo.txt")
if err != nil {
    log.Fatal(err)
}
fmt.Println(string(b))

Output:

Go Cloud

func (*Bucket) SignedURL Uses

func (b *Bucket) SignedURL(ctx context.Context, key string, opts *SignedURLOptions) (string, error)

SignedURL returns a URL that can be used to GET the blob for the duration specified in opts.Expiry. If IsNotImplemented returns true for the returned error, the provider does not support SignedURL.

func (*Bucket) WriteAll Uses

func (b *Bucket) WriteAll(ctx context.Context, key string, p []byte, opt *WriterOptions) error

WriteAll is a shortcut for creating a Writer via NewWriter and writing p.

type ListIterator Uses

type ListIterator struct {
    // contains filtered or unexported fields
}

ListIterator is used to iterate over List results.

func (*ListIterator) Next Uses

func (i *ListIterator) Next(ctx context.Context) (*ListObject, error)

Next returns the next object. It returns nil if there are no more objects.

type ListObject Uses

type ListObject struct {
    // Key is the key for this blob.
    Key string
    // ModTime is the time the blob object was last modified.
    ModTime time.Time
    // Size is the size of the object in bytes.
    Size int64
    // contains filtered or unexported fields
}

ListObject represents a single blob object returned from List.

func (*ListObject) As Uses

func (o *ListObject) As(i interface{}) bool

As converts i to provider-specific types. See Bucket.As for more details.

type ListOptions Uses

type ListOptions struct {
    // Prefix indicates that only objects with a key starting with this prefix
    // should be returned.
    Prefix string

    // PageSize sets the maximum number of objects that will be retrieved
    // at a time from the provider.
    // Must be >= 0 and <= MaxPageSize; 0 defaults to MaxPageSize.
    PageSize int

    // BeforeList is a callback that will be called before each call to the
    // the underlying provider's list functionality.
    // asFunc converts its argument to provider-specific types.
    // See Bucket.As for more details.
    BeforeList func(asFunc func(interface{}) bool) error
}

ListOptions sets options for listing objects. TODO(Issue #541): Add Delimiter.

type Reader Uses

type Reader struct {
    // contains filtered or unexported fields
}

Reader implements io.ReadCloser to read a blob. It must be closed after reads are finished.

func (*Reader) As Uses

func (r *Reader) As(i interface{}) bool

As converts i to provider-specific types. See Bucket.As for more details.

func (*Reader) Close Uses

func (r *Reader) Close() error

Close implements io.ReadCloser to close this reader.

func (*Reader) ContentType Uses

func (r *Reader) ContentType() string

ContentType returns the MIME type of the blob object.

func (*Reader) ModTime Uses

func (r *Reader) ModTime() time.Time

ModTime is the time the blob object was last modified.

func (*Reader) Read Uses

func (r *Reader) Read(p []byte) (int, error)

Read implements io.ReadCloser to read from this reader.

func (*Reader) Size Uses

func (r *Reader) Size() int64

Size returns the content size of the blob object.

type SignedURLOptions Uses

type SignedURLOptions struct {
    // Expiry sets how long the returned URL is valid for.
    // Defaults to DefaultSignedURLExpiry.
    Expiry time.Duration
}

SignedURLOptions sets options for SignedURL.

type Writer Uses

type Writer struct {
    // contains filtered or unexported fields
}

Writer implements io.WriteCloser to write to blob. It must be closed after all writes are done.

func (*Writer) Close Uses

func (w *Writer) Close() error

Close flushes any buffered data and completes the Write. It is the user's responsibility to call it after finishing the write and handle the error if returned. Close will return an error if the ctx provided to create w is canceled.

func (*Writer) Write Uses

func (w *Writer) Write(p []byte) (n int, err error)

Write implements the io.Writer interface.

The writes happen asynchronously, which means the returned error can be nil even if the actual write fails. Use the error returned from Close to check and handle errors.

type WriterOptions Uses

type WriterOptions struct {
    // BufferSize changes the default size in bytes of the maximum part Writer can
    // write in a single request. Larger objects will be split into multiple requests.
    //
    // The support specification of this operation varies depending on the underlying
    // blob service. If zero value is given, it is set to a reasonable default value.
    // If negative value is given, it will be either disabled (if supported by the
    // service), which means Writer will write as a whole, or reset to default value.
    // It could be a no-op when not supported at all.
    //
    // If the Writer is used to write small objects concurrently, set the buffer size
    // to a smaller size to avoid high memory usage.
    BufferSize int

    // ContentType specifies the MIME type of the object being written. If not set,
    // then it will be inferred from the content using the algorithm described at
    // http://mimesniff.spec.whatwg.org/
    ContentType string

    // Metadata are key/value strings to be associated with the blob, or nil.
    // Keys may not be empty, and are lowercased before being written.
    // Duplicate case-insensitive keys (e.g., "foo" and "FOO") are an error.
    Metadata map[string]string

    // BeforeWrite is a callback that will be called exactly once, before
    // any data is written (unless NewWriter returns an error, in which case
    // it will not be called at all). Note that this is not necessarily during
    // or after the first Write call, as providers may buffer.
    // asFunc converts its argument to provider-specific types.
    // See Bucket.As for more details.
    BeforeWrite func(asFunc func(interface{}) bool) error
}

WriterOptions controls Writer behaviors.

Directories

PathSynopsis
driverPackage driver defines a set of interfaces that the blob package uses to interact with the underlying blob services.
drivertestPackage drivertest provides a conformance test for implementations of driver.
fileblobPackage fileblob provides a bucket implementation that operates on the local filesystem.
gcsblobPackage gcsblob provides an implementation of blob that uses GCS.
s3blobPackage s3blob provides an implementation of blob using S3.

Package blob imports 10 packages (graph) and is imported by 12 packages. Updated 2018-10-18. Refresh now. Tools for package owners.