swift: github.com/ncw/swift Index | Examples | Files | Directories

package swift

import "github.com/ncw/swift"

Package swift provides an easy to use interface to Swift / Openstack Object Storage / Rackspace Cloud Files

Standard Usage

Most of the work is done through the Container*() and Object*() methods.

All methods are safe to use concurrently in multiple go routines.

Object Versioning

As defined by http://docs.openstack.org/api/openstack-object-storage/1.0/content/Object_Versioning-e1e3230.html#d6e983 one can create a container which allows for version control of files. The suggested method is to create a version container for holding all non-current files, and a current container for holding the latest version that the file points to. The container and objects inside it can be used in the standard manner, however, pushing a file multiple times will result in it being copied to the version container and the new file put in it's place. If the current file is deleted, the previous file in the version container will replace it. This means that if a file is updated 5 times, it must be deleted 5 times to be completely removed from the system.

Rackspace Sub Module

This module specifically allows the enabling/disabling of Rackspace Cloud File CDN management on a container. This is specific to the Rackspace API and not Swift/Openstack, therefore it has been placed in a submodule. One can easily create a RsConnection and use it like the standard Connection to access and manipulate containers and objects.

Index

Examples

Package Files

auth.go auth_v3.go compatibility_1_1.go dlo.go doc.go largeobjects.go meta.go slo.go swift.go timeout_reader.go watchdog_reader.go

Constants

const (
    // Use public URL as storage URL
    EndpointTypePublic = EndpointType("public")

    // Use internal URL as storage URL
    EndpointTypeInternal = EndpointType("internal")

    // Use admin URL as storage URL
    EndpointTypeAdmin = EndpointType("admin")
)
const (
    DefaultUserAgent = "goswift/1.0"         // Default user agent
    DefaultRetries   = 3                     // Default number of retries on token expiry
    TimeFormat       = "2006-01-02T15:04:05" // Python date format for json replies parsed as UTC
    UploadTar        = "tar"                 // Data format specifier for Connection.BulkUpload().
    UploadTarGzip    = "tar.gz"              // Data format specifier for Connection.BulkUpload().
    UploadTarBzip2   = "tar.bz2"             // Data format specifier for Connection.BulkUpload().

)

Variables

var (
    // Specific Errors you might want to check for equality
    NotModified         = newError(304, "Not Modified")
    BadRequest          = newError(400, "Bad Request")
    AuthorizationFailed = newError(401, "Authorization Failed")
    ContainerNotFound   = newError(404, "Container Not Found")
    ContainerNotEmpty   = newError(409, "Container Not Empty")
    ObjectNotFound      = newError(404, "Object Not Found")
    ObjectCorrupted     = newError(422, "Object Corrupted")
    TimeoutError        = newError(408, "Timeout when reading or writing data")
    Forbidden           = newError(403, "Operation forbidden")
    TooLargeObject      = newError(413, "Too Large Object")
    RateLimit           = newError(498, "Rate Limit")

    // Mappings for container errors
    ContainerErrorMap = errorMap{
        400: BadRequest,
        403: Forbidden,
        404: ContainerNotFound,
        409: ContainerNotEmpty,
        498: RateLimit,
    }
)
var NotLargeObject = errors.New("Not a large object")

NotLargeObject is returned if an operation is performed on an object which isn't large.

var SLONotSupported = errors.New("SLO not supported")

func FloatStringToTime Uses

func FloatStringToTime(s string) (t time.Time, err error)

FloatStringToTime converts a floating point number string to a time.Time

The string is floating point number of seconds since the epoch (Unix time). The number should be in fixed point format (not exponential), eg "1354040105.123456789" which represents the time "2012-11-27T18:15:05.123456789Z"

Some care is taken to preserve all the accuracy in the time.Time (which wouldn't happen with a naive conversion through float64) so a round trip conversion won't change the data.

If an error is returned then time will be returned as the zero time.

func TimeToFloatString Uses

func TimeToFloatString(t time.Time) string

TimeToFloatString converts a time.Time object to a floating point string

The string is floating point number of seconds since the epoch (Unix time). The number is in fixed point format (not exponential), eg "1354040105.123456789" which represents the time "2012-11-27T18:15:05.123456789Z". Trailing zeros will be dropped from the output.

Some care is taken to preserve all the accuracy in the time.Time (which wouldn't happen with a naive conversion through float64) so a round trip conversion won't change the data.

type Account Uses

type Account struct {
    BytesUsed  int64 // total number of bytes used
    Containers int64 // total number of containers
    Objects    int64 // total number of objects
}

Account contains information about this account.

type Authenticator Uses

type Authenticator interface {
    // Request creates an http.Request for the auth - return nil if not needed
    Request(*Connection) (*http.Request, error)
    // Response parses the http.Response
    Response(resp *http.Response) error
    // The public storage URL - set Internal to true to read
    // internal/service net URL
    StorageUrl(Internal bool) string
    // The access token
    Token() string
    // The CDN url if available
    CdnUrl() string
}

Auth defines the operations needed to authenticate with swift

This encapsulates the different authentication schemes in use

type BulkDeleteResult Uses

type BulkDeleteResult struct {
    NumberNotFound int64            // # of objects not found.
    NumberDeleted  int64            // # of deleted objects.
    Errors         map[string]error // Mapping between object name and an error.
    Headers        Headers          // Response HTTP headers.
}

BulkDeleteResult stores results of BulkDelete().

Individual errors may (or may not) be returned by Errors. Errors is a map whose keys are a full path of where the object was to be deleted, and whose values are Error objects. A full path of object looks like "/API_VERSION/USER_ACCOUNT/CONTAINER/OBJECT_PATH".

type BulkUploadResult Uses

type BulkUploadResult struct {
    NumberCreated int64            // # of created objects.
    Errors        map[string]error // Mapping between object name and an error.
    Headers       Headers          // Response HTTP headers.
}

BulkUploadResult stores results of BulkUpload().

Individual errors may (or may not) be returned by Errors. Errors is a map whose keys are a full path of where an object was to be created, and whose values are Error objects. A full path of object looks like "/API_VERSION/USER_ACCOUNT/CONTAINER/OBJECT_PATH".

type Connection Uses

type Connection struct {
    // Parameters - fill these in before calling Authenticate
    // They are all optional except UserName, ApiKey and AuthUrl
    Domain         string            // User's domain name
    DomainId       string            // User's domain Id
    UserName       string            // UserName for api
    UserId         string            // User Id
    ApiKey         string            // Key for api access
    AuthUrl        string            // Auth URL
    Retries        int               // Retries on error (default is 3)
    UserAgent      string            // Http User agent (default goswift/1.0)
    ConnectTimeout time.Duration     // Connect channel timeout (default 10s)
    Timeout        time.Duration     // Data channel timeout (default 60s)
    Region         string            // Region to use eg "LON", "ORD" - default is use first region (v2,v3 auth only)
    AuthVersion    int               // Set to 1, 2 or 3 or leave at 0 for autodetect
    Internal       bool              // Set this to true to use the the internal / service network
    Tenant         string            // Name of the tenant (v2,v3 auth only)
    TenantId       string            // Id of the tenant (v2,v3 auth only)
    EndpointType   EndpointType      // Endpoint type (v2,v3 auth only) (default is public URL unless Internal is set)
    TenantDomain   string            // Name of the tenant's domain (v3 auth only), only needed if it differs from the user domain
    TenantDomainId string            // Id of the tenant's domain (v3 auth only), only needed if it differs the from user domain
    TrustId        string            // Id of the trust (v3 auth only)
    Transport      http.RoundTripper `json:"-" xml:"-"` // Optional specialised http.Transport (eg. for Google Appengine)
    // These are filled in after Authenticate is called as are the defaults for above
    StorageUrl string
    AuthToken  string

    Auth Authenticator `json:"-" xml:"-"` // the current authenticator
    // contains filtered or unexported fields
}

Connection holds the details of the connection to the swift server.

You need to provide UserName, ApiKey and AuthUrl when you create a connection then call Authenticate on it.

The auth version in use will be detected from the AuthURL - you can override this with the AuthVersion parameter.

If using v2 auth you can also set Region in the Connection structure. If you don't set Region you will get the default region which may not be what you want.

For reference some common AuthUrls looks like this:

Rackspace US        https://auth.api.rackspacecloud.com/v1.0
Rackspace UK        https://lon.auth.api.rackspacecloud.com/v1.0
Rackspace v2        https://identity.api.rackspacecloud.com/v2.0
Memset Memstore UK  https://auth.storage.memset.com/v1.0
Memstore v2         https://auth.storage.memset.com/v2.0

When using Google Appengine you must provide the Connection with an appengine-specific Transport:

import (
	"appengine/urlfetch"
	"fmt"
	"github.com/ncw/swift"
)

func handler(w http.ResponseWriter, r *http.Request) {
	ctx := appengine.NewContext(r)
	tr := urlfetch.Transport{Context: ctx}
	c := swift.Connection{
		UserName:  "user",
		ApiKey:    "key",
		AuthUrl:   "auth_url",
		Transport: tr,
	}
	_ := c.Authenticate()
	containers, _ := c.ContainerNames(nil)
	fmt.Fprintf(w, "containers: %q", containers)
}

If you don't supply a Transport, one is made which relies on http.ProxyFromEnvironment (http://golang.org/pkg/net/http/#ProxyFromEnvironment). This means that the connection will respect the HTTP proxy specified by the environment variables $HTTP_PROXY and $NO_PROXY.

Code:

// Create a v1 auth connection
c := &swift.Connection{
    // This should be your username
    UserName: "user",
    // This should be your api key
    ApiKey: "key",
    // This should be a v1 auth url, eg
    //  Rackspace US        https://auth.api.rackspacecloud.com/v1.0
    //  Rackspace UK        https://lon.auth.api.rackspacecloud.com/v1.0
    //  Memset Memstore UK  https://auth.storage.memset.com/v1.0
    AuthUrl: "auth_url",
}

// Authenticate
err := c.Authenticate()
if err != nil {
    panic(err)
}
// List all the containers
containers, err := c.ContainerNames(nil)
fmt.Println(containers)
// etc...

// ------ or alternatively create a v2 connection ------

// Create a v2 auth connection
c = &swift.Connection{
    // This is the sub user for the storage - eg "admin"
    UserName: "user",
    // This should be your api key
    ApiKey: "key",
    // This should be a version2 auth url, eg
    //  Rackspace v2        https://identity.api.rackspacecloud.com/v2.0
    //  Memset Memstore v2  https://auth.storage.memset.com/v2.0
    AuthUrl: "v2_auth_url",
    // Region to use - default is use first region if unset
    Region: "LON",
    // Name of the tenant - this is likely your username
    Tenant: "jim",
}

// as above...

func (*Connection) Account Uses

func (c *Connection) Account() (info Account, headers Headers, err error)

Account returns info about the account in an Account struct.

func (*Connection) AccountUpdate Uses

func (c *Connection) AccountUpdate(h Headers) error

AccountUpdate adds, replaces or remove account metadata.

Add or update keys by mentioning them in the Headers.

Remove keys by setting them to an empty string.

func (*Connection) ApplyEnvironment Uses

func (c *Connection) ApplyEnvironment() (err error)

ApplyEnvironment reads environment variables and applies them to the Connection structure. It won't overwrite any parameters which are already set in the Connection struct.

To make a new Connection object entirely from the environment you would do:

c := new(Connection)
err := c.ApplyEnvironment()
if err != nil { log.Fatal(err) }

The naming of these variables follows the official Openstack naming scheme so it should be compatible with OpenStack rc files.

For v1 authentication (obsolete)

ST_AUTH - Auth URL
ST_USER - UserName for api
ST_KEY - Key for api access

For v2 authentication

OS_AUTH_URL - Auth URL
OS_USERNAME - UserName for api
OS_PASSWORD - Key for api access
OS_TENANT_NAME - Name of the tenant
OS_TENANT_ID   - Id of the tenant
OS_REGION_NAME - Region to use - default is use first region

For v3 authentication

OS_AUTH_URL - Auth URL
OS_USERNAME - UserName for api
OS_USER_ID - User Id
OS_PASSWORD - Key for api access
OS_USER_DOMAIN_NAME - User's domain name
OS_USER_DOMAIN_ID - User's domain Id
OS_PROJECT_NAME - Name of the project
OS_PROJECT_DOMAIN_NAME - Name of the tenant's domain, only needed if it differs from the user domain
OS_PROJECT_DOMAIN_ID - Id of the tenant's domain, only needed if it differs the from user domain
OS_TRUST_ID - If of the trust
OS_REGION_NAME - Region to use - default is use first region

Other

OS_ENDPOINT_TYPE - Endpoint type public, internal or admin
ST_AUTH_VERSION - Choose auth version - 1, 2 or 3 or leave at 0 for autodetect

For manual authentication

OS_STORAGE_URL - storage URL from alternate authentication
OS_AUTH_TOKEN - Auth Token from alternate authentication

Library specific

GOSWIFT_RETRIES - Retries on error (default is 3)
GOSWIFT_USER_AGENT - HTTP User agent (default goswift/1.0)
GOSWIFT_CONNECT_TIMEOUT - Connect channel timeout with unit, eg "10s", "100ms" (default "10s")
GOSWIFT_TIMEOUT - Data channel timeout with unit, eg "10s", "100ms" (default "60s")
GOSWIFT_INTERNAL - Set this to "true" to use the the internal network (obsolete - use OS_ENDPOINT_TYPE)

func (*Connection) Authenticate Uses

func (c *Connection) Authenticate() (err error)

Authenticate connects to the Swift server.

If you don't call it before calling one of the connection methods then it will be called for you on the first access.

func (*Connection) Authenticated Uses

func (c *Connection) Authenticated() bool

Authenticated returns a boolean to show if the current connection is authenticated.

Doesn't actually check the credentials against the server.

func (*Connection) BulkDelete Uses

func (c *Connection) BulkDelete(container string, objectNames []string) (result BulkDeleteResult, err error)

BulkDelete deletes multiple objectNames from container in one operation.

Some servers may not accept bulk-delete requests since bulk-delete is an optional feature of swift - these will return the Forbidden error.

See also: * http://docs.openstack.org/trunk/openstack-object-storage/admin/content/object-storage-bulk-delete.html * http://docs.rackspace.com/files/api/v1/cf-devguide/content/Bulk_Delete-d1e2338.html

func (*Connection) BulkUpload Uses

func (c *Connection) BulkUpload(uploadPath string, dataStream io.Reader, format string, h Headers) (result BulkUploadResult, err error)

BulkUpload uploads multiple files in one operation.

uploadPath can be empty, a container name, or a pseudo-directory within a container. If uploadPath is empty, new containers may be automatically created.

Files are read from dataStream. The format of the stream is specified by the format parameter. Available formats are: * UploadTar - Plain tar stream. * UploadTarGzip - Gzip compressed tar stream. * UploadTarBzip2 - Bzip2 compressed tar stream.

Some servers may not accept bulk-upload requests since bulk-upload is an optional feature of swift - these will return the Forbidden error.

See also: * http://docs.openstack.org/trunk/openstack-object-storage/admin/content/object-storage-extract-archive.html * http://docs.rackspace.com/files/api/v1/cf-devguide/content/Extract_Archive-d1e2338.html

func (*Connection) Call Uses

func (c *Connection) Call(targetUrl string, p RequestOpts) (resp *http.Response, headers Headers, err error)

Call runs a remote command on the targetUrl, returns a response, headers and possible error.

operation is GET, HEAD etc container is the name of a container Any other parameters (if not None) are added to the targetUrl

Returns a response or an error. If response is returned then the resp.Body must be read completely and resp.Body.Close() must be called on it, unless noResponse is set in which case the body will be closed in this function

If "Content-Length" is set in p.Headers it will be used - this can be used to override the default chunked transfer encoding for uploads.

This will Authenticate if necessary, and re-authenticate if it receives a 401 error which means the token has expired

This method is exported so extensions can call it.

func (*Connection) Container Uses

func (c *Connection) Container(container string) (info Container, headers Headers, err error)

Container returns info about a single container including any metadata in the headers.

func (*Connection) ContainerCreate Uses

func (c *Connection) ContainerCreate(container string, h Headers) error

ContainerCreate creates a container.

If you don't want to add Headers just pass in nil

No error is returned if it already exists but the metadata if any will be updated.

func (*Connection) ContainerDelete Uses

func (c *Connection) ContainerDelete(container string) error

ContainerDelete deletes a container.

May return ContainerDoesNotExist or ContainerNotEmpty

func (*Connection) ContainerNames Uses

func (c *Connection) ContainerNames(opts *ContainersOpts) ([]string, error)

ContainerNames returns a slice of names of containers in this account.

func (*Connection) ContainerNamesAll Uses

func (c *Connection) ContainerNamesAll(opts *ContainersOpts) ([]string, error)

ContainerNamesAll is like ContainerNamess but it returns all the Containers

It calls ContainerNames multiple times using the Marker parameter

It has a default Limit parameter but you may pass in your own

func (*Connection) ContainerUpdate Uses

func (c *Connection) ContainerUpdate(container string, h Headers) error

ContainerUpdate adds, replaces or removes container metadata.

Add or update keys by mentioning them in the Metadata.

Remove keys by setting them to an empty string.

Container metadata can only be read with Container() not with Containers().

func (*Connection) Containers Uses

func (c *Connection) Containers(opts *ContainersOpts) ([]Container, error)

Containers returns a slice of structures with full information as described in Container.

func (*Connection) ContainersAll Uses

func (c *Connection) ContainersAll(opts *ContainersOpts) ([]Container, error)

ContainersAll is like Containers but it returns all the Containers

It calls Containers multiple times using the Marker parameter

It has a default Limit parameter but you may pass in your own

func (*Connection) DynamicLargeObjectCreate Uses

func (c *Connection) DynamicLargeObjectCreate(opts *LargeObjectOpts) (LargeObjectFile, error)

DynamicLargeObjectCreate creates or truncates an existing dynamic large object returning a writeable object. This sets opts.Flags to an appropriate value before calling DynamicLargeObjectCreateFile

func (*Connection) DynamicLargeObjectCreateFile Uses

func (c *Connection) DynamicLargeObjectCreateFile(opts *LargeObjectOpts) (LargeObjectFile, error)

DynamicLargeObjectCreateFile creates a dynamic large object returning an object which satisfies io.Writer, io.Seeker, io.Closer and io.ReaderFrom. The flags are as passes to the largeObjectCreate method.

func (*Connection) DynamicLargeObjectDelete Uses

func (c *Connection) DynamicLargeObjectDelete(container string, path string) error

DynamicLargeObjectDelete deletes a dynamic large object and all of its segments.

func (*Connection) DynamicLargeObjectMove Uses

func (c *Connection) DynamicLargeObjectMove(srcContainer string, srcObjectName string, dstContainer string, dstObjectName string) error

DynamicLargeObjectMove moves a dynamic large object from srcContainer, srcObjectName to dstContainer, dstObjectName

func (*Connection) LargeObjectDelete Uses

func (c *Connection) LargeObjectDelete(container string, objectName string) error

LargeObjectDelete deletes the large object named by container, path

func (*Connection) LargeObjectGetSegments Uses

func (c *Connection) LargeObjectGetSegments(container string, path string) (string, []Object, error)

LargeObjectGetSegments returns all the segments that compose an object If the object is a Dynamic Large Object (DLO), it just returns the objects that have the prefix as indicated by the manifest. If the object is a Static Large Object (SLO), it retrieves the JSON content of the manifest and return all the segments of it.

func (*Connection) Object Uses

func (c *Connection) Object(container string, objectName string) (info Object, headers Headers, err error)

Object returns info about a single object including any metadata in the header.

May return ObjectNotFound.

Use headers.ObjectMetadata() to read the metadata in the Headers.

func (*Connection) ObjectCopy Uses

func (c *Connection) ObjectCopy(srcContainer string, srcObjectName string, dstContainer string, dstObjectName string, h Headers) (headers Headers, err error)

ObjectCopy does a server side copy of an object to a new position

All metadata is preserved. If metadata is set in the headers then it overrides the old metadata on the copied object.

The destination container must exist before the copy.

You can use this to copy an object to itself - this is the only way to update the content type of an object.

func (*Connection) ObjectCreate Uses

func (c *Connection) ObjectCreate(container string, objectName string, checkHash bool, Hash string, contentType string, h Headers) (file *ObjectCreateFile, err error)

ObjectCreate creates or updates the object in the container. It returns an io.WriteCloser you should write the contents to. You MUST call Close() on it and you MUST check the error return from Close().

If checkHash is True then it will calculate the MD5 Hash of the file as it is being uploaded and check it against that returned from the server. If it is wrong then it will return ObjectCorrupted on Close()

If you know the MD5 hash of the object ahead of time then set the Hash parameter and it will be sent to the server (as an Etag header) and the server will check the MD5 itself after the upload, and this will return ObjectCorrupted on Close() if it is incorrect.

If you don't want any error protection (not recommended) then set checkHash to false and Hash to "".

If contentType is set it will be used, otherwise one will be guessed from objectName using mime.TypeByExtension

func (*Connection) ObjectDelete Uses

func (c *Connection) ObjectDelete(container string, objectName string) error

ObjectDelete deletes the object.

May return ObjectNotFound if the object isn't found

func (*Connection) ObjectGet Uses

func (c *Connection) ObjectGet(container string, objectName string, contents io.Writer, checkHash bool, h Headers) (headers Headers, err error)

ObjectGet gets the object into the io.Writer contents.

Returns the headers of the response.

If checkHash is true then it will calculate the md5sum of the file as it is being received and check it against that returned from the server. If it is wrong then it will return ObjectCorrupted.

headers["Content-Type"] will give the content type if desired.

func (*Connection) ObjectGetBytes Uses

func (c *Connection) ObjectGetBytes(container string, objectName string) (contents []byte, err error)

ObjectGetBytes returns an object as a []byte.

This is a simplified interface which checks the MD5

func (*Connection) ObjectGetString Uses

func (c *Connection) ObjectGetString(container string, objectName string) (contents string, err error)

ObjectGetString returns an object as a string.

This is a simplified interface which checks the MD5

func (*Connection) ObjectMove Uses

func (c *Connection) ObjectMove(srcContainer string, srcObjectName string, dstContainer string, dstObjectName string) (err error)

ObjectMove does a server side move of an object to a new position

This is a convenience method which calls ObjectCopy then ObjectDelete

All metadata is preserved.

The destination container must exist before the copy.

func (*Connection) ObjectNames Uses

func (c *Connection) ObjectNames(container string, opts *ObjectsOpts) ([]string, error)

ObjectNames returns a slice of names of objects in a given container.

func (*Connection) ObjectNamesAll Uses

func (c *Connection) ObjectNamesAll(container string, opts *ObjectsOpts) ([]string, error)

ObjectNamesAll is like ObjectNames but it returns all the Objects

It calls ObjectNames multiple times using the Marker parameter

It has a default Limit parameter but you may pass in your own

func (*Connection) ObjectOpen Uses

func (c *Connection) ObjectOpen(container string, objectName string, checkHash bool, h Headers) (file *ObjectOpenFile, headers Headers, err error)

ObjectOpen returns an ObjectOpenFile for reading the contents of the object. This satisfies the io.ReadCloser and the io.Seeker interfaces.

You must call Close() on contents when finished

Returns the headers of the response.

If checkHash is true then it will calculate the md5sum of the file as it is being received and check it against that returned from the server. If it is wrong then it will return ObjectCorrupted. It will also check the length returned. No checking will be done if you don't read all the contents.

Note that objects with X-Object-Manifest or X-Static-Large-Object set won't ever have their md5sum's checked as the md5sum reported on the object is actually the md5sum of the md5sums of the parts. This isn't very helpful to detect a corrupted download as the size of the parts aren't known without doing more operations. If you want to ensure integrity of an object with a manifest then you will need to download everything in the manifest separately.

headers["Content-Type"] will give the content type if desired.

func (*Connection) ObjectPut Uses

func (c *Connection) ObjectPut(container string, objectName string, contents io.Reader, checkHash bool, Hash string, contentType string, h Headers) (headers Headers, err error)

ObjectPut creates or updates the path in the container from contents. contents should be an open io.Reader which will have all its contents read.

This is a low level interface.

If checkHash is True then it will calculate the MD5 Hash of the file as it is being uploaded and check it against that returned from the server. If it is wrong then it will return ObjectCorrupted.

If you know the MD5 hash of the object ahead of time then set the Hash parameter and it will be sent to the server (as an Etag header) and the server will check the MD5 itself after the upload, and this will return ObjectCorrupted if it is incorrect.

If you don't want any error protection (not recommended) then set checkHash to false and Hash to "".

If contentType is set it will be used, otherwise one will be guessed from objectName using mime.TypeByExtension

func (*Connection) ObjectPutBytes Uses

func (c *Connection) ObjectPutBytes(container string, objectName string, contents []byte, contentType string) (err error)

ObjectPutBytes creates an object from a []byte in a container.

This is a simplified interface which checks the MD5.

func (*Connection) ObjectPutString Uses

func (c *Connection) ObjectPutString(container string, objectName string, contents string, contentType string) (err error)

ObjectPutString creates an object from a string in a container.

This is a simplified interface which checks the MD5

func (*Connection) ObjectTempUrl Uses

func (c *Connection) ObjectTempUrl(container string, objectName string, secretKey string, method string, expires time.Time) string

ObjectTempUrl returns a temporary URL for an object

func (*Connection) ObjectUpdate Uses

func (c *Connection) ObjectUpdate(container string, objectName string, h Headers) error

ObjectUpdate adds, replaces or removes object metadata.

Add or Update keys by mentioning them in the Metadata. Use Metadata.ObjectHeaders and Headers.ObjectMetadata to convert your Metadata to and from normal HTTP headers.

This removes all metadata previously added to the object and replaces it with that passed in so to delete keys, just don't mention them the headers you pass in.

Object metadata can only be read with Object() not with Objects().

This can also be used to set headers not already assigned such as X-Delete-At or X-Delete-After for expiring objects.

You cannot use this to change any of the object's other headers such as Content-Type, ETag, etc.

Refer to copying an object when you need to update metadata or other headers such as Content-Type or CORS headers.

May return ObjectNotFound.

func (*Connection) ObjectUpdateContentType Uses

func (c *Connection) ObjectUpdateContentType(container string, objectName string, contentType string) (err error)

ObjectUpdateContentType updates the content type of an object

This is a convenience method which calls ObjectCopy

All other metadata is preserved.

func (*Connection) Objects Uses

func (c *Connection) Objects(container string, opts *ObjectsOpts) ([]Object, error)

Objects returns a slice of Object with information about each object in the container.

If Delimiter is set in the opts then PseudoDirectory may be set, with ContentType 'application/directory'. These are not real objects but represent directories of objects which haven't had an object created for them.

func (*Connection) ObjectsAll Uses

func (c *Connection) ObjectsAll(container string, opts *ObjectsOpts) ([]Object, error)

ObjectsAll is like Objects but it returns an unlimited number of Objects in a slice

It calls Objects multiple times using the Marker parameter

func (*Connection) ObjectsWalk Uses

func (c *Connection) ObjectsWalk(container string, opts *ObjectsOpts, walkFn ObjectsWalkFn) error

ObjectsWalk is uses to iterate through all the objects in chunks as returned by Objects or ObjectNames using the Marker and Limit parameters in the ObjectsOpts.

Pass in a closure `walkFn` which calls Objects or ObjectNames with the *ObjectsOpts passed to it and does something with the results.

Errors will be returned from this function

It has a default Limit parameter but you may pass in your own

Code:

c, rollback := makeConnection(nil)
defer rollback()

objects := make([]string, 0)
err := c.ObjectsWalk(container, nil, func(opts *swift.ObjectsOpts) (interface{}, error) {
    newObjects, err := c.ObjectNames(container, opts)
    if err == nil {
        objects = append(objects, newObjects...)
    }
    return newObjects, err
})
fmt.Println("Found all the objects", objects, err)

func (*Connection) QueryInfo Uses

func (c *Connection) QueryInfo() (infos SwiftInfo, err error)

Discover Swift configuration by doing a request against /info

func (*Connection) StaticLargeObjectCreate Uses

func (c *Connection) StaticLargeObjectCreate(opts *LargeObjectOpts) (LargeObjectFile, error)

StaticLargeObjectCreate creates or truncates an existing static large object returning a writeable object. This sets opts.Flags to an appropriate value before calling StaticLargeObjectCreateFile

func (*Connection) StaticLargeObjectCreateFile Uses

func (c *Connection) StaticLargeObjectCreateFile(opts *LargeObjectOpts) (LargeObjectFile, error)

StaticLargeObjectCreateFile creates a static large object returning an object which satisfies io.Writer, io.Seeker, io.Closer and io.ReaderFrom. The flags are as passed to the largeObjectCreate method.

func (*Connection) StaticLargeObjectDelete Uses

func (c *Connection) StaticLargeObjectDelete(container string, path string) error

StaticLargeObjectDelete deletes a static large object and all of its segments.

func (*Connection) StaticLargeObjectMove Uses

func (c *Connection) StaticLargeObjectMove(srcContainer string, srcObjectName string, dstContainer string, dstObjectName string) error

StaticLargeObjectMove moves a static large object from srcContainer, srcObjectName to dstContainer, dstObjectName

func (*Connection) UnAuthenticate Uses

func (c *Connection) UnAuthenticate()

UnAuthenticate removes the authentication from the Connection.

func (*Connection) VersionContainerCreate Uses

func (c *Connection) VersionContainerCreate(current, version string) error

VersionContainerCreate is a helper method for creating and enabling version controlled containers.

It builds the current object container, the non-current object version container, and enables versioning.

If the server doesn't support versioning then it will return Forbidden however it will have created both the containers at that point.

Code:

c, rollback := makeConnection(nil)
defer rollback()

// Use the helper method to create the current and versions container.
if err := c.VersionContainerCreate("cds", "cd-versions"); err != nil {
    fmt.Print(err.Error())
}

func (*Connection) VersionDisable Uses

func (c *Connection) VersionDisable(current string) error

VersionDisable disables versioning on the current container.

Code:

c, rollback := makeConnection(nil)
defer rollback()

// Disable versioning on a container.  Note that this does not delete the versioning container.
c.VersionDisable("movies")

func (*Connection) VersionEnable Uses

func (c *Connection) VersionEnable(current, version string) error

VersionEnable enables versioning on the current container with version as the tracking container.

May return Forbidden if this isn't supported by the server

Code:

c, rollback := makeConnection(nil)
defer rollback()

// Build the containers manually and enable them.
if err := c.ContainerCreate("movie-versions", nil); err != nil {
    fmt.Print(err.Error())
}
if err := c.ContainerCreate("movies", nil); err != nil {
    fmt.Print(err.Error())
}
if err := c.VersionEnable("movies", "movie-versions"); err != nil {
    fmt.Print(err.Error())
}

// Access the primary container as usual with ObjectCreate(), ObjectPut(), etc.
// etc...

func (*Connection) VersionObjectList Uses

func (c *Connection) VersionObjectList(version, object string) ([]string, error)

VersionObjectList returns a list of older versions of the object.

Objects are returned in the format <length><object_name>/<timestamp>

type Container Uses

type Container struct {
    Name  string // Name of the container
    Count int64  // Number of objects in the container
    Bytes int64  // Total number of bytes used in the container
}

Container contains information about a container

type ContainersOpts Uses

type ContainersOpts struct {
    Limit     int     // For an integer value n, limits the number of results to at most n values.
    Prefix    string  // Given a string value x, return container names matching the specified prefix.
    Marker    string  // Given a string value x, return container names greater in value than the specified marker.
    EndMarker string  // Given a string value x, return container names less in value than the specified marker.
    Headers   Headers // Any additional HTTP headers - can be nil
}

ContainersOpts is options for Containers() and ContainerNames()

type CustomEndpointAuthenticator Uses

type CustomEndpointAuthenticator interface {
    StorageUrlForEndpoint(endpointType EndpointType) string
}

type DynamicLargeObjectCreateFile Uses

type DynamicLargeObjectCreateFile struct {
    // contains filtered or unexported fields
}

DynamicLargeObjectCreateFile represents an open static large object

func (*DynamicLargeObjectCreateFile) Close Uses

func (file *DynamicLargeObjectCreateFile) Close() error

Close satisfies the io.Closer interface

func (*DynamicLargeObjectCreateFile) Flush Uses

func (file *DynamicLargeObjectCreateFile) Flush() error

func (*DynamicLargeObjectCreateFile) Seek Uses

func (file *DynamicLargeObjectCreateFile) Seek(offset int64, whence int) (int64, error)

Seek sets the offset for the next write operation

func (*DynamicLargeObjectCreateFile) Size Uses

func (file *DynamicLargeObjectCreateFile) Size() int64

func (*DynamicLargeObjectCreateFile) Write Uses

func (file *DynamicLargeObjectCreateFile) Write(buf []byte) (int, error)

Write satisfies the io.Writer interface

type EndpointType Uses

type EndpointType string

type Error Uses

type Error struct {
    StatusCode int // HTTP status code if relevant or 0 if not
    Text       string
}

Error - all errors generated by this package are of this type. Other error may be passed on from library functions though.

func (*Error) Error Uses

func (e *Error) Error() string

Error satisfy the error interface.

type Headers Uses

type Headers map[string]string

Headers stores HTTP headers (can only have one of each header like Swift).

func (Headers) AccountMetadata Uses

func (h Headers) AccountMetadata() Metadata

AccountMetadata converts Headers from account to a Metadata.

The keys in the Metadata will be converted to lower case.

func (Headers) ContainerMetadata Uses

func (h Headers) ContainerMetadata() Metadata

ContainerMetadata converts Headers from container to a Metadata.

The keys in the Metadata will be converted to lower case.

func (Headers) IsLargeObject Uses

func (headers Headers) IsLargeObject() bool

func (Headers) IsLargeObjectDLO Uses

func (headers Headers) IsLargeObjectDLO() bool

func (Headers) IsLargeObjectSLO Uses

func (headers Headers) IsLargeObjectSLO() bool

func (Headers) Metadata Uses

func (h Headers) Metadata(metaPrefix string) Metadata

Metadata gets the Metadata starting with the metaPrefix out of the Headers.

The keys in the Metadata will be converted to lower case

func (Headers) ObjectMetadata Uses

func (h Headers) ObjectMetadata() Metadata

ObjectMetadata converts Headers from object to a Metadata.

The keys in the Metadata will be converted to lower case.

type LargeObjectFile Uses

type LargeObjectFile interface {
    io.Writer
    io.Seeker
    io.Closer
    Size() int64
    Flush() error
}

type LargeObjectOpts Uses

type LargeObjectOpts struct {
    Container        string  // Name of container to place object
    ObjectName       string  // Name of object
    Flags            int     // Creation flags
    CheckHash        bool    // If set Check the hash
    Hash             string  // If set use this hash to check
    ContentType      string  // Content-Type of the object
    Headers          Headers // Additional headers to upload the object with
    ChunkSize        int64   // Size of chunks of the object, defaults to 10MB if not set
    MinChunkSize     int64   // Minimum chunk size, automatically set for SLO's based on info
    SegmentContainer string  // Name of the container to place segments
    SegmentPrefix    string  // Prefix to use for the segments
    NoBuffer         bool    // Prevents using a bufio.Writer to write segments
}

LargeObjectOpts describes how a large object should be created

type Metadata Uses

type Metadata map[string]string

Metadata stores account, container or object metadata.

func (Metadata) AccountHeaders Uses

func (m Metadata) AccountHeaders() Headers

AccountHeaders converts the Metadata for the account.

func (Metadata) ContainerHeaders Uses

func (m Metadata) ContainerHeaders() Headers

ContainerHeaders converts the Metadata for the container.

func (Metadata) GetModTime Uses

func (m Metadata) GetModTime() (t time.Time, err error)

Read a modification time (mtime) from a Metadata object

This is a defacto standard (used in the official python-swiftclient amongst others) for storing the modification time (as read using os.Stat) for an object. It is stored using the key 'mtime', which for example when written to an object will be 'X-Object-Meta-Mtime'.

If an error is returned then time will be returned as the zero time.

func (Metadata) Headers Uses

func (m Metadata) Headers(metaPrefix string) Headers

Headers convert the Metadata starting with the metaPrefix into a Headers.

The keys in the Metadata will be converted from lower case to http Canonical (see http.CanonicalHeaderKey).

func (Metadata) ObjectHeaders Uses

func (m Metadata) ObjectHeaders() Headers

ObjectHeaders converts the Metadata for the object.

func (Metadata) SetModTime Uses

func (m Metadata) SetModTime(t time.Time)

Write an modification time (mtime) to a Metadata object

This is a defacto standard (used in the official python-swiftclient amongst others) for storing the modification time (as read using os.Stat) for an object. It is stored using the key 'mtime', which for example when written to an object will be 'X-Object-Meta-Mtime'.

type Object Uses

type Object struct {
    Name               string     `json:"name"`          // object name
    ContentType        string     `json:"content_type"`  // eg application/directory
    Bytes              int64      `json:"bytes"`         // size in bytes
    ServerLastModified string     `json:"last_modified"` // Last modified time, eg '2011-06-30T08:20:47.736680' as a string supplied by the server
    LastModified       time.Time  // Last modified time converted to a time.Time
    Hash               string     `json:"hash"`     // MD5 hash, eg "d41d8cd98f00b204e9800998ecf8427e"
    SLOHash            string     `json:"slo_etag"` // MD5 hash of all segments' MD5 hash, eg "d41d8cd98f00b204e9800998ecf8427e"
    PseudoDirectory    bool       // Set when using delimiter to show that this directory object does not really exist
    SubDir             string     `json:"subdir"` // returned only when using delimiter to mark "pseudo directories"
    ObjectType         ObjectType // type of this object
}

Object contains information about an object

type ObjectCreateFile Uses

type ObjectCreateFile struct {
    // contains filtered or unexported fields
}

ObjectCreateFile represents a swift object open for writing

func (*ObjectCreateFile) Close Uses

func (file *ObjectCreateFile) Close() error

Close the object and checks the md5sum if it was required.

Also returns any other errors from the server (eg container not found) so it is very important to check the errors on this method.

func (*ObjectCreateFile) Headers Uses

func (file *ObjectCreateFile) Headers() (Headers, error)

Headers returns the response headers from the created object if the upload has been completed. The Close() method must be called on an ObjectCreateFile before this method.

func (*ObjectCreateFile) Write Uses

func (file *ObjectCreateFile) Write(p []byte) (n int, err error)

Write bytes to the object - see io.Writer

type ObjectOpenFile Uses

type ObjectOpenFile struct {
    // contains filtered or unexported fields
}

ObjectOpenFile represents a swift object open for reading

func (*ObjectOpenFile) Close Uses

func (file *ObjectOpenFile) Close() (err error)

Close the object and checks the length and md5sum if it was required and all the object was read

func (*ObjectOpenFile) Length Uses

func (file *ObjectOpenFile) Length() (int64, error)

Length gets the objects content length either from a cached copy or from the server.

func (*ObjectOpenFile) Read Uses

func (file *ObjectOpenFile) Read(p []byte) (n int, err error)

Read bytes from the object - see io.Reader

func (*ObjectOpenFile) Seek Uses

func (file *ObjectOpenFile) Seek(offset int64, whence int) (newPos int64, err error)

Seek sets the offset for the next Read to offset, interpreted according to whence: 0 means relative to the origin of the file, 1 means relative to the current offset, and 2 means relative to the end. Seek returns the new offset and an Error, if any.

Seek uses HTTP Range headers which, if the file pointer is moved, will involve reopening the HTTP connection.

Note that you can't seek to the end of a file or beyond; HTTP Range requests don't support the file pointer being outside the data, unlike os.File

Seek(0, 1) will return the current file pointer.

type ObjectType Uses

type ObjectType int

ObjectType is the type of the swift object, regular, static large, or dynamic large.

const (
    RegularObjectType ObjectType = iota
    StaticLargeObjectType
    DynamicLargeObjectType
)

Values that ObjectType can take

type ObjectsOpts Uses

type ObjectsOpts struct {
    Limit     int     // For an integer value n, limits the number of results to at most n values.
    Marker    string  // Given a string value x, return object names greater in value than the  specified marker.
    EndMarker string  // Given a string value x, return object names less in value than the specified marker
    Prefix    string  // For a string value x, causes the results to be limited to object names beginning with the substring x.
    Path      string  // For a string value x, return the object names nested in the pseudo path
    Delimiter rune    // For a character c, return all the object names nested in the container
    Headers   Headers // Any additional HTTP headers - can be nil
}

ObjectOpts is options for Objects() and ObjectNames()

type ObjectsWalkFn Uses

type ObjectsWalkFn func(*ObjectsOpts) (interface{}, error)

A closure defined by the caller to iterate through all objects

Call Objects or ObjectNames from here with the *ObjectOpts passed in

Do whatever is required with the results then return them

type RequestOpts Uses

type RequestOpts struct {
    Container  string
    ObjectName string
    Operation  string
    Parameters url.Values
    Headers    Headers
    ErrorMap   errorMap
    NoResponse bool
    Body       io.Reader
    Retries    int
    // if set this is called on re-authentication to refresh the targetUrl
    OnReAuth func() (string, error)
}

RequestOpts contains parameters for Connection.storage.

type StaticLargeObjectCreateFile Uses

type StaticLargeObjectCreateFile struct {
    // contains filtered or unexported fields
}

StaticLargeObjectCreateFile represents an open static large object

func (*StaticLargeObjectCreateFile) Close Uses

func (file *StaticLargeObjectCreateFile) Close() error

func (*StaticLargeObjectCreateFile) Flush Uses

func (file *StaticLargeObjectCreateFile) Flush() error

func (*StaticLargeObjectCreateFile) Seek Uses

func (file *StaticLargeObjectCreateFile) Seek(offset int64, whence int) (int64, error)

Seek sets the offset for the next write operation

func (*StaticLargeObjectCreateFile) Size Uses

func (file *StaticLargeObjectCreateFile) Size() int64

func (*StaticLargeObjectCreateFile) Write Uses

func (file *StaticLargeObjectCreateFile) Write(buf []byte) (int, error)

Write satisfies the io.Writer interface

type SwiftInfo Uses

type SwiftInfo map[string]interface{}

SwiftInfo contains the JSON object returned by Swift when the /info route is queried. The object contains, among others, the Swift version, the enabled middlewares and their configuration

func (SwiftInfo) SLOMinSegmentSize Uses

func (i SwiftInfo) SLOMinSegmentSize() int64

func (SwiftInfo) SupportsBulkDelete Uses

func (i SwiftInfo) SupportsBulkDelete() bool

func (SwiftInfo) SupportsSLO Uses

func (i SwiftInfo) SupportsSLO() bool

Directories

PathSynopsis
rs
swifttestThis implements a very basic Swift server Everything is stored in memory

Package swift imports 22 packages (graph) and is imported by 216 packages. Updated 2018-10-11. Refresh now. Tools for package owners.