azureblob

package
v0.18.1-0...-1aa001a Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Feb 27, 2020 License: Apache-2.0 Imports: 22 Imported by: 0

Documentation

Overview

Package azureblob provides a blob implementation that uses Azure Storage’s BlockBlob. Use OpenBucket to construct a *blob.Bucket.

NOTE: SignedURLs for PUT created with this package are not fully portable; they will not work unless the PUT request includes a "x-ms-blob-type" header set to "BlockBlob". See https://stackoverflow.com/questions/37824136/put-on-sas-blob-url-without-specifying-x-ms-blob-type-header.

URLs

For blob.OpenBucket, azureblob registers for the scheme "azblob". The default URL opener will use credentials from the environment variables AZURE_STORAGE_ACCOUNT, AZURE_STORAGE_KEY, and AZURE_STORAGE_SAS_TOKEN. AZURE_STORAGE_ACCOUNT is required, along with one of the other two. AZURE_STORAGE_DOMAIN can optionally be used to provide an Azure Environment blob storage domain to use. If no AZURE_STORAGE_DOMAIN is provided, the default Azure public domain "blob.core.windows.net" will be used. Check the Azure Developer Guide for your particular cloud environment to see the proper blob storage domain name to provide. To customize the URL opener, or for more details on the URL format, see URLOpener. See https://github.com/kainoaseto/go-cloud/concepts/urls/ for background information.

Escaping

Go CDK supports all UTF-8 strings; to make this work with services lacking full UTF-8 support, strings must be escaped (during writes) and unescaped (during reads). The following escapes are performed for azureblob:

  • Blob keys: ASCII characters 0-31, 92 ("\"), and 127 are escaped to "__0x<hex>__". Additionally, the "/" in "../" and a trailing "/" in a key (e.g., "foo/") are escaped in the same way.
  • Metadata keys: Per https://docs.microsoft.com/en-us/azure/storage/blobs/storage-properties-metadata, Azure only allows C# identifiers as metadata keys. Therefore, characters other than "[a-z][A-z][0-9]_" are escaped using "__0x<hex>__". In addition, characters "[0-9]" are escaped when they start the string. URL encoding would not work since "%" is not valid.
  • Metadata values: Escaped using URL encoding.

As

azureblob exposes the following types for As:

  • Bucket: *azblob.ContainerURL
  • Error: azblob.StorageError
  • ListObject: azblob.BlobItem for objects, azblob.BlobPrefix for "directories"
  • ListOptions.BeforeList: *azblob.ListBlobsSegmentOptions
  • Reader: azblob.DownloadResponse
  • Reader.BeforeRead: *azblob.BlockBlobURL, *azblob.BlobAccessConditions
  • Attributes: azblob.BlobGetPropertiesResponse
  • CopyOptions.BeforeCopy: azblob.Metadata, *azblob.ModifiedAccessConditions, *azblob.BlobAccessConditions
  • WriterOptions.BeforeWrite: *azblob.UploadStreamToBlockBlobOptions
Example (OpenBucketFromURL)
package main

import (
	"context"
	"log"

	"github.com/kainoaseto/go-cloud/blob"
)

func main() {
	// PRAGMA: This example is used on github.com/kainoaseto/go-cloud; PRAGMA comments adjust how it is shown and can be ignored.
	// PRAGMA: On github.com/kainoaseto/go-cloud, add a blank import: _ "github.com/kainoaseto/go-cloud/blob/azureblob"
	// PRAGMA: On github.com/kainoaseto/go-cloud, hide lines until the next blank line.
	ctx := context.Background()

	// blob.OpenBucket creates a *blob.Bucket from a URL.
	// This URL will open the container "my-container" using default
	// credentials found in the environment variables
	// AZURE_STORAGE_ACCOUNT plus at least one of AZURE_STORAGE_KEY
	// and AZURE_STORAGE_SAS_TOKEN.
	bucket, err := blob.OpenBucket(ctx, "azblob://my-container")
	if err != nil {
		log.Fatal(err)
	}
	defer bucket.Close()
}
Output:

Index

Examples

Constants

View Source
const Scheme = "azblob"

Scheme is the URL scheme gcsblob registers its URLOpener under on blob.DefaultMux.

Variables

DefaultIdentity is a Wire provider set that provides an Azure storage account name, key, and SharedKeyCredential from environment variables.

SASTokenIdentity is a Wire provider set that provides an Azure storage account name, SASToken, and anonymous credential from environment variables.

View Source
var Set = wire.NewSet(
	NewPipeline,
	wire.Struct(new(Options), "Credential", "SASToken"),
	wire.Struct(new(URLOpener), "AccountName", "Pipeline", "Options"),
)

Set holds Wire providers for this package.

Functions

func NewCredential

func NewCredential(accountName AccountName, accountKey AccountKey) (*azblob.SharedKeyCredential, error)

NewCredential creates a SharedKeyCredential.

func NewPipeline

func NewPipeline(credential azblob.Credential, opts azblob.PipelineOptions) pipeline.Pipeline

NewPipeline creates a Pipeline for making HTTP requests to Azure.

func OpenBucket

func OpenBucket(ctx context.Context, pipeline pipeline.Pipeline, accountName AccountName, containerName string, opts *Options) (*blob.Bucket, error)

OpenBucket returns a *blob.Bucket backed by Azure Storage Account. See the package documentation for an example and https://godoc.org/github.com/Azure/azure-storage-blob-go/azblob for more details.

Example
package main

import (
	"context"
	"log"

	"github.com/Azure/azure-storage-blob-go/azblob"
	"github.com/kainoaseto/go-cloud/blob/azureblob"
)

func main() {
	// PRAGMA: This example is used on github.com/kainoaseto/go-cloud; PRAGMA comments adjust how it is shown and can be ignored.
	// PRAGMA: On github.com/kainoaseto/go-cloud, hide lines until the next blank line.
	ctx := context.Background()

	const (
		// Fill in with your Azure Storage Account and Access Key.
		accountName azureblob.AccountName = "my-account"
		accountKey  azureblob.AccountKey  = "my-account-key"
		// Fill in with the storage container to access.
		containerName = "my-container"
	)

	// Create a credentials object.
	credential, err := azureblob.NewCredential(accountName, accountKey)
	if err != nil {
		log.Fatal(err)
	}

	// Create a Pipeline, using whatever PipelineOptions you need.
	pipeline := azureblob.NewPipeline(credential, azblob.PipelineOptions{})

	// Create a *blob.Bucket.
	// The credential Option is required if you're going to use blob.SignedURL.
	bucket, err := azureblob.OpenBucket(ctx, pipeline, accountName, containerName,
		&azureblob.Options{Credential: credential})
	if err != nil {
		log.Fatal(err)
	}
	defer bucket.Close()
}
Output:

Example (UsingAADCredentials)
package main

import (
	"context"
	"log"

	"github.com/Azure/azure-storage-blob-go/azblob"
	"github.com/Azure/go-autorest/autorest/azure/auth"
	"github.com/kainoaseto/go-cloud/blob/azureblob"
)

func main() {
	const (
		// Your Azure Storage Account.
		accountName = azureblob.AccountName("my-account")

		// Your Azure AAD Service Principal with access to the storage account.
		// https://docs.microsoft.com/en-us/azure/storage/common/storage-auth-aad-app
		clientID     = "123"
		clientSecret = "456"
		tenantID     = "789"

		// The storage container to access.
		containerName = "my-container"
	)

	// Get an Oauth2 token for the account for use with Azure Storage.
	ccc := auth.NewClientCredentialsConfig(clientID, clientSecret, tenantID)

	// Set the target resource to the Azure storage. This is available as a
	// constant using "azure.PublicCloud.ResourceIdentifiers.Storage".
	ccc.Resource = "https://storage.azure.com/"
	token, err := ccc.ServicePrincipalToken()
	if err != nil {
		log.Fatal(err)
	}

	// Refresh OAuth2 token.
	if err := token.RefreshWithContext(context.Background()); err != nil {
		log.Fatal(err)
	}

	// Create the credential using the OAuth2 token.
	credential := azblob.NewTokenCredential(token.OAuthToken(), nil)

	// Create a Pipeline, using whatever PipelineOptions you need.
	pipeline := azureblob.NewPipeline(credential, azblob.PipelineOptions{})

	// Create a *blob.Bucket.
	// Note that we're not supplying azureblob.Options.Credential, so SignedURL
	// won't work. To use SignedURL, you need a real credential (see the other
	// example).
	ctx := context.Background()
	b, err := azureblob.OpenBucket(ctx, pipeline, accountName, containerName, new(azureblob.Options))
	if err != nil {
		log.Fatal(err)
	}
	defer b.Close()
}
Output:

Example (UsingSASToken)
package main

import (
	"context"
	"log"

	"github.com/Azure/azure-storage-blob-go/azblob"
	"github.com/kainoaseto/go-cloud/blob/azureblob"
)

func main() {
	const (
		// Your Azure Storage Account and SASToken.
		accountName = azureblob.AccountName("my-account")
		sasToken    = azureblob.SASToken("my-SAS-token")
		// The storage container to access.
		containerName = "my-container"
	)

	// Since we're using a SASToken, we can use anonymous credentials.
	credential := azblob.NewAnonymousCredential()

	// Create a Pipeline, using whatever PipelineOptions you need.
	pipeline := azureblob.NewPipeline(credential, azblob.PipelineOptions{})

	// Create a *blob.Bucket.
	// Note that we're not supplying azureblob.Options.Credential, so SignedURL
	// won't work. To use SignedURL, you need a real credential (see the other
	// example).
	ctx := context.Background()
	b, err := azureblob.OpenBucket(ctx, pipeline, accountName, containerName, &azureblob.Options{SASToken: sasToken})
	if err != nil {
		log.Fatal(err)
	}
	defer b.Close()

	// Now we can use b to read or write files to the container.
	data, err := b.ReadAll(ctx, "my-key")
	if err != nil {
		log.Fatal(err)
	}
	_ = data
}
Output:

Types

type AccountKey

type AccountKey string

AccountKey is an Azure storage account key (primary or secondary).

func DefaultAccountKey

func DefaultAccountKey() (AccountKey, error)

DefaultAccountKey loads the Azure storage account key (primary or secondary) from the AZURE_STORAGE_KEY environment variable.

type AccountName

type AccountName string

AccountName is an Azure storage account name.

func DefaultAccountName

func DefaultAccountName() (AccountName, error)

DefaultAccountName loads the Azure storage account name from the AZURE_STORAGE_ACCOUNT environment variable.

type Options

type Options struct {
	// Credential represents the authorizer for SignedURL.
	// Required to use SignedURL.
	Credential azblob.StorageAccountCredential

	// SASToken can be provided along with anonymous credentials to use
	// delegated privileges.
	// See https://docs.microsoft.com/en-us/azure/storage/common/storage-dotnet-shared-access-signature-part-1#shared-access-signature-parameters.
	SASToken SASToken

	// StorageDomain can be provided to specify an Azure Cloud Environment
	// domain to target for the blob storage account (i.e. public, government, china).
	// The default value is "blob.core.windows.net". Possible values will look similar
	// to this but are different for each cloud (i.e. "blob.core.govcloudapi.net" for USGovernment).
	// Check the Azure developer guide for the cloud environment where your bucket resides.
	StorageDomain StorageDomain
}

Options sets options for constructing a *blob.Bucket backed by Azure Block Blob.

type SASToken

type SASToken string

SASToken is an Azure shared access signature. https://docs.microsoft.com/en-us/azure/storage/common/storage-dotnet-shared-access-signature-part-1

func DefaultSASToken

func DefaultSASToken() (SASToken, error)

DefaultSASToken loads a Azure SAS token from the AZURE_STORAGE_SAS_TOKEN environment variable.

type StorageDomain

type StorageDomain string

StorageDomain is an Azure Cloud Environment domain name to target (i.e. blob.core.windows.net, blob.core.govcloudapi.net, blob.core.chinacloudapi.cn). It is read from the AZURE_STORAGE_DOMAIN environment variable.

func DefaultStorageDomain

func DefaultStorageDomain() (StorageDomain, error)

DefaultStorageDomain loads the desired Azure Cloud to target from the AZURE_STORAGE_DOMAIN environment variable.

type URLOpener

type URLOpener struct {
	// AccountName must be specified.
	AccountName AccountName

	// Pipeline must be set to a non-nil value.
	Pipeline pipeline.Pipeline

	// Options specifies the options to pass to OpenBucket.
	Options Options
}

URLOpener opens Azure URLs like "azblob://mybucket".

The URL host is used as the bucket name.

No query parameters are supported.

func (*URLOpener) OpenBucketURL

func (o *URLOpener) OpenBucketURL(ctx context.Context, u *url.URL) (*blob.Bucket, error)

OpenBucketURL opens a blob.Bucket based on u.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL