azureblob

Overview ¶

Package azureblob provides a blob implementation that uses Azure Storage’s BlockBlob. Use OpenBucket to construct a *blob.Bucket.

NOTE: SignedURLs for PUT created with this package are not fully portable; they will not work unless the PUT request includes a "x-ms-blob-type" header set to "BlockBlob". See https://stackoverflow.com/questions/37824136/put-on-sas-blob-url-without-specifying-x-ms-blob-type-header.

URLs ¶

For blob.OpenBucket, azureblob registers for the scheme "azblob".

The default URL opener will use environment variables to generate credentials and a service URL; see https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/storage/azblob for a more complete descriptions of each approach.

• AZURE_STORAGE_ACCOUNT: The service account name. Required if used along with AZURE_STORAGE KEY, because it defines authentication mechanism to be azblob.NewSharedKeyCredential, which creates immutable shared key credentials. Otherwise, "storage_account" in the URL query string parameter can be used.
• AZURE_STORAGE_KEY: To use a shared key credential. The service account name and key are passed to NewSharedKeyCredential and then the resulting credential is passed to NewClientWithSharedKeyCredential.
• AZURE_STORAGE_CONNECTION_STRING: To use a connection string, passed to NewClientFromConnectionString.
• AZURE_STORAGE_SAS_TOKEN: To use a SAS token. The SAS token is added as a URL parameter to the service URL, and passed to NewClientWithNoCredential.
• If none of the above are provided, azureblob defaults to azidentity.NewDefaultAzureCredential: https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azidentity#NewDefaultAzureCredential. See the documentation there for the credential types it supports, including CLI creds, environment variables like AZURE_CLIENT_ID, AZURE_TENANT_ID, etc.

In addition, the environment variables AZURE_STORAGE_ACCOUNT, AZURE_STORAGE_DOMAIN, AZURE_STORAGE_PROTOCOL, AZURE_STORAGE_IS_CDN, and AZURE_STORAGE_IS_LOCAL_EMULATOR can be used to configure how the default URLOpener generates the Azure Service URL via ServiceURLOptions. These can all be configured via URL parameters as well. See ServiceURLOptions and NewDefaultServiceURL for more details.

To customize the URL opener, or for more details on the URL format, see URLOpener.

See https://gocloud.dev/concepts/urls/ for background information.

Escaping ¶

Go CDK supports all UTF-8 strings; to make this work with services lacking full UTF-8 support, strings must be escaped (during writes) and unescaped (during reads). The following escapes are performed for azureblob:

• Blob keys: ASCII characters 0-31, 34 ("\""), 35 ("#"), 37 ("%"), 63 ("?"), 92 ("\"), and 127 are escaped to "__0x<hex>__". Additionally, the "/" in "../" and a trailing "/" in a key (e.g., "foo/") are escaped in the same way.
• Metadata keys: Per https://docs.microsoft.com/en-us/azure/storage/blobs/storage-properties-metadata, Azure only allows C# identifiers as metadata keys. Therefore, characters other than "[a-z][A-z][0-9]_" are escaped using "__0x<hex>__". In addition, characters "[0-9]" are escaped when they start the string. URL encoding would not work since "%" is not valid.
• Metadata values: Escaped using URL encoding.

As ¶

azureblob exposes the following types for As:

• Bucket: *container.Client
• Error: *azcore.ReponseError. You can use bloberror.HasCode directly though.
• ListObject: container.BlobItem for objects, container.BlobPrefix for "directories"
• ListOptions.BeforeList: *container.ListBlobsHierarchyOptions
• Reader: azblobblob.DownloadStreamResponse
• Reader.BeforeRead: *azblob.DownloadStreamOptions
• Attributes: azblobblob.GetPropertiesResponse
• CopyOptions.BeforeCopy: *azblobblob.StartCopyFromURLOptions
• WriterOptions.BeforeWrite: *azblob.UploadStreamOptions
• SignedURLOptions.BeforeSign: *sas.BlobPermissions

package main

import (
	"context"
	"log"

	"gocloud.dev/blob"
)

func main() {
	// PRAGMA: This example is used on gocloud.dev; PRAGMA comments adjust how it is shown and can be ignored.
	// PRAGMA: On gocloud.dev, add a blank import: _ "gocloud.dev/blob/azureblob"
	// PRAGMA: On gocloud.dev, hide lines until the next blank line.
	ctx := context.Background()

	// blob.OpenBucket creates a *blob.Bucket from a URL.
	// This URL will open the container "my-container" using default
	// credentials found in environment variables as documented in
	// the package.
	// Assuming AZURE_STORAGE_ACCOUNT is set to "myaccount",
	// and other options aren't set, the service URL will look like:
	// "https://myaccount.blob.core.windows.net/my-container".
	bucket, err := blob.OpenBucket(ctx, "azblob://my-container")
	if err != nil {
		log.Fatal(err)
	}
	defer bucket.Close()

	// Another example, against a local emulator.
	// Assuming AZURE_STORAGE_ACCOUNT is set to "myaccount",
	// the service URL will look like:
	// "http://localhost:10001/myaccount/my-container".
	localbucket, err := blob.OpenBucket(ctx, "azblob://my-container?protocol=http&domain=localhost:10001")
	if err != nil {
		log.Fatal(err)
	}
	defer localbucket.Close()
}

Output: