storage

package module
v1.2.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 30, 2020 License: Apache-2.0 Imports: 5 Imported by: 1

README

storage

Build Status GoDoc Go Report Card codecov License Codacy Badge Join the chat

An application-oriented unified storage layer for Golang.

Goal

  • Production ready
  • High performance
  • Vendor agnostic

Features

Widely services support
Servicer operation support
  • List: list all Storager in service
  • Get: get a Storager via name
  • Create: create a Storager
  • Delete: delete a Storager
Storager operation support

Basic operations

  • Metadata: get storager's metadata
  • Read: read file content
  • Write: write content into file
  • Stat: get file's metadata
  • Delete: delete a file or directory

Extended operations

  • Copy: copy a file inside storager
  • Move: move a file inside storager
  • Reach: generate a public accessible url
  • Statistical: get storage service's statistics

Multiple list style support

  • ListDir: list files and directories under a directory
  • ListPrefix: list files under a prefix

Segment/Multipart support

  • ListPrefixSegment: list segments under a prefix
  • InitIndexSegment: initiate an index type segment
  • WriteIndexSegment: write content into an index type segment
  • CompleteSegment: complete a segment to create a file
  • AbortSegment: abort a segment
File metadata support

Required metadata

  • id: unique key in service
  • name: relative path towards service's work dir
  • size: size of this object
  • updated_at: last update time of this object

Optional metadata

  • content-md5: md5 digest as defined in rfc2616
  • content-type: media type as defined in rfc2616
  • etag: entity tag as defined in rfc2616
  • storage-class: object's storage class as defined in storage proposal

Quick Start

import (
    "log"

    "github.com/Xuanwo/storage"
    "github.com/Xuanwo/storage/coreutils"
    "github.com/Xuanwo/storage/types/pairs"
)

// Init a service.
store, err := coreutils.OpenStorager("fs", pairs.WithWorkDir("/tmp"))
if err != nil {
    log.Fatalf("service init failed: %v", err)
}

// Use Storager API to maintain data.
r, err := store.Read("path/to/file")
if err != nil {
    log.Printf("storager read: %v", err)
}

Documentation

Overview

Package storage intend to provide a unified storage layer for Golang.

Goals

- Production ready: high test coverage, enterprise storage software adaptation, semantic versioning, well documented.

- High performance: more code generation, less runtime reflect.

- Vendor agnostic: more generic abstraction, less internal details.

Details

There two main public interfaces: Servicer and Storager. Storager is a fully functional storage client, and Servicer is a manager of Storager instances, which will be useful for services like object storage. For any service, Storager is required to implement and Servicer is optional.

Examples

The most common case to use a Storager service could be following:

1. Init a service.

    store, err := coreutils.OpenStorager("fs", pairs.WithWorkDir("/tmp"))
	if err != nil {
		log.Fatalf("service init failed: %v", err)
	}

2. Use Storager API to maintain data.

r, err := store.Read("path/to/file")
if err != nil {
	log.Printf("storager read: %v", err)
}

Notes

- Storage uses error wrapping added by go 1.13, go version before 1.13 could be behaved as unexpected.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Copier added in v0.4.0

type Copier interface {

	// Copy will copy an Object or multiple object in the service.
	Copy(src string, dst string, pairs ...*types.Pair) (err error)
	// CopyWithContext will copy an Object or multiple object in the service.
	CopyWithContext(ctx context.Context, src string, dst string, pairs ...*types.Pair) (err error)
}

Copier is the interface for Copy.

type DirLister added in v1.0.0

type DirLister interface {

	// ListDir will return list a specific dir.
	ListDir(dir string, pairs ...*types.Pair) (err error)
	// ListDirWithContext will return list a specific dir.
	ListDirWithContext(ctx context.Context, dir string, pairs ...*types.Pair) (err error)
}

DirLister is used for directory based storage service to list objects under a dir.

type DirSegmentsLister added in v1.0.0

type DirSegmentsLister interface {

	// ListDirSegments will list segments via dir.
	ListDirSegments(dir string, pairs ...*types.Pair) (err error)
	// ListDirSegmentsWithContext will list segments via dir.
	ListDirSegmentsWithContext(ctx context.Context, dir string, pairs ...*types.Pair) (err error)
	// contains filtered or unexported methods
}

DirSegmentsLister is used for directory based storage service to list segments under a dir.

type IndexSegmenter added in v1.1.0

type IndexSegmenter interface {

	// InitIndexSegment will init an index based segment.
	InitIndexSegment(path string, pairs ...*types.Pair) (seg segment.Segment, err error)
	// InitIndexSegmentWithContext will init an index based segment.
	InitIndexSegmentWithContext(ctx context.Context, path string, pairs ...*types.Pair) (seg segment.Segment, err error)

	// WriteIndexSegment will write a part into an index based segment.
	WriteIndexSegment(seg segment.Segment, r io.Reader, index int, size int64, pairs ...*types.Pair) (err error)
	// WriteIndexSegmentWithContext will write a part into an index based segment.
	WriteIndexSegmentWithContext(ctx context.Context, seg segment.Segment, r io.Reader, index int, size int64, pairs ...*types.Pair) (err error)
	// contains filtered or unexported methods
}

IndexSegmenter is the interface for index based segment.

type Mover added in v0.4.0

type Mover interface {

	// Move will move an object in the service.
	Move(src string, dst string, pairs ...*types.Pair) (err error)
	// MoveWithContext will move an object in the service.
	MoveWithContext(ctx context.Context, src string, dst string, pairs ...*types.Pair) (err error)
}

Mover is the interface for Move.

type PrefixLister added in v1.0.0

type PrefixLister interface {

	// ListPrefix will return list a specific dir.
	ListPrefix(prefix string, pairs ...*types.Pair) (err error)
	// ListPrefixWithContext will return list a specific dir.
	ListPrefixWithContext(ctx context.Context, prefix string, pairs ...*types.Pair) (err error)
}

PrefixLister is used for prefix based storage service to list objects under a prefix.

type PrefixSegmentsLister added in v1.0.0

type PrefixSegmentsLister interface {

	// ListPrefixSegments will list segments.
	ListPrefixSegments(prefix string, pairs ...*types.Pair) (err error)
	// ListPrefixSegmentsWithContext will list segments.
	ListPrefixSegmentsWithContext(ctx context.Context, prefix string, pairs ...*types.Pair) (err error)
	// contains filtered or unexported methods
}

PrefixSegmentsLister is used for prefix based storage service to list segments under a prefix.

type Reacher added in v0.4.0

type Reacher interface {

	// Reach will provide a way, which can reach the object.
	Reach(path string, pairs ...*types.Pair) (url string, err error)
	// ReachWithContext will provide a way, which can reach the object.
	ReachWithContext(ctx context.Context, path string, pairs ...*types.Pair) (url string, err error)
}

Reacher is the interface for Reach.

type Servicer

type Servicer interface {
	String() string

	// Create will create a new storager instance.
	Create(name string, pairs ...*types.Pair) (store Storager, err error)
	// CreateWithContext will create a new storager instance.
	CreateWithContext(ctx context.Context, name string, pairs ...*types.Pair) (store Storager, err error)

	// Delete will delete a storager instance.
	Delete(name string, pairs ...*types.Pair) (err error)
	// DeleteWithContext will delete a storager instance.
	DeleteWithContext(ctx context.Context, name string, pairs ...*types.Pair) (err error)

	// Get will get a valid storager instance for service.
	Get(name string, pairs ...*types.Pair) (store Storager, err error)
	// GetWithContext will get a valid storager instance for service.
	GetWithContext(ctx context.Context, name string, pairs ...*types.Pair) (store Storager, err error)

	// List will list all storager instances under this service.
	List(pairs ...*types.Pair) (err error)
	// ListWithContext will list all storager instances under this service.
	ListWithContext(ctx context.Context, pairs ...*types.Pair) (err error)
}

Servicer can maintain multipart storage services.

type Statistician added in v0.4.0

type Statistician interface {

	// Statistical will count service's statistics, such as Size, Count.
	Statistical(pairs ...*types.Pair) (statistic info.StorageStatistic, err error)
	// StatisticalWithContext will count service's statistics, such as Size, Count.
	StatisticalWithContext(ctx context.Context, pairs ...*types.Pair) (statistic info.StorageStatistic, err error)
}

Statistician is the interface for Statistical.

type Storager

type Storager interface {
	String() string

	// Delete will delete an Object from service.
	Delete(path string, pairs ...*types.Pair) (err error)
	// DeleteWithContext will delete an Object from service.
	DeleteWithContext(ctx context.Context, path string, pairs ...*types.Pair) (err error)

	// Metadata will return current storager's metadata.
	Metadata(pairs ...*types.Pair) (meta info.StorageMeta, err error)
	// MetadataWithContext will return current storager's metadata.
	MetadataWithContext(ctx context.Context, pairs ...*types.Pair) (meta info.StorageMeta, err error)

	// Read will read the file's data.
	Read(path string, pairs ...*types.Pair) (rc io.ReadCloser, err error)
	// ReadWithContext will read the file's data.
	ReadWithContext(ctx context.Context, path string, pairs ...*types.Pair) (rc io.ReadCloser, err error)

	// Stat will stat a path to get info of an object.
	Stat(path string, pairs ...*types.Pair) (o *types.Object, err error)
	// StatWithContext will stat a path to get info of an object.
	StatWithContext(ctx context.Context, path string, pairs ...*types.Pair) (o *types.Object, err error)

	// Write will write data into a file.
	Write(path string, r io.Reader, pairs ...*types.Pair) (err error)
	// WriteWithContext will write data into a file.
	WriteWithContext(ctx context.Context, path string, r io.Reader, pairs ...*types.Pair) (err error)
}

Storager is the interface for storage service.

type StoragerFunc added in v0.4.0

type StoragerFunc func(Storager)

StoragerFunc will handle a storager.

Directories

Path Synopsis
Code generated by go generate via internal/cmd/coreutils; DO NOT EDIT.
Code generated by go generate via internal/cmd/coreutils; DO NOT EDIT.
pkg
segment
Package segment provided segment metadata support for storage.
Package segment provided segment metadata support for storage.
azblob
Package azblob provided support for Azure Storage containers and blobs objects (https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) Code generated by go generate via internal/cmd/service; DO NOT EDIT.
Package azblob provided support for Azure Storage containers and blobs objects (https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) Code generated by go generate via internal/cmd/service; DO NOT EDIT.
cos
Package cos provided support for Tencent Cloud's Cloud Object Storage (https://intl.cloud.tencent.com/product/cos) Code generated by go generate via internal/cmd/service; DO NOT EDIT.
Package cos provided support for Tencent Cloud's Cloud Object Storage (https://intl.cloud.tencent.com/product/cos) Code generated by go generate via internal/cmd/service; DO NOT EDIT.
dropbox
Package dropbox provides support for Dropbox (https://www.dropbox.com/).
Package dropbox provides support for Dropbox (https://www.dropbox.com/).
fs
Package fs provided support for local file system.
Package fs provided support for local file system.
gcs
Package gcs provided support for Google Cloud Storage (https://cloud.google.com/storage/) Code generated by go generate via internal/cmd/service; DO NOT EDIT.
Package gcs provided support for Google Cloud Storage (https://cloud.google.com/storage/) Code generated by go generate via internal/cmd/service; DO NOT EDIT.
kodo
Package kodo provided support for qiniu kodo object storage (https://www.qiniu.com/en/products/kodo) Code generated by go generate via internal/cmd/service; DO NOT EDIT.
Package kodo provided support for qiniu kodo object storage (https://www.qiniu.com/en/products/kodo) Code generated by go generate via internal/cmd/service; DO NOT EDIT.
oss
Package oss provided support for Aliyun Object Storage Service (https://cn.aliyun.com/product/oss) Code generated by go generate via internal/cmd/service; DO NOT EDIT.
Package oss provided support for Aliyun Object Storage Service (https://cn.aliyun.com/product/oss) Code generated by go generate via internal/cmd/service; DO NOT EDIT.
qingstor
Package qingstor provided support for qingstor object storage (https://www.qingcloud.com/products/qingstor/) Code generated by go generate via internal/cmd/service; DO NOT EDIT.
Package qingstor provided support for qingstor object storage (https://www.qingcloud.com/products/qingstor/) Code generated by go generate via internal/cmd/service; DO NOT EDIT.
s3
Package s3 provided support for AWS s3 (https://docs.aws.amazon.com/AmazonS3/latest/dev/Welcome.html) Code generated by go generate via internal/cmd/service; DO NOT EDIT.
Package s3 provided support for AWS s3 (https://docs.aws.amazon.com/AmazonS3/latest/dev/Welcome.html) Code generated by go generate via internal/cmd/service; DO NOT EDIT.
uss
Package uss provided support for UPYUN Storage Service (https://www.upyun.com/products/file-storage) Code generated by go generate via internal/cmd/service; DO NOT EDIT.
Package uss provided support for UPYUN Storage Service (https://www.upyun.com/products/file-storage) Code generated by go generate via internal/cmd/service; DO NOT EDIT.
info
Code generated by go generate internal/cmd; DO NOT EDIT.
Code generated by go generate internal/cmd; DO NOT EDIT.
pairs
Package pairs intend to provide all available pairs.
Package pairs intend to provide all available pairs.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL