storagesystem

package
v0.5.14 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 29, 2024 License: Apache-2.0, MIT Imports: 64 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var BackendMap = make(map[string]Backend)
View Source 
var Backends []Backend
View Source 
var ErrBackendNotSupported = errors.New("This backend is not supported")
View Source 
var ErrGetUsageNotSupported = errors.New("The backend does not support getting usage quota")
View Source 
var ErrMoveNotSupported = errors.New("The backend does not support moving files")
View Source 
var ErrStorageNotAvailable = errors.New("storage not available")

Functions

func GetHash 

func GetHash(ctx context.Context, object fs.ObjectInfo) (string, error)

GetHash computes the hash value of a given object based on the supported hash type by its filesystem. If the filesystem's hash computation is slow or if there is any error during hash computation, the function returns an empty string or an error respectively.

Parameters:

  • ctx: A context to allow for timeout or cancellation of operations.
  • object: An fs.ObjectInfo representing the object for which the hash needs to be computed.

Returns:

  • The computed hash string of the object. If the filesystem's hash computation is considered slow or if there is an error in hash computation, an empty string is returned.
  • An error if the function encounters any issues during its operation, such as failure in hash computation. If no error occurs, the error is nil.

func IsSameEntry 

func IsSameEntry(ctx context.Context, file model.File, object fs.ObjectInfo) (bool, string)

IsSameEntry checks if a given model.File and a given fs.Object represent the same entry, based on their size, hash, and last modification time.

Parameters:

  • ctx: Context that allows for asynchronous task cancellation.
  • file: A model.File instance representing a file in the model.
  • object: An fs.Object instance representing a file or object in the filesystem.

Returns:

  • bool: A boolean indicating whether the given model.File and fs.Object are considered the same entry.
  • string: A string providing details in case of a mismatch.

The function performs the following checks:

  1. Compares the sizes of 'file' and 'object'. If there is a mismatch, it returns false along with a formatted string that shows the mismatched sizes.
  2. Retrieves the last modified time of 'object'.
  3. Identifies a supported hash type for the storage backend of 'object' and computes its hash value.
  4. The hash computation is skipped if the storage backend is a local file system or does not support any hash types.
  5. If both 'file' and 'object' have non-empty hash values and these values do not match, it returns false along with a formatted string that shows the mismatched hash values.
  6. Compares the last modified times of 'file' and 'object' at the nanosecond precision. If there is a mismatch, it returns false along with a formatted string that shows the mismatched times.
  7. If all the checks pass (sizes, hashes, and last modified times match), the function returns true, indicating that 'file' and 'object' are considered to be the same entry.

Note:

  • In certain cases (e.g., failures during fetch), the last modified time might not be reliable.
  • For local file systems, hash computation is skipped to avoid inefficient operations.

func Open added in v0.5.0 

func Open(h Handler, ctx context.Context, path string) (io.ReadSeekCloser, fs.DirEntry, error)

Open offers a file handler like interface (io.ReadSeekCloser) to an RClone path

Types

type Backend 

type Backend struct {
	// Name of this fs
	Name string
	// Description of this fs - defaults to Name
	Description string
	// Prefix for command line flags for this fs - defaults to Name if not set
	Prefix string
	// Different Config by Provider
	ProviderOptions []ProviderOptions
}

type EmptyReadCloser 

type EmptyReadCloser struct{}

EmptyReadCloser is a ReadCloser that always returns EOF.

func (*EmptyReadCloser) Close 

func (e *EmptyReadCloser) Close() error

func (*EmptyReadCloser) Read 

func (e *EmptyReadCloser) Read(p []byte) (n int, err error)

type Entry 

type Entry struct {
	Error error
	Info  fs.Object
	Dir   fs.Directory
}

Entry is a struct that represents a single file or directory during a data source scan.

type Handler 

type Handler interface {
	Reader
	Writer
	Lister
	fs.Abouter
}

Handler is an interface for all relevant operations allowed by an RClone backend.

type Lister 

type Lister interface {
	Name
	// List lists the files at the given path.
	List(ctx context.Context, path string) ([]fs.DirEntry, error)

	// Scan scans the data source starting at the given path and returns a channel of entries.
	// The `last` parameter is used to resume scanning from the last entry returned by a previous scan. It is exclusive.
	// The returned entries must be sorted by path in ascending order.
	Scan(ctx context.Context, path string) <-chan Entry

	// Check checks the size and last modified time of the file at the given path.
	Check(ctx context.Context, path string) (fs.DirEntry, error)
}

type Name 

type Name interface {
	Name() string
}

type Option 

type Option fs.Option

func (*Option) ToCLIFlag 

func (option *Option) ToCLIFlag(prefix string, useBuiltIn bool, category string) cli.Flag

type ProviderOptions 

type ProviderOptions struct {
	Provider            string
	ProviderDescription string
	Options             []Option
}

func (ProviderOptions) ToCLICommand 

func (p ProviderOptions) ToCLICommand(short string, long string, description string) *cli.Command

type RCloneHandler 

type RCloneHandler struct {
	// contains filtered or unexported fields
}

func NewRCloneHandler 

func NewRCloneHandler(ctx context.Context, s model.Storage) (*RCloneHandler, error)

func (RCloneHandler) About 

func (h RCloneHandler) About(ctx context.Context) (*fs.Usage, error)

func (RCloneHandler) Check 

func (h RCloneHandler) Check(ctx context.Context, path string) (fs.DirEntry, error)

func (RCloneHandler) List 

func (h RCloneHandler) List(ctx context.Context, path string) ([]fs.DirEntry, error)

func (RCloneHandler) Move 

func (h RCloneHandler) Move(ctx context.Context, from fs.Object, to string) (fs.Object, error)

func (RCloneHandler) Name 

func (h RCloneHandler) Name() string

func (RCloneHandler) Read 

func (h RCloneHandler) Read(ctx context.Context, path string, offset int64, length int64) (io.ReadCloser, fs.Object, error)

func (RCloneHandler) Remove 

func (h RCloneHandler) Remove(ctx context.Context, obj fs.Object) error

func (RCloneHandler) Scan 

func (h RCloneHandler) Scan(ctx context.Context, path string) <-chan Entry

func (RCloneHandler) Write 

func (h RCloneHandler) Write(ctx context.Context, path string, in io.Reader) (fs.Object, error)

type Reader 

type Reader interface {
	Name
	// Read reads data from the source storage file starting at the given path and offset, and returns a ReadCloser.
	// The `length` parameter specifies the number of bytes to read.
	// This method is most likely used for retrieving a single block of data.
	Read(ctx context.Context, path string, offset int64, length int64) (io.ReadCloser, fs.Object, error)
}

type Writer 

type Writer interface {
	Name
	// Write writes data to the output storage file.
	Write(ctx context.Context, path string, in io.Reader) (fs.Object, error)
	// Move moves the given object to the given path.
	Move(ctx context.Context, from fs.Object, to string) (fs.Object, error)
	// Remove removes the given object.
	Remove(ctx context.Context, obj fs.Object) error
}

func GetRandomOutputWriter 

func GetRandomOutputWriter(ctx context.Context, storages []model.Storage) (*model.StorageID, Writer, error)

GetRandomOutputWriter selects a storage from the provided storages list based on its available space and returns an associated Writer to interact with that storage.

The function works as follows:

  1. Iterates over each storage to obtain its usage information (free and total space).
  2. Calculates a weight for each storage based on its free space ratio.
  3. If the available space of a storage is below a threshold, it will not be considered for selection.
  4. Selects a storage based on its weight and returns the Writer for it.

Parameters:

  • ctx: A context that allows for timeout or cancellation of operations.
  • storages: A slice of storage models to choose from.

Returns:

  • A pointer to the ID of the selected storage.
  • A Writer object that can be used to write data to the selected storage.
  • An error if the function encounters any issues during its operation or if no suitable storage is found. If all storages are full, it returns the ErrStorageNotAvailable error.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.