vfs

package module
v1.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 23, 2018 License: MIT Imports: 8 Imported by: 0

README

vfs - Virtual File System

Go library to generalize commands and behavior when interacting with various file systems.

The vfs library includes interfaces which allow you to interact with files and locations on various file systems in a generic way. Currently supported file systems:

  • Local fs (Windows, OS X, Linux)
  • Amazon S3
  • GCS

These interfaces are composed of standard Go library interfaces, allowing for simple file manipulation within, and between the supported file systems.

At C2FO we have created a factory system that is integrated with our app configuration that allows for simply initializing the various locations we tend to do file work in. You can build your own similar system directly on top of the various file system implementations and the provided generic interfaces, or you can use the simple interface included in the vfs package. The usage examples below will detail this simple interface. We will eventually be providing a version of our factory as an example of how this library can be used in a more complex project.

A couple notes on configuration for this interface (vfssimple.NewFile and vfssimple.NewLocation):

  • Before calling either function you must initialize any file systems you expect to be using.
  • Local: The local file system requires no configuration. Simply call vfssimple.InitializeLocalFileSystem so the internals are prepared to expect "file:///" URIs.
  • S3: The vfssimple.InitializeS3FileSystem() method requires authentication parameters for the user, see godoc for this function.
  • GCS: In addition to calling vfssimple.InitializeGSFileSystem, you are expected to have authenticated with GCS using the Google Cloud Shell for the user running the app. We will be looking into more flexible forms of authentication (similar to the S3 library) in the future, but this was an ideal use case for us to start with, and therefore, all that is currently provided.

Installation

OS X, Linux, and Windows:

glide install github.com/c2fo/vfs

Usage example

import "github.com/c2fo/vfs/vfssimple"

// The following functions tell vfssimple we expect to handle a particular file system in subsequent calls to
// vfssimple.NewFile() and vfssimple.NewLocation
// Local files, ie: "file:///"
vfssimple.InitializeLocalFileSystem()

// Google Cloud Storage, ie: "gs://"
vfs.InitializeGSFileSystem()

// Amazon S3, ie: "s3://"
vfssimple.InitializeS3FileSystem(accessKeyId, secreteAccessKey, token)

// alternative to above for S3, if you've already initialized a client of interface s3iface.S3API
vfssimple.SetS3Client(client)

You can then use those file systems to initialize locations which you'll be referencing frequently, or initialize files directly

osFile, err := vfssimple.NewFile("file:///path/to/file.txt")
s3File, err := vfssimple.NewFile("s3://bucket/prefix/file.txt")

osLocation, err := vfssimple.NewLocation("file:///tmp")
s3Location, err := vfssimple.NewLocation("s3://bucket")

osTmpFile, err := osLocation.NewFile("anotherFile.txt") // file at /tmp/anotherFile.txt

With a number of files and locations between s3 and the local file system you can perform a number of actions without any consideration for the system's api or implementation details.

osFileExists, err := osFile.Exists() // true, nil
s3FileExists, err := s3File.Exists() // false, nil
err = osFile.CopyToFile(s3File) // nil
s3FileExists, err = s3File.Exists() // true, nil

movedOsFile, err := osFile.MoveToLocation(osLocation)
osFileExists, err = osFile.Exists() // false, nil (move actions delete the original file)
movedOsFileExists, err := movedOsFile.Exists() // true, nil

s3FileUri := s3File.URI() // s3://bucket/prefix/file.txt
s3FileName := s3File.Name() // file.txt
s3FilePath := s3File.Path() // /prefix/file.txt

// vfs.File and vfs.Location implement fmt.Stringer, returning x.URI()
fmt.Sprintf("Working on file: %s", s3File) // "Working on file: s3://bucket/prefix/file.txt"

Development setup

Fork the project and clone it locally, then in the cloned directory...

glide install
go test $(glide novendor)

Release History

  • 0.1.0
    • The first release
    • Support for local file system, s3, and gcs
    • Initial README.md
  • 1.0.0
    • Apply last of bugfixes from old repo
  • 1.1.0
    • Enable server-side encryption on S3 (matching GCS) as a more sane, secure default for files is at rest
  • 1.2.0
    • For the S3 implementation of the File interface, ensure the file exists in S3 after it is written before continuing.

Meta

Brought to you by the Enterprise Pipeline team at C2FO:

John Judd - john.judd@c2fo.com

Jason Coble - @jasonkcoble - jason@c2fo.com

Chris Roush – chris.roush@c2fo.com

Distributed under the MIT license. See LICENSE for more information.

https://github.com/c2fo/

Contributing

  1. Fork it (https://github.com/c2fo/vfs/fork)
  2. Create your feature branch (git checkout -b feature/fooBar)
  3. Commit your changes (git commit -am 'Add some fooBar')
  4. Push to the branch (git push origin feature/fooBar)
  5. Create a new Pull Request

Documentation

Overview

This file contains unified error handling for vfs as well as a technique for handling multiple errors in the event of deferred method calls such as file.Close()

Package vfs provides a platform-independent interface to generalized set of filesystem functionality across a number of filesystem types such as os, S3, and GCS.

Index

Examples

Constants

View Source
const (
	Windows       = "windows"
	BadFilePrefix = "expecting only a filename prefix, which may not include slashes or backslashes"
)

Variables

This section is empty.

Functions

func AddTrailingSlash

func AddTrailingSlash(path string) string

AddTrailingSlash is a helper function accepts a path string and returns the path string with a trailing slash if there wasn't one.

func CleanPrefix

func CleanPrefix(prefix string) string

CleanPrefix resolves relative dot pathing, removing any leading . or / and removes any trailing /

func EnsureTrailingSlash

func EnsureTrailingSlash(dir string) string

EnsureTrailingSlash is like AddTrailingSlash but will only ever use / since it's use for web uri's, never an Windows OS path.

func GetFileURI

func GetFileURI(f File) string

GetFile returns a File URI

func GetLocationURI

func GetLocationURI(l Location) string

GetFile returns a Location URI

func StandardizePath

func StandardizePath(path string) string

func TouchCopy

func TouchCopy(writer File, reader File) error

TouchCopy is a wrapper around io.Copy which ensures that even empty source files (reader) will get written as an empty file. It guarantees a Write() call on the target file.

func ValidateFilePrefix

func ValidateFilePrefix(filenamePrefix string) error

Performs a validation check on a prefix. The prefix should not include "/" or "\\" characters. An error is returned if either of those conditions are true.

Types

type File

type File interface {
	io.Closer
	io.Reader
	io.Seeker
	io.Writer
	fmt.Stringer

	// Exists returns boolean if the file exists on the file system.  Also returns an error if any.
	Exists() (bool, error)

	// Location returns the vfs.Location for the File.
	Location() Location

	// CopyToLocation will copy the current file to the provided location. If the file already exists at the location,
	// the contents will be overwritten with the current file's contents. In the case of an error, nil is returned
	// for the file.
	CopyToLocation(location Location) (File, error)

	// CopyToFile will copy the current file to the provided file instance. If the file already exists,
	// the contents will be overwritten with the current file's contents. In the case of an error, nil is returned
	// for the file.
	CopyToFile(File) error

	// MoveToLocation will move the current file to the provided location. If the file already exists at the location,
	// the contents will be overwritten with the current file's contents. In the case of an error, nil is returned
	// for the file.
	MoveToLocation(location Location) (File, error)

	// MoveToFile will move the current file to the provided file instance. If a file with the current file's name already exists,
	// the contents will be overwritten with the current file's contents. The current instance of the file will be removed.
	MoveToFile(File) error

	// Delete unlinks the File on the filesystem.
	Delete() error

	// LastModified returns the timestamp the file was last modified (as *time.Time).
	LastModified() (*time.Time, error)

	// Size returns the size of the file in bytes.
	Size() (uint64, error)

	// Path returns absolute path (with leading slash) including filename, ie /some/path/to/file.txt
	Path() string

	// Name returns the base name of the file path.  For file:///some/path/to/file.txt, it would return file.txt
	Name() string

	// URI returns the fully qualified URI for the File.  IE, s3://bucket/some/path/to/file.txt
	URI() string
}

File represents a file on a filesystem. A File may or may not actually exist on the filesystem.

type FileSystem

type FileSystem interface {
	// NewFile initializes a File on the specified volume at path 'name'. On error, nil is returned
	// for the file.
	NewFile(volume string, name string) (File, error)

	// NewLocation initializes a Location on the specified volume with the given path. On error, nil is returned
	// for the location.
	NewLocation(volume string, path string) (Location, error)

	// Name returns the name of the FileSystem ie: s3, disk, gcs, etc...
	Name() string

	// Scheme, related to Name, is the uri scheme used by the FileSystem: s3, file, gs, etc...
	Scheme() string
}

FileSystem represents a filesystem with any authentication accounted for.

type Location

type Location interface {
	fmt.Stringer

	// List returns a slice of strings representing the base names of the files found at the Location. All implementations
	// are expected to return ([]string{}, nil) in the case of a non-existent directory/prefix/location. If the user
	// cares about the distinction between an empty location and a non-existent one, Location.Exists() should be checked
	// first.
	List() ([]string, error)

	// ListByPrefix returns a slice of strings representing the base names of the files found in Location whose
	// filenames match the given prefix. An empty slice will be returned even for locations that don't exist.
	ListByPrefix(prefix string) ([]string, error)

	// ListByRegex returns a slice of strings representing the base names of the files found in Location that
	// matched the given regular expression. An empty slice will be returned even for locations that don't exist.
	ListByRegex(regex *regexp.Regexp) ([]string, error)

	// Returns the volume as string.  Some filesystems may not have a volume and will return "".  In URI parlance,
	// volume equates to authority.  For example s3://mybucket/path/to/file.txt, volume would return "mybucket".
	Volume() string

	//Path returns absolute path to the Location with leading and trailing slashes, ie /some/path/to/
	Path() string

	// Exists returns boolean if the file exists on the file system. Also returns an error if any.
	Exists() (bool, error)

	// NewLocation is an initializer for a new Location relative to the existing one. For instance, for location:
	// file://some/path/to/, calling NewLocation("../../") would return a new vfs.Location representing file://some/.
	// The new location instance should be on the same file system volume as the location it originated from.
	NewLocation(relativePath string) (Location, error)

	// ChangeDir updates the existing Location's path to the provided relative path. For instance, for location:
	// file://some/path/to/, calling ChangeDir("../../") update the location instance to file://some/.
	ChangeDir(relativePath string) error

	//FileSystem returns the underlying vfs.FileSystem struct for Location.
	FileSystem() FileSystem

	// NewFile will instantiate a vfs.File instance at the current location's path. In the case of an error,
	// nil will be returned.
	NewFile(fileName string) (File, error)

	// DeleteFile deletes the file of the given name at the location. This is meant to be a short cut for
	// instantiating a new file and calling delete on that, with all the necessary error handling overhead.
	DeleteFile(fileName string) error

	// URI returns the fully qualified URI for the Location.  IE, file://bucket/some/path/
	URI() string
}

Location represents a filesystem path which serves as a start point for directory-like functionality. A location may or may not actually exist on the filesystem.

type MultiErr

type MultiErr struct {
	// contains filtered or unexported fields
}

MultiErr provides a set of functions to handle the scenario where, because of errors in defers, we have a way to handle the potenetial of multiple errors. For instance, if you do a open a file, defer it's close, then fail to Seek. The seek fauilure has one error but then the Close fails as well. This ensure neither are ignored.

func NewMutliErr

func NewMutliErr() *MultiErr

Constructor for generating a zero-value MultiErr reference.

func (*MultiErr) Append

func (me *MultiErr) Append(errs ...error) error

Appends the provided errors to the errs slice for future message reporting.

func (*MultiErr) DeferFunc

func (me *MultiErr) DeferFunc(f singleErrReturn)
Example
// NOTE: We use a named error in the function since our first defer will set it based on any appended errors
_ = func(f File) (rerr error) {
	//THESE LINES REQUIRED
	errs := NewMutliErr()
	defer func() { rerr = errs.OrNil() }()

	_, err := f.Read(nil)
	if err != nil {
		//for REGULAR ERROR RETURNS we just return the Appended errors
		return errs.Append(err)
	}

	// for defers, use DeferFunc and pass it the func name
	defer errs.DeferFunc(f.Close)

	_, err = f.Seek(0, 0)
	if err != nil {
		//for REGULAR ERROR RETURNS we just return the Appended errors
		return errs.Append(err)
	}

	return nil
}
Output:

func (*MultiErr) Error

func (me *MultiErr) Error() string

Returns the error message string.

func (*MultiErr) OrNil

func (me *MultiErr) OrNil() error

If there are no errors in the MultErr instance, then return nil, otherwise return the full MultiErr instance.

Directories

Path Synopsis
Google Cloud Storage VFS implementation.
Google Cloud Storage VFS implementation.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL