fs

package
v0.0.0-...-173ccf6 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 5, 2018 License: GPL-3.0 Imports: 20 Imported by: 0

Documentation

Overview

Package fs implements a very simple and limited file system on top of etcd.

Index

Constants

View Source
const (
	// EtcdTimeout is the timeout to wait before erroring.
	EtcdTimeout = 5 * time.Second // FIXME: chosen arbitrarily
	// DefaultDataPrefix is the default path for data storage in etcd.
	DefaultDataPrefix = "/_etcdfs/data"
	// DefaultHash is the default hashing algorithm to use.
	DefaultHash = "sha256"
	// PathSeparator is the path separator to use on this filesystem.
	PathSeparator = os.PathSeparator // usually the slash character
)

Variables

View Source
var (
	IsPathSeparator = os.IsPathSeparator

	// ErrNotImplemented is returned when something is not implemented by design.
	ErrNotImplemented = errors.New("not implemented")

	// ErrExist is returned when requested path already exists.
	ErrExist = os.ErrExist

	// ErrNotExist is returned when we can't find the requested path.
	ErrNotExist = os.ErrNotExist

	ErrFileClosed   = errors.New("File is closed")
	ErrFileReadOnly = errors.New("File handle is read only")
	ErrOutOfRange   = errors.New("Out of range")
)

TODO: https://dave.cheney.net/2016/04/07/constant-errors

Functions

func NewEtcdFs

func NewEtcdFs(client *etcd.Client, metadata string) afero.Fs

NewEtcdFs creates a new filesystem handle on an etcd client connection. You must specify the metadata string that you wish to use.

func PathSplit

func PathSplit(p string) []string

PathSplit splits a path into an array of tokens excluding any trailing empty tokens.

Types

type File

type File struct {
	Path    string // relative path to file, trailing slash if it's a directory
	Mode    os.FileMode
	ModTime time.Time

	Children []*File // dir's use this
	Hash     string  // string not []byte so it's readable, matches data
	// contains filtered or unexported fields
}

File represents a file node. This is the node of our tree structure. This is not thread safe, and you can have at most one open file handle at a time.

func (*File) Close

func (obj *File) Close() error

Close closes the file handle. This will try and run Sync automatically.

func (*File) Name

func (obj *File) Name() string

Name returns the path of the file.

func (*File) Read

func (obj *File) Read(b []byte) (n int, err error)

Read reads up to len(b) bytes from the File. It returns the number of bytes read and any error encountered. At end of file, Read returns 0, io.EOF. NOTE: This reads into the byte input. It's a side effect!

func (*File) ReadAt

func (obj *File) ReadAt(b []byte, off int64) (n int, err error)

ReadAt reads len(b) bytes from the File starting at byte offset off. It returns the number of bytes read and the error, if any. ReadAt always returns a non-nil error when n < len(b). At end of file, that error is io.EOF.

func (*File) Readdir

func (obj *File) Readdir(count int) ([]os.FileInfo, error)

Readdir lists the contents of the directory and returns a list of file info objects for each entry.

func (*File) Readdirnames

func (obj *File) Readdirnames(n int) (names []string, _ error)

Readdirnames returns a list of name is the current file handle's directory. TODO: this implementation shares the dirCursor with Readdir, is this okay? TODO: should Readdirnames even use a dirCursor at all?

func (*File) Seek

func (obj *File) Seek(offset int64, whence int) (int64, error)

Seek sets the offset for the next Read or Write on file to offset, interpreted according to whence: 0 means relative to the origin of the file, 1 means relative to the current offset, and 2 means relative to the end. It returns the new offset and an error, if any. The behavior of Seek on a file opened with O_APPEND is not specified.

func (*File) Stat

func (obj *File) Stat() (os.FileInfo, error)

Stat returns some information about the file.

func (*File) Sync

func (obj *File) Sync() error

Sync flushes the file contents to the server and calls the filesystem metadata sync as well. FIXME: instead of a txn, run a get and then a put in two separate stages. if the get already found the data up there, then we don't need to push it all in the put phase. with the txn it is always all sent up even if the put is never needed. the get should just be a "key exists" test, and not a download of the whole file. if we *do* do the download, we can byte-by-byte check for hash collisions and panic if we find one :)

func (*File) Truncate

func (obj *File) Truncate(size int64) error

Truncate trims the file to the requested size. Since our file system can only read and write data, but never edit existing data blocks, doing this will not cause more space to be available.

func (*File) Write

func (obj *File) Write(b []byte) (n int, err error)

Write writes to the given file.

func (*File) WriteAt

func (obj *File) WriteAt(b []byte, off int64) (n int, err error)

WriteAt writes into the given file at a certain offset.

func (*File) WriteString

func (obj *File) WriteString(s string) (n int, err error)

WriteString writes a string to the file.

type FileInfo

type FileInfo struct {
	// contains filtered or unexported fields
}

FileInfo is a struct which provides some information about a file handle.

func (*FileInfo) IsDir

func (obj *FileInfo) IsDir() bool

IsDir is an abbreviation for Mode().IsDir().

func (*FileInfo) ModTime

func (obj *FileInfo) ModTime() time.Time

ModTime returns the modification time.

func (*FileInfo) Mode

func (obj *FileInfo) Mode() os.FileMode

Mode returns the file mode bits.

func (*FileInfo) Name

func (obj *FileInfo) Name() string

Name returns the base name of the file.

func (*FileInfo) Size

func (obj *FileInfo) Size() int64

Size returns the length in bytes.

func (*FileInfo) Sys

func (obj *FileInfo) Sys() interface{}

Sys returns the underlying data source (can return nil).

type Fs

type Fs struct {
	Client *etcd.Client

	Metadata string // location of "superblock" for this filesystem

	DataPrefix string // prefix of data storage (no trailing slashes)
	Hash       string // eg: sha256

	Debug bool
	// contains filtered or unexported fields
}

Fs is a specialized afero.Fs implementation for etcd. It implements a small subset of the features, and has some special properties. In particular, file data is stored with it's unique reference being a hash of the data. In this way, you cannot actually edit a file, but rather you create a new one, and update the metadata pointer to point to the new blob. This might seem slow, but it has the unique advantage of being relatively straight forward to implement, and repeated uploads of the same file cost almost nothing. Since etcd isn't meant for large file systems, this fits the desired use case. This implementation is designed to have a single writer for each superblock, but as many readers as you like. FIXME: this is not currently thread-safe, nor is it clear if it needs to be. XXX: we probably aren't updating the modification time everywhere we should! XXX: because we never delete data blocks, we need to occasionally "vacuum". XXX: this is harder because we need to list of *all* metadata paths, if we want them to be able to share storage backends. (we do)

func (*Fs) Chmod

func (obj *Fs) Chmod(name string, mode os.FileMode) error

Chmod changes the mode of a file.

func (*Fs) Chtimes

func (obj *Fs) Chtimes(name string, atime time.Time, mtime time.Time) error

Chtimes changes the access and modification times of the named file, similar to the Unix utime() or utimes() functions. The underlying filesystem may truncate or round the values to a less precise time unit. If there is an error, it will be of type *PathError. FIXME: make sure everything we error is a *PathError TODO: atime is not currently implement and so it is silently ignored.

func (*Fs) Create

func (obj *Fs) Create(name string) (afero.File, error)

Create creates a new file.

func (*Fs) Lstat

func (obj *Fs) Lstat(name string) (os.FileInfo, error)

Lstat does exactly the same as Stat because we currently do not support symbolic links.

func (*Fs) Mkdir

func (obj *Fs) Mkdir(name string, perm os.FileMode) error

Mkdir makes a new directory.

func (*Fs) MkdirAll

func (obj *Fs) MkdirAll(path string, perm os.FileMode) error

MkdirAll creates a directory named path, along with any necessary parents, and returns nil, or else returns an error. The permission bits perm are used for all directories that MkdirAll creates. If path is already a directory, MkdirAll does nothing and returns nil.

func (*Fs) Name

func (obj *Fs) Name() string

Name returns the name of this filesystem.

func (*Fs) Open

func (obj *Fs) Open(name string) (afero.File, error)

Open opens a path. It will be opened read-only.

func (*Fs) OpenFile

func (obj *Fs) OpenFile(name string, flag int, perm os.FileMode) (afero.File, error)

OpenFile opens a path with a particular flag and permission.

func (*Fs) ReadDir

func (obj *Fs) ReadDir(dirname string) ([]os.FileInfo, error)

ReadDir reads the directory named by dirname and returns a list of sorted directory entries.

func (*Fs) ReadFile

func (obj *Fs) ReadFile(filename string) ([]byte, error)

ReadFile reads the file named by filename and returns the contents. A successful call returns err == nil, not err == EOF. Because ReadFile reads the whole file, it does not treat an EOF from Read as an error to be reported.

func (*Fs) Remove

func (obj *Fs) Remove(name string) error

Remove removes a path.

func (*Fs) RemoveAll

func (obj *Fs) RemoveAll(path string) error

RemoveAll removes path and any children it contains. It removes everything it can but returns the first error it encounters. If the path does not exist, RemoveAll returns nil (no error).

func (*Fs) Rename

func (obj *Fs) Rename(oldname, newname string) error

Rename moves or renames a file or directory. TODO: seems it's okay to move files or directories, but you can't clobber dirs but you can clobber single files. a dir can't clobber a file and a file can't clobber a dir. but a file can clobber another file but a dir can't clobber another dir. you can also transplant dirs or files into other dirs.

func (*Fs) Stat

func (obj *Fs) Stat(name string) (os.FileInfo, error)

Stat returns some information about the particular path.

func (*Fs) TempDir

func (obj *Fs) TempDir(dir, prefix string) (name string, err error)

TempDir creates a new temporary directory in the directory dir with a name beginning with prefix and returns the path of the new directory. If dir is the empty string, TempDir uses the default directory for temporary files (see os.TempDir). Multiple programs calling TempDir simultaneously will not choose the same directory. It is the caller's responsibility to remove the directory when no longer needed.

func (*Fs) TempFile

func (obj *Fs) TempFile(dir, prefix string) (f afero.File, err error)

TempFile creates a new temporary file in the directory dir with a name beginning with prefix, opens the file for reading and writing, and returns the resulting *File. If dir is the empty string, TempFile uses the default directory for temporary files (see os.TempDir). Multiple programs calling TempFile simultaneously will not choose the same file. The caller can use f.Name() to find the pathname of the file. It is the caller's responsibility to remove the file when no longer needed.

func (*Fs) URI

func (obj *Fs) URI() string

URI returns a URI representing this particular filesystem.

func (*Fs) Walk

func (obj *Fs) Walk(root string, walkFn filepath.WalkFunc) error

Walk walks the file tree rooted at root, calling walkFn for each file or directory in the tree, including root. All errors that arise visiting files and directories are filtered by walkFn. The files are walked in lexical order, which makes the output deterministic but means that for very large directories Walk can be inefficient. Walk does not follow symbolic links.

func (*Fs) WriteFile

func (obj *Fs) WriteFile(filename string, data []byte, perm os.FileMode) error

WriteFile writes data to a file named by filename. If the file does not exist, WriteFile creates it with permissions perm; otherwise WriteFile truncates it before writing.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL