tree

package
v0.0.0-...-be7858c Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 24, 2017 License: BSD-2-Clause Imports: 13 Imported by: 0

Documentation

Overview

Package tree specifies a directory tree interface, for use by the synchronization algorithm. Many different kinds of backends could be created for this interface, like local filesystem, SFTP, a remote dtsync process over SSH, maybe even MTP.

Index

Constants

This section is empty.

Variables

View Source
var (
	ErrInvalidName   = errors.New("tree: invalid file name")
	ErrInvalidFS     = errors.New("tree: invalid filesystem string")
	ErrInvalidFileId = errors.New("tree: invalid file ID string")
	ErrCancelled     = errors.New("tree: write cancelled")
)

Error codes that can be used by any filesystem implementation.

Functions

func Copy

func Copy(this, other Tree, source, targetParent FileInfo, progress chan int64) (info FileInfo, parentInfo FileInfo, err error)

Copy copies the object indicated by the source to the target in the other tree. It returns the file info of the copied file (which theoretically may be different from the hash in the provided FileInfo) and it's parent. It may also return an error when the source metadata (e.g. modtime) does not match the actual file. Before the actual copy, the fingerprint of the source is compared with the stat results of the source.

func CopyFile

func CopyFile(outf Copier, inf io.Reader, source FileInfo, progress chan int64) (FileInfo, FileInfo, error)

CopyFile acts like io.Copy, but it also hashes the data that passes through. The returned FileInfo has this hash.

func Equal

func Equal(file1, file2 Entry) (bool, error)

Equal compares two entries, only returning true when this file and possible children (for directories) are exactly equal. Do not compare modification time.

func ErrChanged

func ErrChanged(path []string) *pathError

ErrChanged is returned when a file's fingerprint changed between scan and sync.

func ErrChangedHash

func ErrChangedHash(path []string) *pathError

ErrChangedHash is returned when a file's fingerprint changed when the hash turns out to be different during copy.

func ErrFound

func ErrFound(path []string) *pathError

ErrFound is returned when a file is found when none was expected (e.g. on copy or when creating a directory).

func ErrNoDirectory

func ErrNoDirectory(path []string) *pathError

ErrNoDirectory is returned when a directory was expected.

func ErrNoRegular

func ErrNoRegular(path []string) *pathError

ErrNoRegular is returned when a regular file was expected.

func ErrNoSymlink(path []string) *pathError

ErrNoSymlink is returned when a symbolic link was expected.

func ErrNotFound

func ErrNotFound(path []string) *pathError

ErrNotFound is returned when a file is not found (e.g. when trying to read it).

func IsChanged

func IsChanged(err error) bool

IsChanged returns true if (and only if) this is an error caused by a fingerprint that was different from the expected fingerprint.

func IsExist

func IsExist(err error) bool

IsExist returns true if (and only if) this is a 'found' error (e.g. on directory creation). Very similar to os.IsExist.

func IsNoDirectory

func IsNoDirectory(err error) bool

IsNoDirectory returns true if this is an error like ErrNoDirectory.

func IsNoRegular

func IsNoRegular(err error) bool

IsNoRegular returns true if this is an error like ErrNoRegular.

func IsNoSymlink(err error) bool

IsNoSymlink returns true if this is an error like ErrNoSymlink.

func IsNotExist

func IsNotExist(err error) bool

IsNotExist returns true if (and only if) this is a 'not found' error. Very similar to os.IsNotExist.

func MatchFingerprint

func MatchFingerprint(info1, info2 FileInfo) bool

MatchFingerprint returns whether two FileInfos would have the same fingerprint without actually creating the fingerprint.

func NewHash

func NewHash() hash.Hash

NewHash returns the hashing function used by interfaces implementing Entry and by Copy and Update.

func SortEntries

func SortEntries(slice []Entry)

SortEntries sorts the given slice by name.

func TreeTest

func TreeTest(t Tester, fs1, fs2 TestTree)

TreeTest is not a test in itself, it is called by trees wanting themselves to be tested in a generic way.

func Update

func Update(this, other Tree, source, target FileInfo, progress chan int64) (FileInfo, FileInfo, UpdateStats, error)

Update replaces this file with the contents and modtime of the other file. It is very similar to Copy.

Types

type Copier

type Copier interface {
	Write([]byte) (n int, err error)
	Finish() (info FileInfo, parentInfo FileInfo, err error)
	Cancel() error // must not do anything when finished
}

Copier is returned by the Copy and Update methods. Calling Finish() closes the file, possibly moves it to the destination location, and returns the file's and the file parent's FileInfo. Calling Cancel() tries to undo writing the file. It also implements io.Writer.

type Entry

type Entry interface {
	// Name returns the name of this file.
	Name() string
	// RelativePath returns the path relative to the root of this tree.
	RelativePath() []string
	// Type returns the file type (see the Type constants above)
	Type() Type
	// Mode returns the mode (permission) bits of this file.
	Mode() Mode
	// HasMode returns the permission bits the filesystem supports.
	HasMode() Mode
	// ModTime returns the last modification time.
	ModTime() time.Time
	// Size returns the filesize for regular files. For others, it's
	// implementation-dependent.
	Size() int64
	// Id returns the inode and the filesystem for this file. If the inode isn't
	// available, return 0 for it. If the id is nonzero, the filesystem must be
	// non-nil as well.
	Id() (inode uint64, fs *LocalFilesystem)
	// Hash calculates the blake2b hash of the file and returns it.
	Hash() (Hash, error)
	// Info returns a FileInfo without a hash from this Entry.
	// It must be fast (it shouldn't do any I/O).
	Info() FileInfo
	// FullInfo returns a FileInfo with hash from this Entry.
	// The FileInfo comes from Info(), but it has the hash set for regular
	// files.
	FullInfo() (FileInfo, error)
	// Tree returns the tree this entry belongs to.
	Tree() Tree
	// Return a list of children, in alphabetic order.
	// Returns an error when this is not a directory.
	List(ListOptions) ([]Entry, error)
}

Entry is one object tree, e.g. a file or directory. It can also be something else, like bookmarks in a browser, or an email message.

type EntrySlice

type EntrySlice []Entry

EntrySlice is a sortable list of Entries, sorting in incrasing order by the name (Entry.Name()).

func (EntrySlice) Len

func (es EntrySlice) Len() int

func (EntrySlice) Less

func (es EntrySlice) Less(i, j int) bool

func (EntrySlice) Swap

func (es EntrySlice) Swap(i, j int)

type ErrNotImplemented

type ErrNotImplemented string

ErrNotImplemented is returned when a method is not implemented or a parameter is not of a required type.

func (ErrNotImplemented) Error

func (feature ErrNotImplemented) Error() string

type FileInfo

type FileInfo interface {
	// Name returns the name of this file.
	Name() string
	// RelativePath returns the path relative to the root of this tree. The last
	// element is always the same as the name.
	RelativePath() []string
	// Type returns the file type (see the Type constants above)
	Type() Type
	// Mode returns the mode bits (mainly permissions), see os.FileMode.
	Mode() Mode
	// HasMode returns the supported mode bits, 0 for the simplest filesystems
	// (e.g. FAT32).
	HasMode() Mode
	// ModTime returns the last modification time.
	ModTime() time.Time
	// Size returns the filesize for regular files. For others, it's
	// implementation-dependent.
	Size() int64
	// Inode returns the inode number, if available (otherwise 0).
	Inode() uint64
	// Hash returns the blake2b hash of the file, or nil if no hash is known
	// (e.g. for directories).
	Hash() Hash
}

FileInfo is like os.FileInfo, but specific for the Tree interface. It is implemented by dtdiff.Entry.

func NewFileInfo

func NewFileInfo(path []string, fileType Type, mode Mode, hasMode Mode, modTime time.Time, size int64, inode uint64, hash Hash) FileInfo

type FileInfoStruct

type FileInfoStruct struct {
	// contains filtered or unexported fields
}

func (*FileInfoStruct) HasMode

func (fi *FileInfoStruct) HasMode() Mode

func (*FileInfoStruct) Hash

func (fi *FileInfoStruct) Hash() Hash

func (*FileInfoStruct) Inode

func (fi *FileInfoStruct) Inode() uint64

func (*FileInfoStruct) ModTime

func (fi *FileInfoStruct) ModTime() time.Time

func (*FileInfoStruct) Mode

func (fi *FileInfoStruct) Mode() Mode

func (*FileInfoStruct) Name

func (fi *FileInfoStruct) Name() string

func (*FileInfoStruct) RelativePath

func (fi *FileInfoStruct) RelativePath() []string

func (*FileInfoStruct) Size

func (fi *FileInfoStruct) Size() int64

func (*FileInfoStruct) String

func (fi *FileInfoStruct) String() string

func (*FileInfoStruct) Type

func (fi *FileInfoStruct) Type() Type

type FileTree

type FileTree interface {
	Tree

	// Open a file for writing with the specified name in the parent, with the
	// metadata as specified in 'source'. Calling Finish() should create the
	// file atomically at the target location. For example, by writing to it at
	// a temporary location and moving it to the destination.
	// Implementations should check that the target doesn't yet exist.
	CreateFile(name string, parent, source FileInfo) (Copier, error)

	// Update this file. May do about the same as CreateFile.
	UpdateFile(file, source FileInfo) (Copier, error)

	// Create a symbolic link.
	CreateSymlink(name string, parent, source FileInfo, contents string) (FileInfo, FileInfo, error)

	// Update a symbolic link.
	UpdateSymlink(file, source FileInfo, contents string) (FileInfo, FileInfo, error)

	// Read the contents (target path) of a symbolic link.
	ReadSymlink(file FileInfo) (string, error)

	// CopySource returns a io.ReadCloser, but checks first whether the FileInfo
	// matches.
	CopySource(info FileInfo) (io.ReadCloser, error)
}

FileTree is a filesystem (like) tree, with normal mkdir and open calls, and objects containing data blobs.

type Filesystem

type Filesystem int64

type Hash

type Hash struct {
	Type HashType
	Data []byte
}

Hash contains a hash type (hash/blake2b, or symlink target), and the contents of the hash (32-byte buffer or variable-width string).

func (Hash) Equal

func (h Hash) Equal(h2 Hash) bool

func (Hash) IsZero

func (h Hash) IsZero() bool

type HashType

type HashType int
const (
	HASH_NONE    HashType = 0 // no hash (nil)
	HASH_DEFAULT HashType = 1 // the hash that is set with the Hash header
	HASH_TARGET  HashType = 2 // path of the symlink (via readlink())
)

type ListOptions

type ListOptions struct {
	Follow func([]string) bool
}

type LocalFileTree

type LocalFileTree interface {
	LocalTree
	FileTree

	// UpdateRsync returns a source file and a file to write, for use by the
	// rsync algorithm.
	UpdateRsync(info, source FileInfo) (RsyncBasis, Copier, error)

	// Get this file. This only exists to read the status file, not to implement
	// copying in the syncer!
	GetFile(name string) (io.ReadCloser, error)

	// PutFile is analogous to GetFile.
	PutFile(name string) (Copier, error)
}

type LocalFilesystem

type LocalFilesystem struct {
	Type     string
	DeviceId uint64
}

type LocalTree

type LocalTree interface {
	// Root returns the root entry of this tree.
	Root() Entry
}

type Mode

type Mode uint32

The mode (mainly Unix-like permissions) used in FileInfo.Mode().

func (Mode) Calc

func (m Mode) Calc(sourceHasMode, targetDefault Mode) Mode

Calculate new mode bits from the source filesystem and the default permissions on the target filesystem.

type PathError

type PathError interface {
	error
	Path() string
}

type RemoteTree

type RemoteTree interface {
	Tree

	// RemoteScan issues a scan on the other end, and returns a reader for the
	// newly created .dtsync file.
	//
	// Implementations may either scan the whole tree and return once finished,
	// or start scanning while returning the data as far as they've scanned.
	//
	// The sendOptions are sent to the remote scanner (scan will start once
	// received), and recvOptions is a channel from which the options sent by
	// the remote can be read.
	RemoteScan(extraOptions *ScanOptions, sendOptions, recvOptions chan *ScanOptions, progress chan<- *ScanProgress, cancel chan struct{}) (io.ReadCloser, error)

	// SendStatus is used to overwrite the status file on the remote. Data (in
	// e.g. protobuf format) can be written to the Copier.
	SendStatus() (Copier, error)

	// Use the rsync algorithm (librsync) to send only small changes in files.
	// RsyncSrc sends a signature and receives the binary delta.
	RsyncSrc(file FileInfo, signature io.Reader) (delta io.ReadCloser, err error)
	// RsyncDst receives a signature and sends the binary delta.
	RsyncDst(file, source FileInfo) (signature io.Reader, delta Copier, err error)
}

type RsyncBasis

type RsyncBasis interface {
	io.Reader
	io.ReaderAt
	io.Closer
}

type ScanOptions

type ScanOptions struct {
	Exclude []string
	Include []string
	Follow  []string
	Perms   Mode
	Replica string
}

ScanOptions holds some options to send to the other replica.

func (*ScanOptions) Add

func (o *ScanOptions) Add(options *ScanOptions)

type ScanProgress

type ScanProgress struct {
	Total uint64
	Done  uint64
	Path  []string
}

ScanProgress holds the current progress (total estimated number and current position) to send from the scanner to the UI.

func (*ScanProgress) After

func (p1 *ScanProgress) After(p2 *ScanProgress) bool

func (*ScanProgress) Percent

func (p *ScanProgress) Percent() float64

type TestTree

type TestTree interface {
	FileTree

	// ReadInfo returns the FileInfo (with hash) for a particular path.
	ReadInfo(path []string) (FileInfo, error)

	// PutFileTest sets the file at the path to the specified contents. The
	// FileInfo returned does not have to contain the hash.
	PutFileTest(path []string, contents []byte) (FileInfo, error)
}

type Tester

type Tester interface {
	Error(...interface{})
	Errorf(string, ...interface{})
	Fatal(...interface{})
	Fatalf(string, ...interface{})
	FailNow()
}

Tester is a helper interface, abstracting away *testing.T. This is useful as we don't have to import package "testing" this way, which would pollute the "flag" package with it's flags.

type Tree

type Tree interface {
	// Close clears all resources allocated for this tree, if any. For example,
	// it closes a network connection.
	Close() error

	// mkdir: create a directory in this directory with the given name. The
	// modification time is unknown and will likely be the current time.
	CreateDir(name string, parent, source FileInfo) (FileInfo, error) // TODO update parent mtime
	// Remove removes the indicated file, but checks the fingerprint first. It
	// returns ErrChanged if the metadata does not match, or another error if
	// the remove failed.
	Remove(file FileInfo) (parentInfo FileInfo, err error)
	// Chmod updates the mode (permission) bits of a file. The newly returned
	// FileInfo contains the new mode.
	Chmod(target, source FileInfo) (FileInfo, error)
}

Tree is an abstraction layer over various types of trees. It tries to be as generic as possible, making it possible to synchronize varying types of trees (filesystem, sftp, remote filesystem, mtp, etc.). It may even be possible to use it for very different trees, e.g. browser bookmark trees.

type Type

type Type int

The type used for TYPE_* constants

const (
	TYPE_UNKNOWN   Type = 0
	TYPE_REGULAR   Type = 1
	TYPE_DIRECTORY Type = 2
	TYPE_SYMLINK   Type = 3
	TYPE_NOTFOUND  Type = 4
)

Constants used as Type() return values. These must be kept stable.

See also: tree/remote/messages.proto

func (Type) Char

func (t Type) Char() string

type UpdateStats

type UpdateStats struct {
	ToSource int64
	ToTarget int64
}

Directories

Path Synopsis
Package file implements the file tree interface (tree.Entry) for local filesystems.
Package file implements the file tree interface (tree.Entry) for local filesystems.
Package memory implements the file tree interface (tree.Entry).
Package memory implements the file tree interface (tree.Entry).
Package remote implements the client and server side of a scan over a networ.
Package remote implements the client and server side of a scan over a networ.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL