repository

package
v0.9.7 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 25, 2020 License: BSD-2-Clause Imports: 28 Imported by: 0

Documentation

Overview

Package repository implements a restic repository on top of a backend. In the following the abstractions used for this package are listed. More information can be found in the restic design document.

File

A file is a named handle for some data saved in the backend. For the local backend, this corresponds to actual files saved to disk. Usually, the SHA256 hash of the content is used for a file's name (hexadecimal, in lower-case ASCII characters). An exception is the file `config`. Most files are encrypted before being saved in a backend. This means that the name is the hash of the ciphertext.

Blob

A blob is a number of bytes that has a type (data or tree). Blobs are identified by an ID, which is the SHA256 hash of the blobs' contents. One or more blobs are bundled together in a Pack and then saved to the backend. Blobs are always encrypted before being bundled in a Pack.

Pack

A Pack is a File in the backend that contains one or more (encrypted) blobs, followed by a header at the end of the Pack. The header is encrypted and contains the ID, type, length and offset for each blob contained in the Pack.

Index

Constants

This section is empty.

Variables

View Source 
var (
	// ErrNoKeyFound is returned when no key for the repository could be decrypted.
	ErrNoKeyFound = errors.New("wrong password or no key found")

	// ErrMaxKeysReached is returned when the maximum number of keys was checked and no key could be found.
	ErrMaxKeysReached = errors.Fatal("maximum number of keys reached")
)
View Source 
var (
	// KDFTimeout specifies the maximum runtime for the KDF.
	KDFTimeout = 500 * time.Millisecond

	// KDFMemory limits the memory the KDF is allowed to use.
	KDFMemory = 60
)
View Source 
var ErrOldIndexFormat = errors.New("index has old format")

ErrOldIndexFormat means an index with the old format was detected.

View Source 
var IndexFull = func(idx *Index) bool {
	idx.m.Lock()
	defer idx.m.Unlock()

	debug.Log("checking whether index %p is full", idx)

	var blobs uint
	for typ := range idx.byType {
		blobs += idx.byType[typ].len()
	}
	age := time.Now().Sub(idx.created)

	switch {
	case age >= indexMaxAge:
		debug.Log("index %p is old enough", idx, age)
		return true
	case blobs >= indexMaxBlobs:
		debug.Log("index %p has %d blobs", idx, blobs)
		return true
	}

	debug.Log("index %p only has %d blobs and is too young (%v)", idx, blobs, age)
	return false

}

IndexFull returns true iff the index is "full enough" to be saved as a preliminary index.

View Source 
var Params *crypto.Params

Params tracks the parameters used for the KDF. If not set, it will be calibrated on the first run of AddKey().

Functions

func DownloadAndHash added in v0.8.3 

func DownloadAndHash(ctx context.Context, be Loader, h restic.Handle) (tmpfile *os.File, hash restic.ID, size int64, err error)

DownloadAndHash is all-in-one helper to download content of the file at h to a temporary filesystem location and calculate ID of the contents. Returned (temporary) file is positioned at the beginning of the file; it is reponsibility of the caller to close and delete the file.

func Repack 

func Repack(ctx context.Context, repo restic.Repository, packs restic.IDSet, keepBlobs restic.BlobSet, p *restic.Progress) (obsoletePacks restic.IDSet, err error)

Repack takes a list of packs together with a list of blobs contained in these packs. Each pack is loaded and the blobs listed in keepBlobs is saved into a new pack. Returned is the list of obsolete packs which can then be removed.

func RunWorkers added in v0.9.7 

func RunWorkers(ctx context.Context, count int, workerFunc func() error, finalFunc func()) error

RunWorkers runs count instances of workerFunc using an errgroup.Group. After all workers have terminated, finalFunc is run. If an error occurs in one of the workers, it is returned. FinalFunc is always run, regardless of any other previous errors.

func SaveIndex 

func SaveIndex(ctx context.Context, repo restic.Repository, index *Index) (restic.ID, error)

SaveIndex saves an index in the repository.

func TestBackend 

func TestBackend(t testing.TB) (be restic.Backend, cleanup func())

TestBackend returns a fully configured in-memory backend.

func TestOpenLocal 

func TestOpenLocal(t testing.TB, dir string) (r restic.Repository)

TestOpenLocal opens a local repository.

func TestRepository 

func TestRepository(t testing.TB) (r restic.Repository, cleanup func())

TestRepository returns a repository initialized with a test password on an in-memory backend. When the environment variable RESTIC_TEST_REPO is set to a non-existing directory, a local backend is created there and this is used instead. The directory is not removed, but left there for inspection.

func TestRepositoryWithBackend 

func TestRepositoryWithBackend(t testing.TB, be restic.Backend) (r restic.Repository, cleanup func())

TestRepositoryWithBackend returns a repository initialized with a test password. If be is nil, an in-memory backend is used. A constant polynomial is used for the chunker and low-security test parameters.

func TestUseLowSecurityKDFParameters 

func TestUseLowSecurityKDFParameters(t logger)

TestUseLowSecurityKDFParameters configures low-security KDF parameters for testing.

Types

type Index 

type Index struct {
	// contains filtered or unexported fields
}

Index holds lookup tables for id -> pack.

func DecodeIndex 

func DecodeIndex(buf []byte) (idx *Index, err error)

DecodeIndex loads and unserializes an index from rd.

func DecodeOldIndex 

func DecodeOldIndex(buf []byte) (idx *Index, err error)

DecodeOldIndex loads and unserializes an index in the old format from rd.

func LoadIndex 

func LoadIndex(ctx context.Context, repo restic.Repository, id restic.ID) (*Index, error)

LoadIndex loads the index id from backend and returns it.

func LoadIndexWithDecoder 

func LoadIndexWithDecoder(ctx context.Context, repo restic.Repository, buf []byte, id restic.ID, fn func([]byte) (*Index, error)) (*Index, []byte, error)

LoadIndexWithDecoder loads the index and decodes it with fn.

func NewIndex 

func NewIndex() *Index

NewIndex returns a new index.

func (*Index) AddToSupersedes 

func (idx *Index) AddToSupersedes(ids ...restic.ID) error

AddToSupersedes adds the ids to the list of indexes superseded by this index. If the index has already been finalized, an error is returned.

func (*Index) Count 

func (idx *Index) Count(t restic.BlobType) (n uint)

Count returns the number of blobs of type t in the index.

func (*Index) Dump 

func (idx *Index) Dump(w io.Writer) error

Dump writes the pretty-printed JSON representation of the index to w.

func (*Index) Each 

func (idx *Index) Each(ctx context.Context) <-chan restic.PackedBlob

Each returns a channel that yields all blobs known to the index. When the context is cancelled, the background goroutine terminates. This blocks any modification of the index.

func (*Index) Encode 

func (idx *Index) Encode(w io.Writer) error

Encode writes the JSON serialization of the index to the writer w.

func (*Index) Final 

func (idx *Index) Final() bool

Final returns true iff the index is already written to the repository, it is finalized.

func (*Index) Finalize 

func (idx *Index) Finalize()

Finalize sets the index to final.

func (*Index) Has 

func (idx *Index) Has(id restic.ID, tpe restic.BlobType) bool

Has returns true iff the id is listed in the index.

func (*Index) IDs added in v0.9.7 

func (idx *Index) IDs() (restic.IDs, error)

ID returns the IDs of the index, if available. If the index is not yet finalized, an error is returned.

func (*Index) ListPack 

func (idx *Index) ListPack(id restic.ID) (list []restic.PackedBlob)

ListPack returns a list of blobs contained in a pack.

func (*Index) Lookup 

func (idx *Index) Lookup(id restic.ID, tpe restic.BlobType) (blobs []restic.PackedBlob, found bool)

Lookup queries the index for the blob ID and returns a restic.PackedBlob.

func (*Index) LookupSize 

func (idx *Index) LookupSize(id restic.ID, tpe restic.BlobType) (plaintextLength uint, found bool)

LookupSize returns the length of the plaintext content of the blob with the given id.

func (*Index) Packs 

func (idx *Index) Packs() restic.IDSet

Packs returns all packs in this index

func (*Index) SetID 

func (idx *Index) SetID(id restic.ID) error

SetID sets the ID the index has been written to. This requires that Finalize() has been called before, otherwise an error is returned.

func (*Index) Store 

func (idx *Index) Store(blob restic.PackedBlob)

Store remembers the id and pack in the index.

func (*Index) StorePack added in v0.9.7 

func (idx *Index) StorePack(id restic.ID, blobs []restic.Blob)

StorePack remembers the ids of all blobs of a given pack in the index

func (*Index) Supersedes 

func (idx *Index) Supersedes() restic.IDs

Supersedes returns the list of indexes this index supersedes, if any.

func (*Index) TreePacks added in v0.8.0 

func (idx *Index) TreePacks() restic.IDs

TreePacks returns a list of packs that contain only tree blobs.

type Key 

type Key struct {
	Created  time.Time `json:"created"`
	Username string    `json:"username"`
	Hostname string    `json:"hostname"`

	KDF  string `json:"kdf"`
	N    int    `json:"N"`
	R    int    `json:"r"`
	P    int    `json:"p"`
	Salt []byte `json:"salt"`
	Data []byte `json:"data"`
	// contains filtered or unexported fields
}

Key represents an encrypted master key for a repository.

func AddKey 

func AddKey(ctx context.Context, s *Repository, password string, template *crypto.Key) (*Key, error)

AddKey adds a new key to an already existing repository.

func LoadKey 

func LoadKey(ctx context.Context, s *Repository, name string) (k *Key, err error)

LoadKey loads a key from the backend.

func OpenKey 

func OpenKey(ctx context.Context, s *Repository, name string, password string) (*Key, error)

OpenKey tries do decrypt the key specified by name with the given password.

func SearchKey 

func SearchKey(ctx context.Context, s *Repository, password string, maxKeys int, keyHint string) (k *Key, err error)

SearchKey tries to decrypt at most maxKeys keys in the backend with the given password. If none could be found, ErrNoKeyFound is returned. When maxKeys is reached, ErrMaxKeysReached is returned. When setting maxKeys to zero, all keys in the repo are checked.

func (Key) Name 

func (k Key) Name() string

Name returns an identifier for the key.

func (*Key) String 

func (k *Key) String() string

func (*Key) Valid 

func (k *Key) Valid() bool

Valid tests whether the mac and encryption keys are valid (i.e. not zero)

type Loader added in v0.9.7 

type Loader interface {
	Load(ctx context.Context, h restic.Handle, length int, offset int64, fn func(rd io.Reader) error) error
}

Loader allows loading data from a backend.

type MasterIndex 

type MasterIndex struct {
	// contains filtered or unexported fields
}

MasterIndex is a collection of indexes and IDs of chunks that are in the process of being saved.

func NewMasterIndex 

func NewMasterIndex() *MasterIndex

NewMasterIndex creates a new master index.

func (*MasterIndex) All 

func (mi *MasterIndex) All() []*Index

All returns all indexes.

func (*MasterIndex) Count 

func (mi *MasterIndex) Count(t restic.BlobType) (n uint)

Count returns the number of blobs of type t in the index.

func (*MasterIndex) Each 

func (mi *MasterIndex) Each(ctx context.Context) <-chan restic.PackedBlob

Each returns a channel that yields all blobs known to the index. When the context is cancelled, the background goroutine terminates. This blocks any modification of the index.

func (*MasterIndex) FinalizeFullIndexes added in v0.9.7 

func (mi *MasterIndex) FinalizeFullIndexes() []*Index

FinalizeFullIndexes finalizes all indexes that are full and returns that list.

func (*MasterIndex) FinalizeNotFinalIndexes added in v0.9.7 

func (mi *MasterIndex) FinalizeNotFinalIndexes() []*Index

FinalizeNotFinalIndexes finalizes all indexes that have not yet been saved and returns that list

func (*MasterIndex) Has 

func (mi *MasterIndex) Has(id restic.ID, tpe restic.BlobType) bool

Has queries all known Indexes for the ID and returns the first match. Also returns true if the ID is pending.

func (*MasterIndex) Insert 

func (mi *MasterIndex) Insert(idx *Index)

Insert adds a new index to the MasterIndex.

func (*MasterIndex) ListPack 

func (mi *MasterIndex) ListPack(id restic.ID) (list []restic.PackedBlob)

ListPack returns the list of blobs in a pack. The first matching index is returned, or nil if no index contains information about the pack id.

func (*MasterIndex) Lookup 

func (mi *MasterIndex) Lookup(id restic.ID, tpe restic.BlobType) (blobs []restic.PackedBlob, found bool)

Lookup queries all known Indexes for the ID and returns the first match.

func (*MasterIndex) LookupSize 

func (mi *MasterIndex) LookupSize(id restic.ID, tpe restic.BlobType) (uint, bool)

LookupSize queries all known Indexes for the ID and returns the first match.

func (*MasterIndex) MergeFinalIndexes added in v0.9.7 

func (mi *MasterIndex) MergeFinalIndexes()

MergeFinalIndexes merges all final indexes together. After calling, there will be only one big final index in MasterIndex containing all final index contents. Indexes that are not final are left untouched. This merging can only be called after all index files are loaded - as removing of superseded index contents is only possible for unmerged indexes.

func (*MasterIndex) RebuildIndex 

func (mi *MasterIndex) RebuildIndex(packBlacklist restic.IDSet) (*Index, error)

RebuildIndex combines all known indexes to a new index, leaving out any packs whose ID is contained in packBlacklist. The new index contains the IDs of all known indexes in the "supersedes" field.

func (*MasterIndex) StorePack added in v0.9.7 

func (mi *MasterIndex) StorePack(id restic.ID, blobs []restic.Blob)

Store remembers the id and pack in the index.

type Packer 

type Packer struct {
	*pack.Packer
	// contains filtered or unexported fields
}

Packer holds a pack.Packer together with a hash writer.

type Repository 

type Repository struct {
	restic.Cache
	// contains filtered or unexported fields
}

Repository is used to access a repository in a backend.

func New 

func New(be restic.Backend) *Repository

New returns a new repository with backend be.

func (*Repository) Backend 

func (r *Repository) Backend() restic.Backend

Backend returns the backend for the repository.

func (*Repository) Close 

func (r *Repository) Close() error

Close closes the repository by closing the backend.

func (*Repository) Config 

func (r *Repository) Config() restic.Config

Config returns the repository configuration.

func (*Repository) Delete 

func (r *Repository) Delete(ctx context.Context) error

Delete calls backend.Delete() if implemented, and returns an error otherwise.

func (*Repository) DisableAutoIndexUpdate added in v0.9.7 

func (r *Repository) DisableAutoIndexUpdate()

func (*Repository) Flush 

func (r *Repository) Flush(ctx context.Context) error

Flush saves all remaining packs and the index

func (*Repository) FlushPacks added in v0.9.7 

func (r *Repository) FlushPacks(ctx context.Context) error

FlushPacks saves all remaining packs.

func (*Repository) Index 

func (r *Repository) Index() restic.Index

Index returns the currently used MasterIndex.

func (*Repository) Init 

func (r *Repository) Init(ctx context.Context, password string) error

Init creates a new master key with the supplied password, initializes and saves the repository config.

func (*Repository) Key 

func (r *Repository) Key() *crypto.Key

Key returns the current master key.

func (*Repository) KeyName 

func (r *Repository) KeyName() string

KeyName returns the name of the current key in the backend.

func (*Repository) List 

func (r *Repository) List(ctx context.Context, t restic.FileType, fn func(restic.ID, int64) error) error

List runs fn for all files of type t in the repo.

func (*Repository) ListPack 

func (r *Repository) ListPack(ctx context.Context, id restic.ID, size int64) ([]restic.Blob, int64, error)

ListPack returns the list of blobs saved in the pack id and the length of the file as stored in the backend.

func (*Repository) LoadAndDecrypt 

func (r *Repository) LoadAndDecrypt(ctx context.Context, buf []byte, t restic.FileType, id restic.ID) ([]byte, error)

LoadAndDecrypt loads and decrypts the file with the given type and ID, using the supplied buffer (which must be empty). If the buffer is nil, a new buffer will be allocated and returned.

func (*Repository) LoadBlob 

func (r *Repository) LoadBlob(ctx context.Context, t restic.BlobType, id restic.ID, buf []byte) ([]byte, error)

LoadBlob loads a blob of type t from the repository. It may use all of buf[:cap(buf)] as scratch space.

func (*Repository) LoadIndex 

func (r *Repository) LoadIndex(ctx context.Context) error

LoadIndex loads all index files from the backend in parallel and stores them in the master index. The first error that occurred is returned.

func (*Repository) LoadJSONUnpacked 

func (r *Repository) LoadJSONUnpacked(ctx context.Context, t restic.FileType, id restic.ID, item interface{}) (err error)

LoadJSONUnpacked decrypts the data and afterwards calls json.Unmarshal on the item.

func (*Repository) LoadTree 

func (r *Repository) LoadTree(ctx context.Context, id restic.ID) (*restic.Tree, error)

LoadTree loads a tree from the repository.

func (*Repository) LookupBlobSize 

func (r *Repository) LookupBlobSize(id restic.ID, tpe restic.BlobType) (uint, bool)

LookupBlobSize returns the size of blob id.

func (*Repository) PrefixLength 

func (r *Repository) PrefixLength(t restic.FileType) (int, error)

PrefixLength returns the number of bytes required so that all prefixes of all IDs of type t are unique.

func (*Repository) PrepareCache added in v0.9.0 

func (r *Repository) PrepareCache(indexIDs restic.IDSet) error

PrepareCache initializes the local cache. indexIDs is the list of IDs of index files still present in the repo.

func (*Repository) SaveAndEncrypt 

func (r *Repository) SaveAndEncrypt(ctx context.Context, t restic.BlobType, data []byte, id restic.ID) error

SaveAndEncrypt encrypts data and stores it to the backend as type t. If data is small enough, it will be packed together with other small blobs. The caller must ensure that the id matches the data.

func (*Repository) SaveBlob 

func (r *Repository) SaveBlob(ctx context.Context, t restic.BlobType, buf []byte, id restic.ID, storeDuplicate bool) (newID restic.ID, known bool, err error)

SaveBlob saves a blob of type t into the repository. It takes care that no duplicates are saved; this can be overwritten by setting storeDuplicate to true. If id is the null id, it will be computed and returned. Also returns if the blob was already known before

func (*Repository) SaveFullIndex 

func (r *Repository) SaveFullIndex(ctx context.Context) error

SaveFullIndex saves all full indexes in the backend.

func (*Repository) SaveIndex 

func (r *Repository) SaveIndex(ctx context.Context) error

SaveIndex saves all new indexes in the backend.

func (*Repository) SaveJSONUnpacked 

func (r *Repository) SaveJSONUnpacked(ctx context.Context, t restic.FileType, item interface{}) (restic.ID, error)

SaveJSONUnpacked serialises item as JSON and encrypts and saves it in the backend as type t, without a pack. It returns the storage hash.

func (*Repository) SaveTree 

func (r *Repository) SaveTree(ctx context.Context, t *restic.Tree) (restic.ID, error)

SaveTree stores a tree into the repository and returns the ID. The ID is checked against the index. The tree is only stored when the index does not contain the ID.

func (*Repository) SaveUnpacked 

func (r *Repository) SaveUnpacked(ctx context.Context, t restic.FileType, p []byte) (id restic.ID, err error)

SaveUnpacked encrypts data and stores it in the backend. Returned is the storage hash.

func (*Repository) SearchKey 

func (r *Repository) SearchKey(ctx context.Context, password string, maxKeys int, keyHint string) error

SearchKey finds a key with the supplied password, afterwards the config is read and parsed. It tries at most maxKeys key files in the repo.

func (*Repository) SetIndex 

func (r *Repository) SetIndex(i restic.Index) error

SetIndex instructs the repository to use the given index.

func (*Repository) UseCache added in v0.8.0 

func (r *Repository) UseCache(c restic.Cache)

UseCache replaces the backend with the wrapped cache.

type Saver 

type Saver interface {
	Save(context.Context, restic.Handle, restic.RewindReader) error
}

Saver implements saving data in a backend.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.