pitreos

package module
v1.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 8, 2020 License: MIT Imports: 23 Imported by: 3

README

pitreos: Point in Time Recovery Tool by EOS Canada

Pronounced like "Patriots"

Backup and restore tool optimized for large sparse files and append-only files.

Perfect for EOS.IO blockchains' state and blocks logs, virtual machine images or other large files that change only in part.

Supported platforms

  • Linux
  • OSX (no sparse file optimization)

How does it work ?

Backing up

  1. It splits each file into smaller chunks and computes the hashes of each of those.
  2. It sends those chunks to a defined storage URL along with an index file linking to the file chunks.
  3. Those chunks can also be cached locally.
  4. Backups are stored with a timestamp and an optional tag. Use tags to differentiate backups which may share data chunks but have different uses (ex: dev vs prod, osx vs linux...)

Restoring

  1. The "list" command will fetch the last backups from your defined storage URL
  2. The "restore" command will fetch the backup index chosen chosen from the list. (Alternatively, you can ask pitreos to restore to the latest backup of a specific tag)
  3. The backup index is parsed: it contains the list of files and their content based on their chunk hashes.
  4. Each non-existing local file is created as an empty sparse file with the expected length, while existing files are truncated (or enlarged) to the expected length
  5. Local chunks are hashed and compared to the expected chunk.
  6. Chunks which should be empty get a hole punched through them (becoming a sparse file)
  7. Chunks which should have different content are downloaded from your backup store.

Optimizations:

  • Empty chunks (no data or only null bytes) are not transferred
  • Unassigned chunks in sparse files are not even read or written to
  • Chunks with data are compressed before transfer
  • Caching can be enabled to keep any downloaded/uploaded chunk locally and quickly restore your files.
  • Chunks are not uploaded again if the same content exist at the same destination (with the same backup path)
  • Existing data in files flagged as "appendonly-files" are not verified on restore. Only missing data at the end of the file is downloaded.

Known issues:

  • File permissions are not managed

How to install ?

  1. Ensure that you have a sane GOLANG environment, with your PATH to $GOPATH/bin
  2. Run the following commands from the repo
$ go get ./...
$ go install -v

Examples

Example .pitreos.yaml in your workspace

store: gs://mybackups/nodeos_data
tag: john_dev

Backup to default location

pitreos backup ./mydata

  • This will send your chunks to $HOME/.pitreos/backups/chunks/{sha3_256sum}
  • This will send your backup index to $HOME/.pitreos/backups/indexes/{timestamp}-default.yaml.gz

Backup to Google Storage

pitreos backup mydata -s gs://mybackups/projectname -t dev -c -m '{"blocknum": 123456, "version": "1.2.1"}'

  • This will send your data chunks under gs://mybackups/projectname/chunks/
  • your backup index file will be located under gs://mybackups/projectname/indexes/{timestamp}-dev.yaml.gz
  • your chunks file will also be savec in your default cache location ($HOME/.pitreos/cache)
  • The backup metadata will contain the provided arbitrary values "blocknum" and "version"

List your 5 last backups

pitreos list --limit 5

  • 2018-08-28-19-24-59--default
  • 2018-08-28-18-15-39--john-dev

Restore a specific backup

pitreos -c restore 2018-08-28-18-15-48--john-dev ./mydata

  • This will restore your data from that backup under ./mydata. Any file already existing there may speed up the process.

Restore from latest backup of a tag

pitreos -c restore -t john-dev ./mydata

  • This will restore your data from the latest backup with the "john-dev" tag.

More examples in help !

Run "pitreos help", "pitreos help backup" and "pitreos help restore" for more examples

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var AllFileFilter = FilterFunc(func(relativePath string) bool { return true })

Functions

func SetLogger 

func SetLogger(logger *zap.Logger)

Types

type BackupIndex 

type BackupIndex struct {
	Version   string                 `json:"version"`
	Date      time.Time              `json:"date"`
	Tag       string                 `json:"tag"`
	Meta      map[string]interface{} `json:"meta"`
	Files     []*FileIndex           `json:"files"`
	ChunkSize int64                  `json:"chunk_size"`
}

func (*BackupIndex) ComputeFileEstimatedDiskSize 

func (backup *BackupIndex) ComputeFileEstimatedDiskSize(filename string) (uint64, error)

func (*BackupIndex) FindFilesMatching 

func (backupIndex *BackupIndex) FindFilesMatching(filter Filter) ([]*FileIndex, error)

type ChunkDef 

type ChunkDef struct {
	Start      int64  `json:"start"`
	End        int64  `json:"end"`
	IsEmpty    bool   `json:"empty,omitempty"`
	ContentSHA string `json:"contentSHA,omitempty"`
}

type DStoreStorage 

type DStoreStorage struct {
	// contains filtered or unexported fields
}

func NewDStoreStorage 

func NewDStoreStorage(ctx context.Context, baseURL string) (*DStoreStorage, error)

func (*DStoreStorage) ChunkExists 

func (s *DStoreStorage) ChunkExists(hash string) (bool, error)

func (*DStoreStorage) ListBackups 

func (s *DStoreStorage) ListBackups(limit int, prefix string) (out []string, err error)

func (*DStoreStorage) OpenBackupIndex 

func (s *DStoreStorage) OpenBackupIndex(name string) (out io.ReadCloser, err error)

func (*DStoreStorage) OpenChunk 

func (s *DStoreStorage) OpenChunk(hash string) (out io.ReadCloser, err error)

func (*DStoreStorage) SetTimeout 

func (s *DStoreStorage) SetTimeout(timeout time.Duration)

func (*DStoreStorage) WriteBackupIndex 

func (s *DStoreStorage) WriteBackupIndex(name string, content []byte) (err error)

func (*DStoreStorage) WriteChunk 

func (s *DStoreStorage) WriteChunk(hash string, content []byte) (err error)

type FileIndex 

type FileIndex struct {
	FileName  string      `json:"filename"`
	Date      time.Time   `json:"date"`
	TotalSize int64       `json:"size"`
	Chunks    []*ChunkDef `json:"chunks"`
}

type FileOps 

type FileOps struct {
	// contains filtered or unexported fields
}

func NewFileOps 

func NewFileOps(filePath string, readWrite bool) *FileOps

func (*FileOps) Close 

func (f *FileOps) Close() error

func (*FileOps) Open 

func (f *FileOps) Open() error

func (*FileOps) Truncate 

func (f *FileOps) Truncate(size int64) error

type Filter 

type Filter interface {
	Match(relativePath string) bool
}

type FilterFunc 

type FilterFunc func(relativePath string) bool

func (FilterFunc) Match 

func (f FilterFunc) Match(relativePath string) bool

type GZipReadCloser 

type GZipReadCloser struct {
	*gzip.Reader
	// contains filtered or unexported fields
}

func NewGZipReadCloser 

func NewGZipReadCloser(src io.ReadCloser) (*GZipReadCloser, error)

func (*GZipReadCloser) Close 

func (g *GZipReadCloser) Close() error

type IncludeThanExcludeFilter 

type IncludeThanExcludeFilter struct {
	// contains filtered or unexported fields
}

func MustNewIncludeThanExcludeFilter 

func MustNewIncludeThanExcludeFilter(includeFilter, excludeFilter string) *IncludeThanExcludeFilter

func NewIncludeThanExcludeFilter 

func NewIncludeThanExcludeFilter(includeFilter, excludeFilter string) (*IncludeThanExcludeFilter, error)

func (*IncludeThanExcludeFilter) Match 

func (f *IncludeThanExcludeFilter) Match(relativePath string) bool

type ListableBackup 

type ListableBackup struct {
	Name string
	Meta map[string]interface{}
}

type PITR 

type PITR struct {
	AppendonlyFiles []string
	// contains filtered or unexported fields
}

func New 

func New(chunkSizeMiB int64, threads int, transferTimeout time.Duration, storage Storage) *PITR

func NewDefaultPITR 

func NewDefaultPITR(storage Storage) *PITR

func (*PITR) GenerateBackup 

func (p *PITR) GenerateBackup(source string, tag string, metadata map[string]interface{}, filter Filter) error

func (*PITR) GetLatestBackup 

func (p *PITR) GetLatestBackup(tag string) (string, error)

func (*PITR) ListBackupFiles 

func (p *PITR) ListBackupFiles(backupName string, filter Filter) error

func (*PITR) ListBackups 

func (p *PITR) ListBackups(limit, offset int, prefix string, withMeta bool) (out []*ListableBackup, err error)

func (*PITR) RestoreFromBackup 

func (p *PITR) RestoreFromBackup(dest string, backupName string, filter Filter) error

func (*PITR) SetCacheStorage 

func (p *PITR) SetCacheStorage(storage Storage)

SetCachingStorage enables caching through the provided Storage object.

type Storage 

type Storage interface {
	ListBackups(limit int, prefix string) ([]string, error)
	OpenBackupIndex(name string) (io.ReadCloser, error)
	WriteBackupIndex(name string, content []byte) error

	OpenChunk(hash string) (io.ReadCloser, error)
	WriteChunk(hash string, content []byte) error
	ChunkExists(hash string) (bool, error)
	SetTimeout(timeout time.Duration)
}

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.