README
¶
diskrsync -- rsync for block devices and disk images
This is a utility for remote backup of disk image files or devices. It uses ssh as a transport and tries to minimise the amount of data transferred. This is done by building a Merkle tree of blake2 hashes on both the source and the target hosts and then traversing the tree only transmitting those blocks where the hashes differ.
It is important that the source file does not change during the process. Typically this would be achieved by creating an LVM snapshot or by freezing the VM if that's not possible.
By default the resulting file is compressed using the spgz library (this can be disabled by using --no-compress flag). Note this only works on filesystems that support punching holes in files like xfs or ext4.
The utility handles sparse files (or just files with a lot of zeros) efficiently. The resulting file will be sparse (even if not compressed).
Size changes are also supported (both shrinks and expansions).
Installation
-
Install go. Version 1.16 is minimum required but the newer the better. If your distribution lacks the required version, check backports or updates (e.g. for debian or ubuntu) Alternatively, install manually.
-
Run the following commands:
mkdir workspace
cd workspace
GOPATH=$(pwd) go install github.com/dop251/diskrsync/diskrsync@latest
sudo cp -a bin/diskrsync /usr/local/bin
- Make sure the binary is copied to the remote machine as well. If the remote machine has a different CPU or OS you may want to use cross-compilation. For example if you are want to build a binary for ARM:
GOPATH=$(pwd) GOARCH=arm go install github.com/dop251/diskrsync/diskrsync@latest
ls -l bin/linux_arm/diskrsync
Usage examples
diskrsync /dev/vg00/lv_snap ruser@backuphost:/mnt/backup/disk
This ensures that /mnt/backup/disk is up-to-date with the LV snapshot. The file will be compressed using spgz and can be recovered using the following command:
spgz -x /mnt/backup/disk /dev/vg00/....
diskrsync --verbose --no-compress --ssh-flags="-i id_file" /var/lib/libvirt/images/disk.img ruser@rbackuphost:/mnt/backup/
This ensures that /mnt/backup/disk.img is up-to-date with the source file.
Documentation
¶
Index ¶
- Constants
- Variables
- func Source(reader io.ReadSeeker, size int64, cmdReader io.Reader, cmdWriter io.Writer, ...) (err error)
- func Target(writer spgz.SparseFile, size int64, cmdReader io.Reader, cmdWriter io.Writer, ...) (err error)
- type CountingReader
- type CountingWriteCloser
- type FixingSpgzFileWrapper
- type ProgressListener
- type TargetFile
Constants ¶
const (
DefTargetBlockSize = 128 * 1024
)
Variables ¶
var (
ErrInvalidFormat = errors.New("invalid data format")
)
Functions ¶
Types ¶
type CountingReader ¶
type CountingWriteCloser ¶
type CountingWriteCloser struct {
io.WriteCloser
// contains filtered or unexported fields
}
type FixingSpgzFileWrapper ¶ added in v1.2.0
FixingSpgzFileWrapper conceals read errors caused by compressed data corruption by re-writing the corrupt blocks with zeros. Such errors are usually caused by abrupt termination of the writing process. This wrapper is used as the sync target so the corrupt blocks will be updated during the sync process.
type ProgressListener ¶ added in v1.3.0
type TargetFile ¶ added in v1.2.0
type TargetFile interface {
io.ReadWriteSeeker
io.WriterAt
io.Closer
spgz.Truncatable
}
Source Files
Directories