grsync

package module
v0.0.0-...-ca39cd2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 8, 2023 License: MIT Imports: 10 Imported by: 0

README

grsync — golang rsync wrapper

Quick & Dirty fork of ranchers grsync for my own purposes. Tested with rsync version 3.2.3 protocol version 31

Changes made:

  • Add sshpass-Wrapper to allow passing ssh-password to rsync (sshsync needs to be installed duh, see example)
  • Archive-Flag is not set forcefully
  • Changed Progress to reflect current rsync progress format (not a great solution currently, some more parsing & better regex would be nice)
  • Allow disabling the creation of directory if remote target is used ( see #7)
  • Rsync: Added ListOnly-Param (see examples)
  • Updated deps

codecov GoDoc

Repository contains some helpful tools:

  • raw rsync wrapper
  • rsync task — wrapper which provide important information about rsync task: progress, remain items, total items and speed

Task wrapper usage

package main

import (
	"fmt"
	"github.com/ByteSizedMarius/grsync"
	"time"
)

func main() {
	task := grsync.NewTask(
		"/local/source",
		"remote@target::destination",
		true,  // use sshpass
		false, // create directory
		grsync.RsyncOptions{ // optimized for speed in my specific use-case, no guarantees
			Inplace:         true,
			Partial:         true,
			HumanReadable:   true,
			NumericIDs:      true,
			Rsh:             "ssh -T -c aes128-ctr -o Compression=no -x",
			PasswordFile:    "/local/password/file", // contains password that will be used by sshpass
			RsyncBinaryPath: "/usr/bin/rsync",
		},
	)

	go func() {
		for {
			state := task.State()
			fmt.Printf(
				"progress: %d %% / rem. %s / tot. %s / sp. %s \n",
				state.Progress,
				state.TimeRemaining,
				state.DownloadedTotal,
				state.Speed,
			)
			<-time.After(time.Second * 5)
		}
	}()

	if err := task.Run(); err != nil {
		panic(err)
	}

	fmt.Println("well done")
	fmt.Println(task.Log())
}

Output:

progress: 0 % / rem.  / tot.  / sp.  
progress: 0 % / rem. 0:22:27 / tot. 342.82M / sp. 108.93MB/s 
progress: 0 % / rem. 0:22:16 / tot. 918.52M / sp. 109.41MB/s 
progress: 0 % / rem. 0:22:18 / tot. 1.49G / sp. 108.85MB/s 
progress: 1 % / rem. 0:22:16 / tot. 2.06G / sp. 108.58MB/s 
progress: 1 % / rem. 0:22:16 / tot. 2.63G / sp. 108.19MB/s 
progress: 2 % / rem. 0:22:10 / tot. 3.20G / sp. 108.21MB/s 

File list:

package main

import (
	"fmt"
	"github.com/ByteSizedMarius/grsync"
)

func main() {

	task := grsync.NewTask(
		"remote@target::destination", // directory to list the files of
		"",
		true,
		false,
		grsync.RsyncOptions{
			HumanReadable:   true,
			ListOnly:        true,
			Rsh:             "ssh -T -c aes128-ctr -o Compression=no -x",
			PasswordFile:    "/root/pass",
			RsyncBinaryPath: "/usr/bin/rsync",
		},
	)

	if err := task.Run(); err != nil {
		panic(err)
	}

	for _, file := range task.GetFileList() {
		fmt.Printf("Permissions: %s, Size: %s, Date: %s, Time: %s, Name: %s\n", file[0], file[1], file[2], file[3], file[4])
	}

	fmt.Println("done")
}

Output:

Permissions: drwxrwxrwx, Size: 56, Date: 2023/10/07, Time: 13:19:08, Name: .
Permissions: -rwxrwxrwx, Size: 402.65G, Date: 2023/10/08, Time: 00:32:49, Name: File1
Permissions: -rwxrwxrwx, Size: 150.66G, Date: 2023/09/27, Time: 20:53:59, Name: File2
Permissions: drwxrwxrwx, Size: 68, Date: 2023/10/08, Time: 01:13:13, Name: Dir1

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Log

type Log struct {
	Stderr string `json:"stderr"`
	Stdout string `json:"stdout"`
}

Log contains raw stderr and stdout outputs

type Rsync

type Rsync struct {
	Source      string
	Destination string
	CreateDir   bool
	// contains filtered or unexported fields
}

Rsync is wrapper under rsync

func NewRsync

func NewRsync(source, destination string, useSshPass, createDir bool, options RsyncOptions) *Rsync

NewRsync returns task with described options If useSSHpass is true, then the password will be read from the options.PasswordFile file and passed to the rsync command using sshpass. sshpass needs to be available. If createDir is set to true, the destination will be created if it does not exist.

func (Rsync) Run

func (r Rsync) Run() error

Run start rsync task. The method is kept here for backward compatibility

func (Rsync) Start

func (r Rsync) Start() error

Start starts a rsync command

func (Rsync) StderrPipe

func (r Rsync) StderrPipe() (io.ReadCloser, error)

StderrPipe returns a pipe that will be connected to the command's standard error when the command starts.

func (Rsync) StdoutPipe

func (r Rsync) StdoutPipe() (io.ReadCloser, error)

StdoutPipe returns a pipe that will be connected to the command's standard output when the command starts.

func (Rsync) Wait

func (r Rsync) Wait() error

Wait waits for rsync command to finnish

type RsyncOptions

type RsyncOptions struct {
	// RsyncBinaryPath is a path to the rsync binary; by default just `rsync`
	RsyncBinaryPath string
	// RsyncPath specify the rsync to run on remote machine, e.g `--rsync-path="cd /a/b && rsync"`
	RsyncPath string
	// Verbose increase verbosity
	Verbose bool
	// Quet suppress non-error messages
	Quiet bool
	// Checksum skip based on checksum, not mod-time & size
	Checksum bool
	// Archve is archive mode; equals -rlptgoD (no -H,-A,-X)
	Archive bool
	// Recurse into directories
	Recursive bool
	// Relative option to use relative path names
	Relative bool
	// NoImliedDirs don't send implied dirs with --relative
	NoImpliedDirs bool
	// Update skip files that are newer on the receiver
	Update bool
	// Inplace update destination files in-place
	Inplace bool
	// Append data onto shorter files
	Append bool
	// AppendVerify --append w/old data in file checksum
	AppendVerify bool
	// Dirs transfer directories without recursing
	Dirs bool
	// Links copy symlinks as symlinks
	Links bool
	// CopyLinks transform symlink into referent file/dir
	CopyLinks bool
	// CopyUnsafeLinks only "unsafe" symlinks are transformed
	CopyUnsafeLinks bool
	// SafeLinks ignore symlinks that point outside the tree
	SafeLinks bool
	// CopyDirLinks transform symlink to dir into referent dir
	CopyDirLinks bool
	// KeepDirLinks treat symlinked dir on receiver as dir
	KeepDirLinks bool
	// HardLinks preserve hard links
	HardLinks bool
	// Perms preserve permissions
	Perms bool
	// NoPerms preserve permissions
	NoPerms bool
	// Executability preserve executability
	Executability bool
	// CHMOD affect file and/or directory permissions
	CHMOD os.FileMode
	// Acls preserve ACLs (implies -p)
	ACLs bool
	// XAttrs preserve extended attributes
	XAttrs bool
	// Owner preserve owner (super-user only)
	Owner bool
	// NoOwner prevent copying owner information to destination
	NoOwner bool
	// Group preserve group
	Group bool
	// NoGroup prevent copying group information to destination
	NoGroup bool
	// Devices preserve device files (super-user only)
	Devices bool
	// Specials preserve special files
	Specials bool
	// Times preserve modification times
	Times bool
	// NoTimes prevent copying modification times
	NoTimes bool
	// omit directories from --times
	OmitDirTimes bool
	// Super receiver attempts super-user activities
	Super bool
	// FakeSuper store/recover privileged attrs using xattrs
	FakeSuper bool
	// Sparce handle sparse files efficiently
	Sparse bool
	// DryRun perform a trial run with no changes made
	DryRun bool
	// WholeFile copy files whole (w/o delta-xfer algorithm)
	WholeFile bool
	// OneFileSystem don't cross filesystem boundaries
	OneFileSystem bool
	// BlockSize block-size=SIZE force a fixed checksum block-size
	BlockSize int
	// Rsh -rsh=COMMAND specify the remote shell to use
	Rsh string
	// Existing skip creating new files on receiver
	Existing bool
	// IgnoreExisting skip updating files that exist on receiver
	IgnoreExisting bool
	// RemoveSourceFiles sender removes synchronized files (non-dir)
	RemoveSourceFiles bool
	// Delete delete extraneous files from dest dirs
	Delete bool
	// DeleteBefore receiver deletes before transfer, not during
	DeleteBefore bool
	// DeleteDuring receiver deletes during the transfer
	DeleteDuring bool
	// DeleteDelay find deletions during, delete after
	DeleteDelay bool
	// DeleteAfter receiver deletes after transfer, not during
	DeleteAfter bool
	// DeleteExcluded also delete excluded files from dest dirs
	DeleteExcluded bool
	// IgnoreErrors delete even if there are I/O errors
	IgnoreErrors bool
	// Force deletion of dirs even if not empty
	Force bool
	// MaxDelete max-delete=NUM don't delete more than NUM files
	MaxDelete int
	// MaxSize max-size=SIZE don't transfer any file larger than SIZE
	MaxSize int
	// MinSize don't transfer any file smaller than SIZE
	MinSize int
	// Partial keep partially transferred files
	Partial bool
	// PartialDir partial-dir=DIR
	PartialDir string
	// DelayUpdates put all updated files into place at end
	DelayUpdates bool
	// PruneEmptyDirs prune empty directory chains from file-list
	PruneEmptyDirs bool
	// NumericIDs don't map uid/gid values by user/group name
	NumericIDs bool
	// Timeout: timeout=SECONDS set I/O timeout in seconds
	Timeout int
	// Contimeout: contimeout=SECONDS set daemon connection timeout in seconds
	Contimeout int
	// IgnoreTimes don't skip files that match size and time
	IgnoreTimes bool
	// SizeOnly skip files that match in size
	SizeOnly bool
	// ModifyWindow modify-window=NUM compare mod-times with reduced accuracy
	ModifyWindow bool
	// TempDir temp-dir=DIR create temporary files in directory DIR
	TempDir string
	// Fuzzy find similar file for basis if no dest file
	Fuzzy bool
	// CompareDest compare-dest=DIR also compare received files relative to DIR
	CompareDest string
	// CopyDest copy-dest=DIR ... and include copies of unchanged files
	CopyDest string
	// LinkDest link-dest=DIR hardlink to files in DIR when unchanged
	LinkDest string
	// Compress file data during the transfer
	Compress bool
	// CompressLevel explicitly set compression level
	CompressLevel int
	// SkipCompress skip-compress=LIST skip compressing files with suffix in LIST
	SkipCompress []string
	// CVSExclude auto-ignore files in the same way CVS does
	CVSExclude bool
	// Stats give some file-transfer stats
	Stats bool
	// HumanReadable output numbers in a human-readable format
	HumanReadable bool
	// Progress show progress during transfer
	Progress bool
	// Read daemon-access password from FILE
	PasswordFile string
	// limit socket I/O bandwidth
	BandwidthLimit int
	// Info
	Info string
	// Exclude --exclude="", exclude remote paths.
	Exclude []string
	// Include --include="", include remote paths.
	Include []string
	// Filter --filter="", include filter rule.
	Filter string
	// Chown --chown="", chown on receipt.
	Chown string
	// ListOnly --list-only, list the files instead of copying them.
	ListOnly bool

	// ipv4
	IPv4 bool
	// ipv6
	IPv6 bool

	//out-format
	OutFormat bool
}

RsyncOptions for rsync

type State

type State struct {
	TimeRemaining   string `json:"remain"`   // Time Remaining (hh:mm:ss)
	DownloadedTotal string `json:"total"`    // Amount of downloaded Data in unknown unit
	Speed           string `json:"speed"`    // Speed of download in unknown unit
	Progress        int    `json:"progress"` // Progress in percent (0-100)
}

State contains information about rsync process

type Task

type Task struct {
	// contains filtered or unexported fields
}

Task is high-level API under rsync

func NewTask

func NewTask(source, destination string, useSshPass, createDir bool, rsyncOptions RsyncOptions) *Task

NewTask returns new rsync task

func (*Task) GetFileList

func (t *Task) GetFileList() (files [][]string)

GetFileList is a helper function that returns a partially parsed list of files if RsyncOptions.ListOnly is true. The Information is returned as a slice of slices of strings in the following format: Index Value 0 Permissions 1 Size in Bytes 2 Date 3 Time 4 Name

func (*Task) Log

func (t *Task) Log() Log

Log return structure which contains raw stderr and stdout outputs

func (*Task) Run

func (t *Task) Run() error

Run starts rsync process with options

func (*Task) State

func (t *Task) State() State

State returns information about rsync processing task lock mutex to avoid accessing it while ProcessStdout is writing to it

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL