Reactor

Reactor

Reactive synchronizing of cloud data using rclone on Windows

Summary

I've been using rclone for the last couple of years to encrypt and mirror my Nextcloud data to a secondary service as part of a "disaster recovery" plan. The process, however, has been on-demand. I wrote a number of Python scripts to take care of the details, but I always needed to run them manually to get my local data fully synchronized with the backup cloud service.

Not really a big deal, right? I'm a creature of habit, and I can remember to do that every morning ... after I check my email ... and after I get through with this meeting ... and did the builds complete last night? ... and, crap, I can no longer see straight. I'll have to do it tomorrow.

This started leaving a big hole in my "disaster" planning when I would extend the synchronizing window from its already worrying 24 hours to 48, or 72. I realilzed that I needed a tool that would use rclone to synchronize my critical data whenever it changed, not just whenever I remembered.

The solution to this is Reactor.

Fire and forget

Reactor is a fire-and-forget solution. Once started, if it has been configured properly, it will start monitoring one or more locations (or "watchpoints") on your file system, and will ensure that any cloud storage (or "remotes" in rclone parlance) associated with those locations are an exact mirror. You can configure multiple watchpoints, and each watchpoint can have multiple rclone "remotes" associated with it.

Reactor doesn't use any command-line arguments. Instead, it is fully configured using an XML file. (Hey! Why not JSON? Simply put, XML is much more expressive, and it has syntactical support for comments.) Since I've developed this to address my specific needs, it is somewhat biased toward a Windows environment (adjusting it for others should really not require much work). Therefore, after seasoning to your liking, the "Reactor.xml" configruation file should be placed into the %APPDATA% folder so Reactor can find it on startup.

rclone and Remotes

Reactor expects your rclone executable to be visible in your %PATH% setting such that executing rclone --version from a command line will produce rclone output (and not an error).

It goes without saying that you need to configure your remote cloud services in rclone before you use Reactor. Additionally, unless your data footprint is ridiculoulsy small, you should also get your data image situated on the remote storage before you aim Reactor at it. Reactor is really designed to handle incremental updates to existing synchornized data. Expecting Reactor to peform the initial upload of your tens of gigabytes of data would be a silly use of the tool, and the gods will weep at your foolishness. Use rclone directly to initialize environments.

Parallelization

It should be noted that Reactor sequentializes synchronizations between watchpoints (one won't start until a previous one completes), but will parallelize synchronizations among multiple remotes associated with the same watchpoint. I tested this with two remotes; the time to sync took longer on each, but the total time to complete all uploads was equivalent to running them sequentially (and on good days, sometimes even ran a few seconds faster).

winfsnotify

One last thing worth mentioning: Reactor uses a modified version of the golang/exp/winfsnotify package. Much to my surprise, the original package only watched changes in a single folder, ignoring all events of its child folders and files. I enhanced winfsnotify to allow it to report on events in the entire subtree of a watched folder--a critical requirement of Reactor.

My changes have been submitted to the project maintainers of winfsnotify for incorporation into the published package. If those changes ever make their way into it, I will remove this locally modified version and update Reactor to use the official version.

Building

On Windows, compile with: go generate & go build -ldflags "-H=windowsgui -s -w" .

For most of the dependencies, the Go system will pull down all modules that Reactor requires as part of the build process. The one exception is goversioninfo. You wil need to manually retrieve this package so the first 'generate' step succeeds. You can do this with

go get -u github.com/josephspurrier/goversioninfo/cmd/goversioninfo

before building or installing Reactor.

I hope you find this useful.

Documentation

See the Reactor.xml file for detailed comments concerning configuring Reactor.