cloudwatcher

Cloudwatcher

File system notification for Cloud platforms (and not) in Golang.

cloudwatcher is a file system notification library for cloud platforms (and not) in Go. Currently it implements the watchers for the following services:

• Amazon S3
• Google Drive
• Dropbox
• Git
• local filesystem (fsnotify/polling)

It is possible specify the directory to monitor and the polling time (how much often the watcher should check that directory), and every time a new file is added, removed or changed, an event is generated and sent to the Events channel.

Usage

package main

import (
	"fmt"
	"time"

	"github.com/Matrix86/cloudwatcher"
)

func main() {
    // the first parameter is the type of service to use: local, dropbox, gdrive or s3
	s, err := cloudwatcher.New("local", "/home/user/tests", time.Second)
	if err != nil {
		fmt.Printf("ERROR: %s", err)
		return
	}

	config := map[string]string{
		"disable_fsnotify": "false",
	}

	err = s.SetConfig(config)
	if err != nil {
		fmt.Printf("ERROR: %s", err)
		return
	}

	err = s.Start()
	defer s.Close()
	for {
		select {
		case v := <-s.GetEvents():
			fmt.Printf("EVENT: %s %s\n", v.Key, v.TypeString())

		case e := <-s.GetErrors():
			fmt.Printf("ERROR: %s\n", e)
		}
	}
}

The channel returned by GetEvents() function, will return an Event struct that contains the event type, the Key with the name of the file that generates the event and the object itself.

⚠ check the Event.Object field before use it...in some cases it could be nil (FileDelete event with fsnotify)

Amazon S3

The config of the S3 watcher is the following:

config := map[string]string{
    "bucket_name": "storage",
    "endpoint":   "s3-us-west-2.amazonaws.com",
    "access_key":  "user",
    "secret_key":  "secret",
    "token":      "",
    "region":     "us-west-2",
    "ssl_enabled": "true",
}

To use AWS IAM credentials or AWS file, it is possible to set one of the following vars to "true": aws_iam_credentials, aws_file. It is also possible to specify the IAM endpoint (aws_file_profile) or the path of the file to use (aws_file_profile), if not specified it will use the default endpoint and file ($HOME/.aws/credentials).

config := map[string]string{
    "aws_file_profile": "true",
}

An example can be found here.

💎 minio can be used for testing purposes

Google Drive

In order to use it you need to enable the drive API from here . After that you can specify the client-id and the client-secret on the config file. The logic to retrieve the token has to be handled outside the library and the token field should contain the json.

You can find an example in the examples directory.

config := map[string]string{
    "debug":         "true",
    "token":         Token,
    "client_id":     ClientId,
    "client_secret": ClientSecret,
}

Dropbox

First, you need to register a new app from the developer console. Use the client-id and the client-secret to retrieve the user token and use it on the token field of the config.

You can find an example in the examples directory.

config := map[string]string{
    "debug": "true",
    "token": Token,
}

Local filesystem

It is based on fsnotify library, so it is cross platform: Windows, Linux, BSD and macOS. It is not mandatory to call the SetConfig() function, and the polling time argument of cloudwatcher.New is not used.

Setting disable_fsnotify parameter on config to "true" the watcher doesn't use fsnotify and use the listing approach instead.

⚠ not set disable_fsnotify to "true" if you plan to use it on a big directory!!! It could increase the I/O on disk

Git

Git watcher has the following configurations:

If monitor_type is set to "repo", the event channel will receive an event with the Object field filled with commits or tags. If assemble_events is "true" the Object field could contains one or more commits.