sync-cloud-storage

Sync Cloud Storage

This repository includes files, which will syncs Azure latest uploaded files and downloads them to NAS Server. Later that data will be transferred to Amazon Snowball (and later to Amazon S3 Bucket).

Once the data is transferred to Snowball, there's still new content which gets uploaded to Azure, as the revamped website will be in-active. Hence, a separate sync mechanism will be used to download Azure content and gets uploaded to S3.

Technology

I'm using Golang to implement this task.

Algorithm(s):

Algorithm #1: Sync Azure Content Timeline: Till Launch

1. Get entire list of Azure container(s)
2. Store these list in SQLite DB (inside containers table)
3. Taking containers as reference traverse blob and store that in sync table
4. Table Structure (containers): name, status
5. Table Structure (sync): id, container, blob, azure_status, azure_error, aws_status, aws_error

Algorithm #2: Download Azure Content Timeline: Before Snowball Transfer

1. Fetch 10 eligible entries from containers (As of now I'm taking only 10 entries, can be increased if we meet the network performance and error handling)
2. Create 10 Go Routine to initiate 10 concurrent downloads
3. Go Routine response gets updated with azure_status 1 / 2 (+azure_error) on success / failure respectively

Algorithm #3: Download Azure Content and Upload to Amazon S3 Timeline: After Snowball Transfer

1. Execute the "Sync Azure Content" algorithm to fetch any new assets
2. Execute the "Download Azure Content" algorithm to download latest content
3. Taking reference from "Download Azure Content" algorithm, upload content to Amazon S3
4. Go Routine response gets updated with s3_status 2 or 3 (+s3_error) on success and failure respectively

Container Table Structure

Note: Status Code (0: InActive, 100: Live, 200: Success, 404: Container Not Found, 503: Blob Not Found)

Sync Table Structure:

Note: Status Code(0: InActive 1: Success 2: Failure)

Code Execution

To run sync script:

$ cd sync-cloud-storage
$ go run init.go -sync -container

$ cd sync-cloud-storage
$ go run init.go -sync -blob

To reset live containers:

$ cd sync-cloud-storage
$ go run init.go -reset-live

To delete older data:

$ cd sync-cloud-storage
$ go run init.go  -clean

To run azure download script:

$ cd sync-cloud-storage
$ go run init.go -download

To run s3 upload script:

$ cd sync-cloud-storage
$ go run init.go -upload

TODO: To cross-check uploaded content:

$ cd sync-cloud-storage
$ go run init.go -xcheck