README
¶
Cloud storage mass uploader
Compare an FTP archive to a cloud storage bucket, copy files which are either not included in the cloud storage bucket OR where the MD5 mismatches from the ftp archive to the cloud storage bucket.
Operations
Compile Code
$ go build -o mass_uploader mass_uploader.go
Create Service Account
Create a service account to be used in uploading and querying the cloud storage resources. Go to the [IAM -> ServiceAccounts] (https://console.cloud.google.com/iam-admin/serviceaccounts) page, create a new service account.
Create a new key for the service-account, download the JSON format key material. Store this key material securely on the machine which will be performing the synchronization.
Grant the service account the following roles/permissions:
- Cloud Run Invoker - configured in the CloudRun instance permissions.
- Storage Object Creator - configured in IAM permissions.
- Storage Object Viewer - configured in IAM permissions.
- Storage Object Creator - configured for the storage bucket
- Storage Object Viewer - configured for the storage bucket
Run the Upload Process
Run the program, provide the bucket and ftp archive as flags, provide the key file location through the environment variable: GOOGLE_APPLICATION_CREDENTIALS.
$ GOOGLE_APPLICATION_CREDENTIALS=<filesystem_path_to_key> mass_upload -bucket routeviews-archives -archive ftp://archive.routeviews.org/bgpdata
Review Logs
Review logged data for errors, address as required.
Documentation
¶
Overview ¶
Package mass_upload should upload content from a given ftp site to cloud storage. Verification of ftp site content against the cloud storage location before upload must be performed.
Basic flow is:
- start at the top of an FTP site.
- download each file in turn, walking the remote directory tree.
- calculate the md5() checksum for each file downloaded.
- validate that the checksum matches the cloud-storage object's MD5 value.
- if there is a mis-match, upload the ftp content to cloud-storage.
Source Files