README
¶
s3sync
Golang utility for syncing between s3 and local
Usage
Use New to create a manager, and Sync function syncs between s3 and local filesystem.
import (
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/seqsense/s3sync"
)
func main() {
// Creates an AWS session
sess, _ := session.NewSession(&aws.Config{
Region: aws.String("us-east-1"),
})
syncManager := s3sync.New(sess)
// Sync from s3 to local
syncManager.Sync("s3://yourbucket/path/to/dir", "local/path/to/dir")
// Sync from local to s3
syncManager.Sync("local/path/to/dir", "s3://yourbucket/path/to/dir")
}
- Note: Sync from s3 to s3 is not implemented yet.
Sets the custom logger
You can set your custom logger.
import "github.com/seqsense/s3sync"
...
s3sync.SetLogger(&CustomLogger{})
...
The logger needs to implement Log and Logf methods. See the godoc for details.
Sets up the parallelism
You can configure the number of parallel jobs for sync. Default is 16.
s3sync.new(sess, s3sync.WithParallel(16)) // This is the same as default.
s3sync.new(sess, s3sync.WithParallel(1)) // You can sync one by one.
License
Apache 2.0 License. See LICENSE.
Documentation
¶
Overview ¶
Copyright 2019 SEQSENSE, Inc. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Copyright 2019 SEQSENSE, Inc. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Index ¶
- Constants
- func SetLogger(l LoggerIF)
- type LoggerIF
- type Manager
- type Option
- func WithACL(acl string) Option
- func WithContentType(mime string) Option
- func WithDelete() Option
- func WithDownloaderOptions(opts ...func(*s3manager.Downloader)) Option
- func WithDryRun() Option
- func WithParallel(n int) Option
- func WithUploaderOptions(opts ...func(*s3manager.Uploader)) Option
- func WithoutGuessMimeType() Option
- type SyncStatistics
Constants ¶
const (
// Default number of parallel file sync jobs.
DefaultParallel = 16
)
Variables ¶
This section is empty.
Functions ¶
Types ¶
type LoggerIF ¶
type LoggerIF interface {
// Log inserts a log entry. Arguments are handled in the manner
// of fmt.Print.
Log(v ...interface{})
// Log inserts a log entry. Arguments are handled in the manner
// of fmt.Printf.
Logf(format string, v ...interface{})
}
LoggerIF is the logger interface which this library requires.
type Manager ¶
type Manager struct {
// contains filtered or unexported fields
}
Manager manages the sync operation.
func (*Manager) GetStatistics ¶
func (m *Manager) GetStatistics() SyncStatistics
GetStatistics returns the structure that contains the sync statistics
type Option ¶
type Option func(*Manager)
Option is a functional option type of Manager.
func WithContentType ¶
WithContentType overwrites uploading MIME type.
func WithDelete ¶
func WithDelete() Option
WithDelete enables to delete files unexisting on source directory.
func WithDownloaderOptions ¶
func WithDownloaderOptions(opts ...func(*s3manager.Downloader)) Option
WithDownloaderOptions sets underlying s3manager's options.
func WithParallel ¶
WithParallel sets maximum number of parallel file sync jobs.
func WithUploaderOptions ¶
WithUploaderOptions sets underlying s3manager's options.
func WithoutGuessMimeType ¶
func WithoutGuessMimeType() Option
WithoutGuessMimeType disables guessing MIME type from contents.
type SyncStatistics ¶
type SyncStatistics struct {
Bytes int64
Files int64
DeletedFiles int64
// contains filtered or unexported fields
}
SyncStatistics captures the sync statistics.
Source Files
Directories