README
¶
s3sync
Golang utility for syncing between s3 and local
Usage
Use New to create a manager, and Sync function syncs between s3 and local filesystem.
import (
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/seqsense/s3sync"
)
func main() {
// Creates an AWS session
sess, _ := session.NewSession(&aws.Config{
Region: aws.String("us-east-1"),
})
syncManager := s3sync.New(sess)
// Sync from s3 to local
syncManager.Sync("s3://yourbucket/path/to/dir", "local/path/to/dir")
// Sync from local to s3
syncManager.Sync("local/path/to/dir", "s3://yourbucket/path/to/dir")
}
- Note: Sync from s3 to s3 is not implemented yet.
Sets the custom logger
You can set your custom logger.
import "github.com/seqsense/s3sync"
...
s3sync.SetLogger(&CustomLogger{})
...
The logger needs to implement Log and Logf methods. See the godoc for details.
Sets up the parallelism
You can configure the number of parallel jobs for sync. Default is 16.
s3sync.new(sess, s3sync.WithParallel(16)) // This is the same as default.
s3sync.new(sess, s3sync.WithParallel(1)) // You can sync one by one.
License
Apache 2.0 License. See LICENSE.
Documentation
¶
Overview ¶
Copyright 2019 SEQSENSE, Inc. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Copyright 2019 SEQSENSE, Inc. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Index ¶
- Constants
- func SetLogger(l LoggerIF)
- type LoggerIF
- type Manager
- type Option
- func WithACL(acl string) Option
- func WithCacheControl(value string) Option
- func WithContentType(mime string) Option
- func WithContentTypeSelector(contentTypeSelector func(string) (string, error)) Option
- func WithDelete() Option
- func WithDownloaderOptions(opts ...func(*s3manager.Downloader)) Option
- func WithDryRun() Option
- func WithParallel(n int) Option
- func WithSoftDelete() Option
- func WithUploaderOptions(opts ...func(*s3manager.Uploader)) Option
- func WithoutGuessMimeType() Option
Constants ¶
const (
// Default number of parallel file sync jobs.
DefaultParallel = 16
)
Variables ¶
This section is empty.
Functions ¶
Types ¶
type LoggerIF ¶
type LoggerIF interface {
// Log inserts a log entry. Arguments are handled in the manner
// of fmt.Print.
Log(v ...interface{})
// Log inserts a log entry. Arguments are handled in the manner
// of fmt.Printf.
Logf(format string, v ...interface{})
}
LoggerIF is the logger interface which this library requires.
type Manager ¶
type Manager struct {
// contains filtered or unexported fields
}
Manager manages the sync operation.
type Option ¶
type Option func(*Manager)
Option is a functional option type of Manager.
func WithCacheControl ¶ added in v1.8.4
WithCacheControl sets the Cache-Control header for uploaded files as described in https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control
func WithContentType ¶
WithContentType overwrites uploading MIME type.
func WithContentTypeSelector ¶ added in v1.8.3
WithContentTypeSelector sets a MIME type selector into the Manager.
func WithDelete ¶
func WithDelete() Option
WithDelete enables to delete files unexisting on source directory.
func WithDownloaderOptions ¶
func WithDownloaderOptions(opts ...func(*s3manager.Downloader)) Option
WithDownloaderOptions sets underlying s3manager's options.
func WithParallel ¶
WithParallel sets maximum number of parallel file sync jobs.
func WithSoftDelete ¶ added in v1.8.4
func WithSoftDelete() Option
WithSoftDelete removes files with a delay
func WithUploaderOptions ¶
WithUploaderOptions sets underlying s3manager's options.
func WithoutGuessMimeType ¶
func WithoutGuessMimeType() Option
WithoutGuessMimeType disables guessing MIME type from contents.
Source Files
Directories