s3zip

package module
v0.0.0-...-2044f64 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 30, 2023 License: Apache-2.0 Imports: 21 Imported by: 0

README

s3zip

s3zip zip files and upload to Amazon S3.

Motivation

Ref: https://aws.amazon.com/s3/pricing/

  • Reduce the metadata:
    • For each object that is stored in S3 Glacier, Amazon S3 adds 40 KB of chargeable overhead for metadata, with 8 KB charged at S3 Standard rates and 32 KB charged at S3 Glacier Flexible Retrieval or S3 Deep Archive rates.
  • Reduce API requests:
    • Amazon S3 charges for API requests. e.g. PUT, COPY, POST, LIST requests (per 1,000 requests).

Usage

s3zip path/to/config.yaml

Config

dry_run: true

s3:
  region: us-west-2
  bucket: my-bucket
  storage_class: DEEP_ARCHIVE # STANDARD | DEEP_ARCHIVE | etc.

targets:
  - path: D:\User\Desktop\MyPictures
    max_zip_depth: 2 # 0: zip MyPictures folder, 1: zip files under MyPictures/*, 2: zip files under MyPictures/**/*
    out_prefix: s3zip # prefix for s3 object key

Documentation

Index

Constants

View Source
const MetadataKeyHash = "S3zip-Hash"

Variables

This section is empty.

Functions

func Hash

func Hash(name string) (string, error)

Hash returns a hash of the given file or directory. This only uses the file size and name for performance.

func LocalObjects

func LocalObjects(path string, maxDepth int) ([]string, error)

LocalObjects returns a list of relative paths to all files and directories. maxDepth is the maximum depth of recursion, 0 means no recursion.

func Zip

func Zip(name string) io.ReadCloser

Zip returns an uncompressed zip file of the given directory or file.

Types

type Config

type Config struct {
	DryRun  bool           `yaml:"dry_run"`
	S3      ConfigS3       `yaml:"s3"`
	Targets []ConfigTarget `yaml:"targets"`
}

func ReadConfig

func ReadConfig(name string) (*Config, error)

type ConfigS3

type ConfigS3 struct {
	Region       string `yaml:"region"`
	Bucket       string `yaml:"bucket"`
	StorageClass string `yaml:"storage_class"`
}

type ConfigTarget

type ConfigTarget struct {
	Path        string `yaml:"path"`
	MaxZipDepth int    `yaml:"max_zip_depth"`
	OutPrefix   string `yaml:"out_prefix"`
}

type RunInput

type RunInput struct {
	DryRun         bool
	S3Bucket       string
	S3Uploader     S3Uploader
	S3Service      S3Service
	Path           string
	MaxZipDepth    int
	OutPrefix      string
	S3StorageClass string
}

type RunOutput

type RunOutput struct {
	Uploaded int
	Deleted  int
}

func Run

func Run(ctx context.Context, in *RunInput) (*RunOutput, error)

Run zip and upload files in the given path.

type S3Service

type S3Service interface {
	HeadObjectWithContext(ctx context.Context, input *s3.HeadObjectInput, options ...request.Option) (*s3.HeadObjectOutput, error)
	ListObjectsV2PagesWithContext(ctx context.Context, input *s3.ListObjectsV2Input, fn func(*s3.ListObjectsV2Output, bool) bool, options ...request.Option) error
	DeleteObjectsWithContext(ctx context.Context, input *s3.DeleteObjectsInput, options ...request.Option) (*s3.DeleteObjectsOutput, error)
}

type S3Uploader

type S3Uploader interface {
	UploadWithContext(ctx context.Context, input *s3manager.UploadInput, options ...func(*s3manager.Uploader)) (*s3manager.UploadOutput, error)
}

Directories

Path Synopsis
cmd

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL