README
¶
Archive relational data to Amazon S3.
Provide a database and upload manager to upload a table to an S3 bucket.
db, _ := sql.Open(config.driver, config.url)
client := s3.New(session.NewSessionWithOptions(session.Options{}))
uploader := s3manager.NewUploaderWithClient(client)
chiv.Archive(db, uploader, "table", "bucket")
Use options to configure the archival format, upload key, null placeholder, etc.
chiv.Archive(db, uploader, "table", "bucket"
chiv.WithFormat(chiv.JSON),
chiv.WithKey("2019/september/monthly_archive.json"),
chiv.WithNull("empty"),
)
For multiple uploads using the same database and S3 clients, construct an Archiver. Options provided during
construction can be overridden in individual archival calls.
a := chiv.NewArchiver(db, uploader, chiv.WithFormat(chiv.YAML))
a.Archive("first_table", "bucket")
a.Archive("second_table", "bucket")
a.Archive("second_table", "bucket", chiv.WithFormat(chiv.JSON), chiv.WithKey("second_table.json"))
Custom queries can be archived using the ArchiveRows family of functions.
rows, _ := db.Exec("SELECT * FROM table and JOIN all the things...")
chiv.ArchiveRows(rows, uploader, "bucket")
Context-aware versions are also provided, e.g. ArchiveWithContext, ArchiveRowsWithContext, etc.
CLI
A simple CLI wrapping the package is also included.
NAME:
chiv - Archive relational data to Amazon S3
USAGE:
chiv [flags...]
VERSION:
vX.Y.Z
GLOBAL OPTIONS:
--database value, -d value database connection string [$DATABASE_URL]
--table value, -t value database table to archive
--bucket value, -b value upload S3 bucket name
--driver value, -r value database driver type: postgres or mysql (default: "postgres")
--columns value, -c value database columns to archive, comma-separated
--format value, -f value upload format: csv, yaml or json (default: "csv")
--key value, -k value upload key
--extension value, -e value upload extension
--null value, -n value upload null value
--help, -h show usage details
--version, -v print the version
Documentation
¶
Overview ¶
Package chiv archives relational database tables to Amazon S3.
Index ¶
- func Archive(db Database, s3 Uploader, table, bucket string, options ...Option) error
- func ArchiveRows(rows Rows, s3 Uploader, bucket string, options ...Option) error
- func ArchiveRowsWithContext(ctx context.Context, rows Rows, s3 Uploader, bucket string, options ...Option) error
- func ArchiveWithContext(ctx context.Context, db Database, s3 Uploader, table, bucket string, ...) error
- type Archiver
- func (a *Archiver) Archive(table, bucket string, options ...Option) error
- func (a *Archiver) ArchiveRows(rows Rows, bucket string, options ...Option) (err error)
- func (a *Archiver) ArchiveRowsWithContext(ctx context.Context, rows Rows, bucket string, options ...Option) (err error)
- func (a *Archiver) ArchiveWithContext(ctx context.Context, table, bucket string, options ...Option) (err error)
- type Column
- type Database
- type Extensioner
- type Formatter
- type FormatterFunc
- type Option
- type Rows
- type Uploader
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Archive ¶
Archive a database table to S3.
func ArchiveRows ¶
ArchiveRows to S3.
Types ¶
type Archiver ¶
type Archiver struct {
// contains filtered or unexported fields
}
Archiver archives database tables to Amazon S3.
func NewArchiver ¶
NewArchiver constructs an archiver with the given Database, S3 uploader and options. Options set on creation apply to all calls to Archive unless overridden.
func (*Archiver) Archive ¶
Archive a Database table to S3. Any options provided override those set on creation.
func (*Archiver) ArchiveRows ¶
ArchiveRows to S3.
type Column ¶
Column reports its name, database type name and scan type.
type Database ¶
type Database interface {
QueryContext(ctx context.Context, query string, args ...interface{}) (*sql.Rows, error)
}
Database queries tables for rows.
type Extensioner ¶
type Extensioner interface {
Extension() string
}
Extensioner is a Formatter that provides a default extension.
type Formatter ¶
type Formatter interface {
// Open the Formatter and perform any format-specific initialization.
Open() error
// Format and write a single record.
Format([][]byte) error
// Close the Formatter and perform any format-specific cleanup.
Close() error
}
Formatter formats and writes records. A custom Formatter may implement Extensioner to provide chiv with a default file extension.
func CSV ¶
CSV writes column headers and returns an initialized CSV formatter.
func JSON ¶
JSON opens a JSON array and returns an initialized JSON formatter.
type FormatterFunc ¶
FormatterFunc returns an initialized Formatter.
type Option ¶
type Option func(*Archiver)
Option configures the Archiver. Options can be provided when creating an Archiver or on each call to Archive.
func WithColumns ¶
WithColumns configures a list of column names to archive.
func WithExtension ¶
WithExtension configures an extension for object keys uploaded to S3.
type Rows ¶
type Rows interface {
ColumnTypes() ([]*sql.ColumnType, error)
Next() bool
Scan(...interface{}) error
Err() error
}
Rows reports its column types and is iterable.
type Uploader ¶
type Uploader interface {
UploadWithContext(ctx aws.Context, input *s3manager.UploadInput, opts ...func(*s3manager.Uploader)) (*s3manager.UploadOutput, error)
}
Uploader uploads input to S3.
Source Files
Directories