README
¶
Storage package for the dynamodbdump backups
This package is used to organise the code a little better.
It will hold the structures and interfaces related to the storage of the backups.
Documentation
¶
Index ¶
- func Close(r io.Closer)
- func MarshalDynamoAttributeMap(attrs map[string]*dynamodb.AttributeValue) ([]byte, error)
- type BackupIface
- type CustomAttributeValue
- type FileInput
- type Manifest
- type ManifestEntry
- type S3Backup
- func (h *S3Backup) DumpBuffer(input *FileInput, buff *bytes.Buffer)
- func (h *S3Backup) Exists(input *FileInput) (bool, error)
- func (h *S3Backup) Flush(input *FileInput, data []byte) error
- func (h *S3Backup) GetFile(input *FileInput) (*io.ReadCloser, error)
- func (h *S3Backup) LoadManifest(input *FileInput) error
- func (h *S3Backup) Scan(dataReader *io.ReadCloser) error
- func (h *S3Backup) Write(input *FileInput, s3BufferSize int, wg *sync.WaitGroup)
- func (h *S3Backup) WriteToDB(tableName string, batchSize int64, waitPeriod time.Duration, ...) error
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func MarshalDynamoAttributeMap ¶
func MarshalDynamoAttributeMap(attrs map[string]*dynamodb.AttributeValue) ([]byte, error)
MarshalDynamoAttributeMap only purpose is to create a json-format representation of the AttributeValue map without the empty fields. We wouldn't need that if the fields of the struct was set to `json:",omitempty"` and we can't use the dynamodbattribute package because you can't really do a field for field copy with it.
Types ¶
type BackupIface ¶
type BackupIface interface {
LoadManifest(*FileInput) error
GetFile(*FileInput) (*io.ReadCloser, error)
Exists(input *FileInput) (bool, error)
Flush(input *FileInput, data []byte) error
Scan(*io.ReadCloser) error
WriteToDB(string, int64, time.Duration, *sync.WaitGroup) error
DumpBuffer(*FileInput, *bytes.Buffer)
Write(*FileInput, int, *sync.WaitGroup)
}
BackupIface is the interface that each storage backends implement
type CustomAttributeValue ¶
type CustomAttributeValue struct {
B []byte `json:"b,omitempty"`
BOOL *bool `json:"bool,omitempty"`
BS [][]byte `json:"bs,omitempty"`
N *string `json:"n,omitempty"`
NS []*string `json:"ns,omitempty"`
NULL *bool `json:"null,omitempty"`
S *string `json:"s,omitempty"`
SS []*string `json:"ss,omitempty"`
L []*CustomAttributeValue `json:"l,omitempty"`
M map[string]*CustomAttributeValue `json:"m,omitempty"`
}
CustomAttributeValue is here to get rid of the empty fields during the json Marshaling as the dynamodb.AttributeValue don't have any json tag set
func (*CustomAttributeValue) Marshal ¶
func (attr *CustomAttributeValue) Marshal(in *dynamodb.AttributeValue)
Marshal translates a *dynamodb.AttributeValue into a *CustomAttributeValue field for field
func (*CustomAttributeValue) Unmarshal ¶
func (attr *CustomAttributeValue) Unmarshal(out *dynamodb.AttributeValue)
Unmarshal translates the current CustomAttributeValue to a *dynamodb.AttributeValue
type FileInput ¶
type FileInput struct {
Bucket, Path *string
}
FileInput is used as input for the functions that require a file definition, be it a s3 file or a file on disk. Bucket will be used only for S3-related functions. Path will be the Key in the context of S3 functions
type Manifest ¶
type Manifest struct {
Name string `json:"name"`
Version int `json:"version"`
Entries []ManifestEntry `json:"entries"`
}
Manifest represents the backup manifest
type ManifestEntry ¶
ManifestEntry represents an entry in the backup manifest
type S3Backup ¶
type S3Backup struct {
BackupIface
DataPipe chan map[string]*dynamodb.AttributeValue
// contains filtered or unexported fields
}
S3Backup is the structure that implements the backup storage for the S3 backend
func NewS3Backup ¶
func NewS3Backup(sess client.ConfigProvider) *S3Backup
NewS3Backup initlialiaes the s3 client and returns a pointer to a S3Backup struct
func (*S3Backup) DumpBuffer ¶
DumpBuffer dumps the content of the given buffer to a new randomly generated file name in the given s3 path in the given bucket and resets the said buffer
func (*S3Backup) GetFile ¶
func (h *S3Backup) GetFile(input *FileInput) (*io.ReadCloser, error)
GetFile downloads a file from s3 to memory (as the files are small by default - just a few Mb).
func (*S3Backup) LoadManifest ¶
LoadManifest downloads the given manifest file and load it in the Manifest attribute of the struct
func (*S3Backup) Scan ¶
func (h *S3Backup) Scan(dataReader *io.ReadCloser) error
Scan reads the data from a backup line by line, serializes it and sends it to the struct's channel
Source Files