Documentation
¶
Overview ¶
Copyright © 2019 AltoStack <info@altostack.io>
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Copyright © 2019 AltoStack <info@altostack.io>
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Copyright © 2019 AltoStack <info@altostack.io>
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Index ¶
- func MarshalDynamoAttributeMap(attrs map[string]*dynamodb.AttributeValue) ([]byte, error)
- type AwsHelper
- func (h *AwsHelper) ChannelToS3(bucketName, s3Folder string, s3BufferSize int, destination *AwsHelper)
- func (h *AwsHelper) ChannelToTable(tableName string, batchSize int64, waitPeriod time.Duration)
- func (h *AwsHelper) CheckTableEmpty(tbl string) (int64, error)
- func (h *AwsHelper) DumpBuffer(bucketName, s3Folder string, buff *bytes.Buffer)
- func (h *AwsHelper) ExistsInS3(bucketName, s3Path string) (bool, error)
- func (h *AwsHelper) GetFromS3(bucketName, s3Path string) (*io.ReadCloser, error)
- func (h *AwsHelper) LoadManifestFromS3(bucketName, manifestPath string) error
- func (h *AwsHelper) ReaderToChannel(dataReader *io.ReadCloser) error
- func (h *AwsHelper) S3ToDynamo(tableName string, batchSize int64, waitPeriod time.Duration) error
- func (h *AwsHelper) TableToChannel(tableName string, batchSize int64, waitPeriod time.Duration) error
- func (h *AwsHelper) UploadToS3(bucketName, s3Key string, data []byte)
- type CustomAttributeValue
- type S3Manifest
- type S3ManifestEntry
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func MarshalDynamoAttributeMap ¶
func MarshalDynamoAttributeMap(attrs map[string]*dynamodb.AttributeValue) ([]byte, error)
MarshalDynamoAttributeMap only purpose is to create a json-format representation of the AttributeValue map without the empty fields. We wouldn't need that if the fields of the struct was set to `json:",omitempty"` and we can't use the dynamodbattribute package because you can't really do a field for field copy with it.
Types ¶
type AwsHelper ¶
type AwsHelper struct {
AwsSession client.ConfigProvider
DynamoSvc dynamodbiface.DynamoDBAPI
Wg sync.WaitGroup
DataPipe chan map[string]*dynamodb.AttributeValue
ManifestS3 S3Manifest
}
AwsHelper supports a set of helpers around DynamoDB and s3
func NewAwsHelper ¶
NewAwsHelper creates a new AwsHelper, initializing an AWS session and a few objects like a channel or a DynamoDB client
func (*AwsHelper) ChannelToS3 ¶
func (h *AwsHelper) ChannelToS3(bucketName, s3Folder string, s3BufferSize int, destination *AwsHelper)
ChannelToS3 reads from the given channel and sends the data the given bucket in files of s3BufferSize max size
func (*AwsHelper) ChannelToTable ¶
ChannelToTable puts the data from the channel into the given Dynamo table
func (*AwsHelper) CheckTableEmpty ¶
CheckTableEmpty checks if the table exists and is empty. Returns -1 if does not exist, returns the number of items if exists and -10 in case of an error. If the table status is not "ACTIVE", the returned number will be -2 for "CREATING", -4 for "UPDATING", -8 for "DELETING"
func (*AwsHelper) DumpBuffer ¶
DumpBuffer dumps the content of the given buffer to a new randomly generated file name in the given s3 path in the given bucket and resets the said buffer
func (*AwsHelper) ExistsInS3 ¶
ExistsInS3 checks that a given path in s3 exists as a file
func (*AwsHelper) GetFromS3 ¶
func (h *AwsHelper) GetFromS3(bucketName, s3Path string) (*io.ReadCloser, error)
GetFromS3 download a file from s3 to memory (as the files are small by default - just a few Mb).
func (*AwsHelper) LoadManifestFromS3 ¶
LoadManifestFromS3 downloads the given manifest file and load it in the ManifestS3 attribute of the struct
func (*AwsHelper) ReaderToChannel ¶
func (h *AwsHelper) ReaderToChannel(dataReader *io.ReadCloser) error
ReaderToChannel reads the data from a actions line by line, serializes it and sends it to the struct's channel
func (*AwsHelper) S3ToDynamo ¶
S3ToDynamo pulls the s3 files from AwsHelper.ManifestS3 and import them inside the given table using the given batch size (and wait period between each batch)
func (*AwsHelper) TableToChannel ¶
func (h *AwsHelper) TableToChannel(tableName string, batchSize int64, waitPeriod time.Duration) error
TableToChannel scans an entire DynamoDB table, putting all the output records to a given channel and increment a given wait group
func (*AwsHelper) UploadToS3 ¶
UploadToS3 writes the content of a bytes array to the given s3 path
type CustomAttributeValue ¶
type CustomAttributeValue struct {
B []byte `json:"b,omitempty"`
BOOL *bool `json:"bool,omitempty"`
BS [][]byte `json:"bs,omitempty"`
N *string `json:"n,omitempty"`
NS []*string `json:"ns,omitempty"`
NULL *bool `json:"null,omitempty"`
S *string `json:"s,omitempty"`
SS []*string `json:"ss,omitempty"`
L []*CustomAttributeValue `json:"l,omitempty"`
M map[string]*CustomAttributeValue `json:"m,omitempty"`
}
CustomAttributeValue is here to get rid of the empty fields during the json Marshaling as the dynamodb.AttributeValue don't have any json tag set
func (*CustomAttributeValue) Marshal ¶
func (attr *CustomAttributeValue) Marshal(in *dynamodb.AttributeValue)
Marshal translates a *dynamodb.AttributeValue into a *CustomAttributeValue field for field
func (*CustomAttributeValue) Unmarshal ¶
func (attr *CustomAttributeValue) Unmarshal(out *dynamodb.AttributeValue)
Unmarshal translates the current CustomAttributeValue to a *dynamodb.AttributeValue
type S3Manifest ¶
type S3Manifest struct {
Name string `json:"name"`
Version int `json:"version"`
Entries []S3ManifestEntry `json:"entries"`
}
S3Manifest represents the actions manifest stored in the s3 folder of the actions
type S3ManifestEntry ¶
S3ManifestEntry represents an entry in the actions manifest stored in the s3 folder of the actions
Source Files