blobdestination

package
v0.18.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 18, 2021 License: Apache-2.0 Imports: 13 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func New

New returns a valid Blacksmith destination.Destination for a blob storage.

Types

type Blob

type Blob struct {
	// contains filtered or unexported fields
}

Blob implements the Blacksmith destination.Destination interface for working with blob storages.

func (*Blob) Actions

func (d *Blob) Actions() map[string]destination.Action

Actions return a list of actions the destination Blob is able to handle.

func (*Blob) Init

func (d *Blob) Init(tk *destination.Toolkit) error

Init is part of the destination.WithHooks interface. It allows to properly open the connection with the bucket. It is called when starting the scheduler service.

func (*Blob) Options

func (d *Blob) Options() *destination.Options

Options returns common destination options for a blob storage. They will be shared across every actions of this destination, except when overridden.

func (*Blob) Shutdown

func (d *Blob) Shutdown(tk *destination.Toolkit) error

Shutdown is part of the destination.WithHooks interface. It allows to properly close the connection with the bucket. It is called when shutting down the scheduler service.

func (*Blob) String

func (d *Blob) String() string

String returns the string representation of the destination passed by the user when creating the Blob destination.

type Driver

type Driver string

Driver is a custom type allowing the user to only pass supported drivers when creating the destination.

var DriverAWSS3 Driver = "aws/s3"

DriverAWSS3 is used to leverage AWS S3 as the destination's driver.

Environment variables:

  • AWS_ACCESS_KEY_ID (required)
  • AWS_SECRET_ACCESS_KEY (required)
  • AWS_REGION
var DriverAzureBlob Driver = "azure/blob"

DriverAzureBlob is used to leverage Azure Blob Storage as the destination's driver.

Environment variables:

  • AZURE_STORAGE_ACCOUNT (required)
  • AZURE_STORAGE_KEY || AZURE_STORAGE_SAS_TOKEN (required)
var DriverGoogleStorage Driver = "google/storage"

DriverGoogleStorage is used to leverage Google Cloud Storage as the destination's driver.

Environment variables:

  • GOOGLE_APPLICATION_CREDENTIALS (required)

type Options

type Options struct {

	// Realtime indicates if the pubsub adapter of the Blacksmith application shall
	// be used to load events to the destination in realtime or not. When false, the
	// Interval will be used.
	Realtime bool

	// Interval represents an interval or a CRON string at which a job shall be
	// loaded to the destination. It is used as the time-lapse between retries in
	// case of a job failure.
	//
	// Defaults to "@every 1h".
	Interval string

	// MaxRetries indicates the maximum number of retries per job the scheduler will
	// attempt to execute before it succeed. When the limit is reached, the job is
	// marked as "discarded".
	//
	// Defaults to 72.
	MaxRetries uint16

	// Name indicates the identifier of the bucket to use in Blacksmith. The computed
	// name is "blob(<name>)". This does not have any consequences on the bucket's
	// name used in the cloud provider.
	//
	// Example: "mybucket"
	// Required.
	Name string

	// Driver is the driver to leverage for using this destination.
	//
	// Required.
	Driver Driver

	// Connection is the driver's specific connection string to use for writing
	// into the bucket.
	//
	// Format for AWS S3: "<bucket>"
	// Format for Azure Blob Storage: "<container>"
	// Format for Google Cloud Storage: "<bucket>"
	Connection string

	// Params can be used to add specific configuration per driver.
	//
	// Supported fields for AWS S3:
	//   url.Values{
	//     "region": {"<region>"}, // Required if environment variable 'AWS_REGION' is not set.
	//   }
	Params url.Values
}

Options is the options the destination can take as an input to be configured.

type Write

type Write struct {
	Filename string `json:"filename"`
	Content  []byte `json:"content"`
	// contains filtered or unexported fields
}

Write implements the Blacksmith destination.Action interface for the action "write". It holds the complete job's structure to load into the destination.

func (Write) Load

func (a Write) Load(tk *destination.Toolkit, queue *store.Queue, then chan<- destination.Then)

Load is the function being run by the scheduler to load the data into the destination. It is in charge of the "L" in the ETL process.

func (Write) Marshal

func (a Write) Marshal(tk *destination.Toolkit) (*destination.Job, error)

Marshal is the function being run when the action receives data into the Write receiver. It allows to transform and enrich the data before saving it in the store adapter.

func (Write) Schedule

func (a Write) Schedule() *destination.Schedule

Schedule allows the action to override the schedule options of its destination. Do not override.

func (Write) String

func (a Write) String() string

String returns the string representation of the action Write.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL