producer

package module
v0.0.0-...-86e5513 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 1, 2015 License: MIT Imports: 7 Imported by: 0

README

kinesis_producer

Build Status GoDoc

A high level asynchronous kinesis stream producer that handles buffering and sending to multiple kinesis streams

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type KinesisError

type KinesisError struct {
	ErrorCode    string
	ErrorMessage string
	AWSError     error
	Message      *KinesisMessage
}

KinesisError TODO

func (KinesisError) Error

func (e KinesisError) Error() string

func (KinesisError) Log

func (e KinesisError) Log(msg string)

Log error level output

type KinesisMessage

type KinesisMessage struct {
	Data         []byte
	Stream       string
	PartitionKey string
	Metadata     interface{}
}

KinesisMessage TODO

func (KinesisMessage) Log

func (k KinesisMessage) Log(msg string)

Log debug level

type KinesisProducer

type KinesisProducer interface {
	Close()
	Input() chan<- *KinesisMessage
	Successes() <-chan *KinesisMessage
	Errors() <-chan *KinesisError
}

KinesisProducer a high level kinesis producer. Handles sending to multiple streams and batching in an async manner.

type KinesisProducerConfig

type KinesisProducerConfig struct {
	// max time that should elapse before sending buffered requests to kinesis
	FlushFrequency *goprimitives.Duration `json:"flushFrequency"`
	// min num number of buffered bytes to have before sending a request to kinesis
	FlushBytes int `json:"flushBytes"`
	// min number of buffered requests before sending a request to kinesis
	FlushMessages int `json:"flushMessages"`
	// max number of buffered requests before sending a request to kinesis
	FlushMaxMessages int `json:"flushMaxMessages"`
	// max number of outstanding requests to a single kinesis stream
	MaxOpenRequests int `json:"maxOpenRequests"`
	// max size of the data blob (kinesis record) that can be sent to kinesis (no more then 50KB)
	MaxMessageBytes int `json:"maxMessageBytes"`
	// run the AWS client in debug mode
	AWSDebugMode bool `json:"awsDebugMode"`
	// max number of attempts when sending a request to kinesis
	MaxRetries int `json:"maxRetries"`
	// the size of internal buffers
	BufferSize int `json:"bufferSize"`
	// should the producer return successfully acknowledged messages from kinesis (you must read
	// from the Successes() channel if this is true
	AckSuccess bool `json:"AckSuccesses"`
}

KinesisProducerConfig for a kinesis producer (high level, stream, etc.)

type KinesisStreamManager

type KinesisStreamManager struct {
	// contains filtered or unexported fields
}

KinesisStreamManager manages multiple KinesisStreamProducers and implements the KinesisProduver interface.

func NewKinesisStreamManager

func NewKinesisStreamManager(opts *KinesisProducerConfig) (manager *KinesisStreamManager)

NewKinesisStreamManager creates a new kinesis manager. It handles: batching messages

func (*KinesisStreamManager) Close

func (m *KinesisStreamManager) Close()

Close TODO

func (*KinesisStreamManager) Errors

func (m *KinesisStreamManager) Errors() <-chan *KinesisError

Errors TODO

func (*KinesisStreamManager) Input

func (m *KinesisStreamManager) Input() chan<- *KinesisMessage

Input TODO

func (*KinesisStreamManager) Successes

func (m *KinesisStreamManager) Successes() <-chan *KinesisMessage

Successes TODO

type KinesisStreamProducer

type KinesisStreamProducer struct {
	// contains filtered or unexported fields
}

KinesisStreamProducer producer for a single stream. Handles batching in an async manner. Implements the KinesisProducer interface.

func NewStreamProducer

func NewStreamProducer(opts *KinesisProducerConfig) (producer *KinesisStreamProducer)

NewStreamProducer TODO

func (*KinesisStreamProducer) Close

func (p *KinesisStreamProducer) Close()

Close this closes the producer. It must be closed before the producer passes out of scope as you will leave dangling goroutines, memory, etc.

func (*KinesisStreamProducer) Errors

func (p *KinesisStreamProducer) Errors() <-chan *KinesisError

Errors TODO

func (*KinesisStreamProducer) Input

func (p *KinesisStreamProducer) Input() chan<- *KinesisMessage

Input TODO

func (*KinesisStreamProducer) Successes

func (p *KinesisStreamProducer) Successes() <-chan *KinesisMessage

Successes TODO

func (*KinesisStreamProducer) ValidateStream

func (p *KinesisStreamProducer) ValidateStream(streamName string) error

ValidateStream checks if the stream exists (or even if you have a valid AWS connection) and returns and error if the stream does not exist TODO: this should page through results

type KinesisStreamProducerError

type KinesisStreamProducerError string

KinesisStreamProducerError internal error within the producer

func (KinesisStreamProducerError) Error

Error TODO

Directories

Path Synopsis
Godeps
_workspace/src/github.com/awslabs/aws-sdk-go/aws
Package aws provides core functionality for making requests to AWS services.
Package aws provides core functionality for making requests to AWS services.
_workspace/src/github.com/awslabs/aws-sdk-go/aws/awserr
Package of API error interface accessors for the SDK.
Package of API error interface accessors for the SDK.
_workspace/src/github.com/awslabs/aws-sdk-go/aws/credentials
Package credentials provides credential retrieval and management The Credentials is the primary method of getting access to and managing credentials Values.
Package credentials provides credential retrieval and management The Credentials is the primary method of getting access to and managing credentials Values.
Package of API error types.
_workspace/src/github.com/awslabs/aws-sdk-go/service/kinesis
Package kinesis provides a client for Amazon Kinesis.
Package kinesis provides a client for Amazon Kinesis.
_workspace/src/github.com/awslabs/aws-sdk-go/service/kinesis/kinesisiface
Package kinesisiface provides an interface that satisfies the kinesis service.
Package kinesisiface provides an interface that satisfies the kinesis service.
_workspace/src/github.com/stretchr/testify/assert
Package assert provides a set of comprehensive testing tools for use with the normal Go testing system.
Package assert provides a set of comprehensive testing tools for use with the normal Go testing system.
_workspace/src/github.com/suicidejack/throttled
Package throttled implements various helpers to manage the lifecycle of goroutines.
Package throttled implements various helpers to manage the lifecycle of goroutines.
_workspace/src/github.com/vaughan0/go-ini
Package ini provides functions for parsing INI configuration files.
Package ini provides functions for parsing INI configuration files.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL