partitioner

package
v0.0.0-...-91b30c5 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 6, 2019 License: Apache-2.0 Imports: 13 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type BasePartitioner

type BasePartitioner struct {
	sync.RWMutex
	// contains filtered or unexported fields
}

BasePartitioner serves as a base struct for every Partitioner instance.

func (*BasePartitioner) Close

func (b *BasePartitioner) Close()

Close closes all the current subscriptions buffers.

func (*BasePartitioner) Rebalance

func (b *BasePartitioner) Rebalance(n *cluster.Notification)

Rebalance is called when the consumer group rebalances the topics and partitions among the cluster. This method holds the lock until it finishes looping through all the consumers.

type Config

type Config struct {
	Type                string // Type of the selected partitioner.
	BaseFolder          string
	TopicNamePrefix     string
	FieldKeyName        string
	DefaultBufferConfig *buffer.Config
}

Config for partitioner.

type DefaultPartitioner

type DefaultPartitioner struct {
	BasePartitioner
}

DefaultPartitioner partitions the data by topic and partition.

func (*DefaultPartitioner) GetBuffer

GetBuffer returns a buffer that can be used to write the message to.

func (*DefaultPartitioner) GetKey

func (d *DefaultPartitioner) GetKey(f *buffer.Flush) string

GetKey returns the file path used after a file is flushed.

type IsoDateFieldPartitioner

type IsoDateFieldPartitioner struct {
	BasePartitioner
	// contains filtered or unexported fields
}

IsoDateFieldPartitioner partitions the data in hourly buckets based on a given timestamp. The field must be a unix timestamp in milliseconds.

func (*IsoDateFieldPartitioner) GetBuffer

GetBuffer returns a buffer that can be used to write the message to.

func (*IsoDateFieldPartitioner) GetKey

GetKey returns the file path used after a file is flushed.

type Partitioner

type Partitioner interface {
	// GetBuffer returns a buffer that can be used to write the message to.
	// The buffer is decided based on the Partitioner class used.
	GetBuffer(msg *sarama.ConsumerMessage) (*buffer.Buffer, error)

	// GetKey returns the file path used after a file is flushed.
	GetKey(f *buffer.Flush) string

	// Rebalance is called when the consumer group rebalances the topics and
	// partitions among the cluster. This method holds the lock until it finishes looping
	// through all the consumers.
	Rebalance(n *cluster.Notification)

	// Close takes care of closing all the buffers.
	Close()
}

Partitioner.

func New

func New(c *Config) Partitioner

NewPartitioner creates a new object that satisfies the Partitioner interface. Uses a Worker instance for additional configuration fields.

type TimeFieldPartitioner

type TimeFieldPartitioner struct {
	BasePartitioner
	// contains filtered or unexported fields
}

TimeFieldPartitioner partitions the data in hourly buckets based on a given timestamp. The field must be a unix timestamp in milliseconds.

func (*TimeFieldPartitioner) GetBuffer

GetBuffer returns a buffer that can be used to write the message to.

func (*TimeFieldPartitioner) GetKey

func (t *TimeFieldPartitioner) GetKey(f *buffer.Flush) string

GetKey returns the file path used after a file is flushed.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL