kafka

package
v0.0.0-...-fecdc3c Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: May 5, 2021 License: Apache-2.0 Imports: 11 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source
var (
	MapperReadShard = gio.RegisterMapper(readShard)
)

Functions

This section is empty.

Types

type KafkaPartitionInfo

type KafkaPartitionInfo struct {
	Brokers        []string
	Topic          string
	Group          string
	TimeoutSeconds int
	PartitionId    int32
}

func (*KafkaPartitionInfo) ReadSplit

func (s *KafkaPartitionInfo) ReadSplit() error

type KafkaSource

type KafkaSource struct {
	Brokers        []string
	Group          string
	Topic          string
	TimeoutSeconds int
	// contains filtered or unexported fields
}

func New

func New(brokers []string, topic, group string) *KafkaSource

func (*KafkaSource) Generate

func (s *KafkaSource) Generate(f *flow.Flow) *flow.Dataset

Generate generates data shard info, partitions them via round robin, and reads each shard on each executor

func (*KafkaSource) Timeout

func (s *KafkaSource) Timeout(seconds int) *KafkaSource

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL