kafka

package
v0.0.0-...-2374f08 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 11, 2018 License: BSD-3-Clause Imports: 13 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type ConfluentSource

type ConfluentSource struct {
	Source
	RegistryURL string
	// contains filtered or unexported fields
}

ConfluentSource implements pdk.Source using Kafka and the Confluent schema registry.

func NewConfluentSource

func NewConfluentSource() *ConfluentSource

NewConfluentSource returns a new ConfluentSource.

func (*ConfluentSource) Record

func (s *ConfluentSource) Record() (interface{}, error)

Record returns the next value from kafka.

type Main

type Main struct {
	Hosts         []string `help:"Comma separated list of Kafka hosts and ports"`
	Topics        []string `help:"Comma separated list of Kafka topics"`
	Group         string   `help:"Kafka group"`
	RegistryURL   string   `help:"URL of the confluent schema registry. Pass an empty string to use JSON instead of Avro."`
	Framer        pdk.DashField
	PilosaHosts   []string `help:"Comma separated list of Pilosa hosts and ports."`
	Index         string   `help:"Pilosa index."`
	BatchSize     uint     `help:"Batch size for Pilosa imports (latency/throughput tradeoff)."`
	SubjectPath   []string `help:"Comma separated path to value in each record that should be mapped to column ID. Blank gets a sequential ID"`
	Proxy         string   `help:"Bind to this address to proxy and translate requests to Pilosa"`
	AllowedFields []string `help:"If any are passed, only frame names in this comma separated list will be indexed."`
	MaxRecords    int      `help:"Maximum number of records to ingest from kafka before stopping."`
	// contains filtered or unexported fields
}

Main holds the options for running Pilosa ingestion from Kafka.

func NewMain

func NewMain() *Main

NewMain returns a new Main.

func (*Main) Close

func (m *Main) Close() error

func (*Main) Run

func (m *Main) Run() error

Run begins indexing data from Kafka into Pilosa.

type Schema

type Schema struct {
	Schema  string `json:"schema"`  // The actual AVRO schema
	Subject string `json:"subject"` // Subject where the schema is registered for
	Version int    `json:"version"` // Version within this subject
	ID      int    `json:"id"`      // Registry's unique id
}

The Schema type is an object produced by the schema registry.

type Source

type Source struct {
	Hosts   []string
	Topics  []string
	Group   string
	Type    string
	MaxMsgs int
	// contains filtered or unexported fields
}

Source implements the pdk.Source interface using kafka as a data source.

func NewSource

func NewSource() *Source

NewSource gets a new Source

func (*Source) Close

func (s *Source) Close() error

Close closes the underlying kafka consumer.

func (*Source) Open

func (s *Source) Open() error

Open initializes the kafka source.

func (*Source) Record

func (s *Source) Record() (interface{}, error)

Record returns the value of the next kafka message.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL