file

package
v0.0.0-...-2374f08 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 11, 2018 License: BSD-3-Clause Imports: 9 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Main

type Main struct {
	Path        string   `help:"File or directory path to read from."`
	PilosaHosts []string `help:"Comma separated list of Pilosa hosts and ports."`
	Index       string   `help:"Pilosa index."`
	BatchSize   uint     `help:"Batch size for Pilosa imports (latency/throughput tradeoff)."`
	Framer      pdk.DashField
	SubjectAt   string   `help:"Tells the source to add a unique 'subject' key to each record which is the filename + record number."`
	SubjectPath []string `help:"Path to value in each record that should be mapped to column ID. Blank gets a sequential ID."`
	Proxy       string   `help:"Bind to this address to proxy and translate requests to Pilosa"`
}

Main contains the configuration for an ingester with an S3 Source.

func NewMain

func NewMain() *Main

NewMain gets a new Main with the default configuration.

func (*Main) Run

func (m *Main) Run() error

Run runs the ingester.

type Source

type Source struct {
	// contains filtered or unexported fields
}

Source is a pdk.Source which reads json objects from files on disk.

func NewSource

func NewSource(opts ...SrcOption) (*Source, error)

NewSource gets a new file source which will index json data from a file or all files in a directory.

func (*Source) Record

func (s *Source) Record() (interface{}, error)

Record implements pdk.Record returning a map[string]interface{} for each json object in the source files.

type SrcOption

type SrcOption func(s *Source) error

SrcOption is a functional option for the file Source.

func OptSrcPath

func OptSrcPath(pathname string) SrcOption

OptSrcPath sets the path name for the file or directory to use for source data.

func OptSrcSubjectAt

func OptSrcSubjectAt(key string) SrcOption

OptSrcSubjectAt tells the source to add a new key to each record whose value will be <filename>#<record number>.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL