pdk: github.com/pilosa/pdk Index | Files | Directories

package pdk

import "github.com/pilosa/pdk"

package pdk is the Pilosa Development Kit! It contains various helper functions and documentation to assist in using pilosa.

Of principal importance in the PDK is the ingest pipeline. Interfaces and basic implementations of each stage listed below are included in the PDK, and a number of more sophisticated implementations which may rely on other software are in sub-packages of the PDK.

1. Source

A pdk.Source is at the beginning of every indexing journey. We know
you, and we know your data is everywhere - S3 buckets, local files, Kafka
topics, hard-coded in tests, SQL databases, document DBs, triple stores.
Different Sources know how to interact with the various systems holding
your data, and get it out, one piece at a time, all wrapped up behind one
convenient interface. To write a new Source, simply implement the Source
interface, returning whatever comes naturally from the underlying client
library or API with which you are interacting. It is not the job of the
source to manipulate or massage the data in any way - that job falls to
the Parser which is the next stage of the ingestion journey. The reason
for this separation is twofold: first, you may get the same type of data
from many different sources, so it may be convenient to couple one parser
to several different sources. Secondly, you may require different
concurrency or scaling properties from fetching the data vs parsing it.
For example, if you are interacting with an HTTP endpoint at significant
latency, you may want many routines issuing concurrent calls in order to
achieve the desired throughput, but parsing is relatively lightweight, and
a single routine is sufficient to process the load.

2. Parser

The Parser does the heavy lifting for turning some arbitrary type of data
into something slightly more structured, recognizable, and type-safe.
There are many choices to be made when indexing data in Pilosa around
tradeoffs like speed vs precision, or storage size. When to use bucketing
vs range encoding, when time quantum support is needed and at what
granularity, etc. These things are not the job of the Parser. The Parser
should only get the data into a regular format so that the Mapper can make
those tradeoffs without having to worry excessively over decoding the
data. The Parser must convert incoming data into an RDF-triple like
representation using a handful of supported basic values detailed in
entity.go. Determining how to collapse (e.g.) arbitrary JSON data
into this format is not a trivial task, and indeed there may be multiple
ways to go about it and so it is possible that multiple parsers may exist
which operate on the same type of Source data.

2.5. Transformer

One may optionally provide a number of Transformers which do in-place
operations on the Entity before it is passed to the Mapper.

3. Mapper

The Mapper's job is to take instances of pdk.Entity and create
pdk.PilosaRecord objects. Because the pdk.Entity is fairly well-defined,
it is possible to do this generically, and it may not be necessary to use
a bespoke Mapper in many cases. However, as mentioned in the Parser
description, there are performance and capability tradeoffs based on how
one decides to map data into Pilosa. (TODO expand with more examples as
mappers are implemented, also reference generic mapper and its config
options)

4. Indexer

The Indexer is responsible for getting data into Pilosa. Primarily, there
is a latency/throughput tradeoff depending on the batch size selected.

Index

Package Files

bytes.go doc.go entity.go filesplitter.go framer.go ingest.go map.go mapper.go nexter.go parse.go parser.go pilosa.go pipeline.go proxy.go source.go statlogiface.go transformer.go translator.go walker.go

Variables

var (
    ErrPathNotFound   = errors.New("path not found in Entity")
    ErrNotALiteral    = errors.New("value at path is not a Literal")
    ErrEmptyPath      = errors.New("path is empty")
    ErrUnexpectedType = errors.New("unexpected type")
)

func GetFields Uses

func GetFields(body []byte) ([]string, error)

GetFields interprets body as pql queries and then tries to determine the field of each. Some queries do not have fields, and the empty string will be returned for these.

func Int64ize Uses

func Int64ize(val Literal) int64

func NewIntField Uses

func NewIntField(index *gopilosa.Index, name string, min, max int64) *gopilosa.Field

func NewPilosaForwarder Uses

func NewPilosaForwarder(phost string, t Translator, colTranslator ...FieldTranslator) *pilosaForwarder

NewPilosaForwarder returns a new pilosaForwarder which forwards all requests to `phost`. It inspects pilosa responses and runs the row ids through the Translator `t` to translate them to whatever they were mapped from.

func NewPilosaProxy Uses

func NewPilosaProxy(host string, client *http.Client) *pilosaProxy

NewPilosaProxy returns a pilosaProxy based on `host` and `client`.

func NewRankedField Uses

func NewRankedField(index *gopilosa.Index, name string, size int) *gopilosa.Field

func NexterStartFrom Uses

func NexterStartFrom(s uint64) func(n *Nexter)

NexterStartFrom returns an option which makes a Nexter start from integer "s".

func StartMappingProxy Uses

func StartMappingProxy(bind string, h http.Handler) error

StartMappingProxy listens for incoming http connections on `bind` and uses h to handle all requests. This function does not return unless there is a problem (like http.ListenAndServe).

func ToBytes Uses

func ToBytes(l Literal) []byte

ToBytes converts a literal into a typed byte slice representation.

func ToString Uses

func ToString(l Literal) string

ToString converts a Literal into a string with a type byte prepended.

func Walk Uses

func Walk(e *Entity, call func(path []string, l Literal) error) error

Walk recursively visits every Object in the Entity and calls "call" with every Literal and it's path.

type AttrMapper Uses

type AttrMapper struct {
    Mapper  Mapper
    Parsers []Parser
    Fields  []int
}

AttrMapper is a struct for mapping some set of data fields to a value for sending to Pilosa as a SetColumnAttrs query

type B Uses

type B bool

func (B) MarshalJSON Uses

func (B B) MarshalJSON() ([]byte, error)

type BinaryFloatMapper Uses

type BinaryFloatMapper struct {
    Min      float64
    Max      float64
    BitDepth int
    // contains filtered or unexported fields
}

BinaryFloatMapper is a Mapper for float types, mapping to a set of buckets representing the value in a binary sense

func (BinaryFloatMapper) ID Uses

func (m BinaryFloatMapper) ID(fi ...interface{}) (rowIDs []int64, err error)

ID maps floats to binary column sets

type BinaryIntMapper Uses

type BinaryIntMapper struct {
    Min      int64
    Max      int64
    BitDepth int
    // contains filtered or unexported fields
}

BinaryIntMapper is a Mapper for int types, mapping to a set of buckets representing the value in a binary sense

func (BinaryIntMapper) ID Uses

func (m BinaryIntMapper) ID(ii ...interface{}) (rowIDs []int64, err error)

ID maps floats to binary column sets

type BlankSubjecter Uses

type BlankSubjecter struct{}

BlankSubjecter is a Subjecter which always returns an empty subject. Typically this means that a sequential ID will be generated for each record.

func (BlankSubjecter) Subject Uses

func (b BlankSubjecter) Subject(d interface{}) (string, error)

Subject implements Subjecter, and always returns an empty string and nil error.

type BoolMapper Uses

type BoolMapper struct {
}

BoolMapper is a trivial Mapper for boolean types

func (BoolMapper) ID Uses

func (m BoolMapper) ID(bi ...interface{}) (rowIDs []int64, err error)

ID maps a bool to a rowID (identity mapper)

type Bytes Uses

type Bytes uint64

Bytes is a wrapper type for numbers which represent bytes. It provides a String method which produces sensible readable output like 1.2G or 4M, etc.

func (Bytes) String Uses

func (b Bytes) String() string

Returns a human-readable byte string of the form 10M, 12.5K, and so forth. The following units are available:

T: Terabyte
G: Gigabyte
M: Megabyte
K: Kilobyte
B: Byte

The unit that results in the smallest number greater than or equal to 1 is always chosen.

type CollapsingMapper Uses

type CollapsingMapper struct {
    Translator    Translator
    ColTranslator FieldTranslator
    Framer        Framer
    Nexter        INexter
}

CollapsingMapper processes Entities into PilosaRecords by walking the tree of properties and collapsing every path down to a concrete value into a single property name.

func NewCollapsingMapper Uses

func NewCollapsingMapper() *CollapsingMapper

NewCollapsingMapper returns a CollapsingMapper with basic implementations of its components. In order to track mapping of Pilosa columns to records, you must replace the ColTranslator with something other than a NexterFieldTranslator which just allocates ids and does not store a mapping.

func (*CollapsingMapper) Map Uses

func (m *CollapsingMapper) Map(e *Entity) (PilosaRecord, error)

Map implements the RecordMapper interface.

type ColumnMapper Uses

type ColumnMapper struct {
    Field   string
    Mapper  Mapper
    Parsers []Parser
    Fields  []int
}

ColumnMapper is a struct for mapping some set of data fields to a (field, id) combination for sending to Pilosa as a SetBit query

type Context Uses

type Context map[string]interface{}

type CustomMapper Uses

type CustomMapper struct {
    Func   func(...interface{}) interface{}
    Mapper Mapper
}

CustomMapper is a Mapper that applies a function to a slice of fields, then applies a simple Mapper to the result of that, returning a rowID. This is a generic way to support mappings which span multiple fields. It is not supported by the importing config system.

func (CustomMapper) ID Uses

func (m CustomMapper) ID(fields ...interface{}) (rowIDs []int64, err error)

ID maps a set of fields using a custom function

type DashField Uses

type DashField struct {
    Ignore   []string `help:"Do not index paths containing any of these components"`
    Collapse []string `help:"Remove these components from the path before getting field."`
}

DashField creates a field name from the path by joining the path elements with the "-" character.

func (*DashField) Field Uses

func (d *DashField) Field(path []string) (string, error)

Field gets a field from a path by joining the path elements with dashes.

type DayOfMonthMapper Uses

type DayOfMonthMapper struct {
}

func (DayOfMonthMapper) ID Uses

func (m DayOfMonthMapper) ID(ti ...interface{}) (rowIDs []int64, err error)

ID maps a timestamp to a day of month bucket (1-31)

type DayOfWeekMapper Uses

type DayOfWeekMapper struct {
}

DayOfWeekMapper is a Mapper for timestamps, mapping the day of week only

func (DayOfWeekMapper) ID Uses

func (m DayOfWeekMapper) ID(ti ...interface{}) (rowIDs []int64, err error)

ID maps a timestamp to a day of week bucket

type Entity Uses

type Entity struct {
    Subject IRI `json:"@id"`
    Objects map[Property]Object
}

Entity is the "root" node of a graph branching out from a certain resource denoted by the Subject. This is a convenience vs just handling a list of Triples as we expect to structure indexing around a particular class of thing which we ingest many instances of as records.

func NewEntity Uses

func NewEntity() *Entity

NewEntity returns a newly allocated Entity.

func (*Entity) Equal Uses

func (e *Entity) Equal(e2 *Entity) error

func (*Entity) F64 Uses

func (e *Entity) F64(path ...string) (F64, error)

F64 tries to get a float64 at the given path in the Entity.

func (*Entity) Literal Uses

func (e *Entity) Literal(path ...string) (Literal, error)

Literal gets the literal at the path in the Entity.

func (*Entity) MarshalJSON Uses

func (e *Entity) MarshalJSON() ([]byte, error)

MarshalJSON is a custom JSON marshaler for Entity objects to ensure that they serialize to valid JSON-LD (https://json-ld.org/ spec/latest/json-ld/). This allows for easy (if not particularly performant) interoperation with other variants of RDF linked data.

func (*Entity) SetPath Uses

func (e *Entity) SetPath(path ...string) (*Entity, error)

SetPath ensures that a path exists, creating Entities along the way if necessary. If it encounters a non-Entity, it will return an error.

func (*Entity) SetString Uses

func (e *Entity) SetString(value string, path ...string) error

SetString

type EntitySubjecter Uses

type EntitySubjecter interface {
    Subject(e *Entity) (string, error)
}

EntitySubjecter is an alternate interface for getting the Subject of a record, which operates on the parsed Entity rather than the unparsed data.

type EntityWithContext Uses

type EntityWithContext struct {
    Entity
    Context Context `json:"@context"`
}

EntityWithContext associates a Context (https://json-ld.org/spec/latest/json-ld/#the-context) with an Entity so that it can be Marshaled to valid and useful JSON-LD.

type F32 Uses

type F32 float32

func (F32) MarshalJSON Uses

func (F F32) MarshalJSON() ([]byte, error)

type F64 Uses

type F64 float64

func (F64) MarshalJSON Uses

func (F F64) MarshalJSON() ([]byte, error)

type FieldTranslator Uses

type FieldTranslator interface {
    Get(id uint64) (interface{}, error)
    GetID(val interface{}) (uint64, error)
}

FieldTranslator works like a Translator, but the methods don't take fields as arguments. Typically a Translator will include a FieldTranslator for each field.

type FileFragment Uses

type FileFragment struct {
    // contains filtered or unexported fields
}

FileFragment implements io.ReadCloser for part of a file.

func NewFileFragment Uses

func NewFileFragment(f *os.File, startLoc, endLoc int64) (*FileFragment, error)

NewFileFragment returns a FileFragment which will read only from startLoc to endLoc in a file.

func SplitFileLines Uses

func SplitFileLines(f *os.File, numParts int64) ([]*FileFragment, error)

SplitFileLines returns a slice of file fragments which is numParts in length. Each FileFragment will read a different section of the file, but the split points are guaranteed to be on line breaks.

func (*FileFragment) Close Uses

func (ff *FileFragment) Close() error

Close implements io.Closer for a FileFragment.

func (*FileFragment) Read Uses

func (ff *FileFragment) Read(b []byte) (n int, err error)

Read implements io.Reader for FileFragment.

type FloatMapper Uses

type FloatMapper struct {
    Buckets []float64 // slice representing bucket intervals [left0 left1 ... leftN-1 rightN-1]
    // contains filtered or unexported fields
}

FloatMapper is a Mapper for float types, mapping to arbitrary buckets

func (FloatMapper) ID Uses

func (m FloatMapper) ID(fi ...interface{}) (rowIDs []int64, err error)

ID maps floats to arbitrary buckets

type FloatParser Uses

type FloatParser struct {
}

FloatParser is a parser for float types

func (FloatParser) Parse Uses

func (p FloatParser) Parse(field string) (result interface{}, err error)

Parse parses a float string to a float64 value

type Framer Uses

type Framer interface {
    // The Field method should return an empty string and a nil error if the value
    // at the given path should be ignored. It should return an error, only if
    // something unexpected has occurred which means the record cannot be properly
    // processed.
    Field(path []string) (field string, err error)
}

Framer is an interface for extracting field names from paths denoted by []string. The path could be (e.g.) a list of keys in a nested map which arrives at a non-container value (string, int, etc).

type FramerFunc Uses

type FramerFunc func([]string) (string, error)

FramerFunc is similar to http.HandlerFunc in that you can make a bare function satisfy the Framer interface by doing FramerFunc(yourfunc).

func (FramerFunc) Field Uses

func (f FramerFunc) Field(path []string) (string, error)

Field on FramerFunc simply calls the wrapped function.

type GenericParser Uses

type GenericParser struct {
    Subjecter       Subjecter
    EntitySubjecter EntitySubjecter
    SubjectAll      bool

    // Strict controls whether failure to parse a single value or key will cause
    // the entire record to fail.
    Strict bool

    Stats Statter
    Log   Logger
}

GenericParser tries to make no assumptions about the value passed to its Parse method. At the top level it accepts a map or struct (or pointer or interface holding one of these). It will only parse exported fields on structs.

func NewDefaultGenericParser Uses

func NewDefaultGenericParser() *GenericParser

NewDefaultGenericParser returns a GenericParser with basic implementations of its components. In order to track the mapping of Pilosa columns to records, you must replace the Subjecter with something other than a BlankSubjecter.

func (*GenericParser) Parse Uses

func (m *GenericParser) Parse(data interface{}) (e *Entity, err error)

Parse of the GenericParser tries to parse any value into a pdk.Entity.

type GridMapper Uses

type GridMapper struct {
    Xmin float64
    Xmax float64
    Xres int64
    Ymin float64
    Ymax float64
    Yres int64
    // contains filtered or unexported fields
}

GridMapper is a Mapper for a 2D grid (e.g. small-scale latitude/longitude)

func (GridMapper) ID Uses

func (m GridMapper) ID(xyi ...interface{}) (rowIDs []int64, err error)

ID maps pairs of floats to regular buckets

type GridToFloatMapper Uses

type GridToFloatMapper struct {
    // contains filtered or unexported fields
}

func NewGridToFloatMapper Uses

func NewGridToFloatMapper(gm GridMapper, lfm LinearFloatMapper, gridVals []float64) GridToFloatMapper

func (GridToFloatMapper) ID Uses

func (m GridToFloatMapper) ID(vals ...interface{}) ([]int64, error)

type I Uses

type I int

func (I) MarshalJSON Uses

func (I I) MarshalJSON() ([]byte, error)

func (*I) UnmarshalJSON Uses

func (i *I) UnmarshalJSON(b []byte) error

type I16 Uses

type I16 int16

func (I16) MarshalJSON Uses

func (I I16) MarshalJSON() ([]byte, error)

func (*I16) UnmarshalJSON Uses

func (i *I16) UnmarshalJSON(b []byte) error

type I32 Uses

type I32 int32

func (I32) MarshalJSON Uses

func (I I32) MarshalJSON() ([]byte, error)

func (*I32) UnmarshalJSON Uses

func (i *I32) UnmarshalJSON(b []byte) error

type I64 Uses

type I64 int64

func (I64) MarshalJSON Uses

func (I I64) MarshalJSON() ([]byte, error)

func (*I64) UnmarshalJSON Uses

func (i *I64) UnmarshalJSON(b []byte) error

type I8 Uses

type I8 int8

func (I8) MarshalJSON Uses

func (I I8) MarshalJSON() ([]byte, error)

func (*I8) UnmarshalJSON Uses

func (i *I8) UnmarshalJSON(b []byte) error

type INexter Uses

type INexter interface {
    Next() uint64
    Last() uint64
}

INexter is the horribly named interface for threadsafe, monotonic, sequential, unique ID generation.

type IPParser Uses

type IPParser struct {
}

IPParser is a parser for IP addresses

func (IPParser) Parse Uses

func (p IPParser) Parse(field string) (result interface{}, err error)

Parse parses an IP string into a TODO

type IRI Uses

type IRI string

IRI is either a full IRI, or will map to one when the record in which it is contained is processed in relation to a context: (https://json-ld.org/spec/latest/json-ld/#the-context)

type Index Uses

type Index struct {
    // contains filtered or unexported fields
}

func (*Index) AddColumn Uses

func (i *Index) AddColumn(field string, col, row uint64OrString)

AddColumn adds a column to be imported to Pilosa.

func (*Index) AddColumnTimestamp Uses

func (i *Index) AddColumnTimestamp(field string, row, col uint64OrString, ts time.Time)

AddColumnTimestamp adds a column to be imported to Pilosa with a timestamp.

func (*Index) AddValue Uses

func (i *Index) AddValue(fieldName string, col uint64OrString, val int64)

AddValue adds a value to be imported to Pilosa.

func (*Index) Client Uses

func (i *Index) Client() *gopilosa.Client

Client returns a Pilosa client.

func (*Index) Close Uses

func (i *Index) Close() error

Close ensures that all ongoing imports have finished and cleans up internal state.

type Indexer Uses

type Indexer interface {
    AddColumn(field string, col, row uint64OrString)
    AddColumnTimestamp(field string, col, row uint64OrString, ts time.Time)
    AddValue(field string, col uint64OrString, val int64)
    // AddRowAttr(field string, row uint64, key string, value AttrVal)
    // AddColAttr(col uint64, key string, value AttrVal)
    Close() error
    Client() *gopilosa.Client
}

Indexer puts stuff into Pilosa.

func SetupPilosa Uses

func SetupPilosa(hosts []string, indexName string, schema *gopilosa.Schema, batchsize uint, options ...PilosaOption) (Indexer, error)

type Ingester Uses

type Ingester struct {
    ParseConcurrency int

    Transformers  []Transformer
    AllowedFields map[string]bool

    Stats Statter
    Log   Logger
    // contains filtered or unexported fields
}

Ingester combines a Source, Parser, Mapper, and Indexer, and uses them to ingest data into Pilosa. This could be a streaming situation where the Source never ends, and calling it just waits for more data to be available, or a batch situation where the Source eventually returns io.EOF (or some other error), and the Ingester completes (after the other components are done).

func NewIngester Uses

func NewIngester(source Source, parser RecordParser, mapper RecordMapper, indexer Indexer) *Ingester

NewIngester gets a new Ingester.

func (*Ingester) Run Uses

func (n *Ingester) Run() error

Run runs the ingest.

type IntMapper Uses

type IntMapper struct {
    Min int64
    Max int64
    Res int64 // number of bins
    // contains filtered or unexported fields
}

IntMapper is a Mapper for integer types, mapping each int in the range to a row

func (IntMapper) ID Uses

func (m IntMapper) ID(ii ...interface{}) (rowIDs []int64, err error)

ID maps an int range to a rowID range

type IntParser Uses

type IntParser struct {
}

IntParser is a parser for integer types

func (IntParser) Parse Uses

func (p IntParser) Parse(field string) (result interface{}, err error)

Parse parses an integer string to an int64 value

type KeyMapper Uses

type KeyMapper interface {
    MapRequest(body []byte) ([]byte, error)
    MapResult(field string, res interface{}) (interface{}, error)
}

KeyMapper describes the functionality for mapping the keys contained in requests and responses.

type LinearFloatMapper Uses

type LinearFloatMapper struct {
    Min   float64
    Max   float64
    Res   float64
    Scale string // linear, logarithmic
    // contains filtered or unexported fields
}

LinearFloatMapper is a Mapper for float types, mapping to regularly spaced buckets TODO: consider defining this in terms of a linear mapping ID = floor(a*value + b)

func (LinearFloatMapper) ID Uses

func (m LinearFloatMapper) ID(fi ...interface{}) (rowIDs []int64, err error)

ID maps floats to regularly spaced buckets

type Literal Uses

type Literal interface {
    // contains filtered or unexported methods
}

Literal interface is implemented by types which correspond to RDF Literals.

func FromBytes Uses

func FromBytes(bs []byte) Literal

FromBytes converts an encoded byte slice (from ToBytes) back to a Literal. DEV: May add an error and bounds checking.

func FromString Uses

func FromString(s string) Literal

FromString converts a Literal encoded with ToString back to a Literal.

type Logger Uses

type Logger interface {
    Printf(format string, v ...interface{})
    Debugf(format string, v ...interface{})
}

Logger is the interface that loggers must implement to get PDK logs.

type MapFieldTranslator Uses

type MapFieldTranslator struct {
    // contains filtered or unexported fields
}

MapFieldTranslator is an in-memory implementation of FieldTranslator using sync.Map and a slice.

func NewMapFieldTranslator Uses

func NewMapFieldTranslator() *MapFieldTranslator

NewMapFieldTranslator creates a new MapFieldTranslator.

func (*MapFieldTranslator) Get Uses

func (m *MapFieldTranslator) Get(id uint64) (interface{}, error)

Get returns the value mapped to the given id.

func (*MapFieldTranslator) GetID Uses

func (m *MapFieldTranslator) GetID(val interface{}) (id uint64, err error)

GetID returns the integer id associated with the given value. It allocates a new ID if the value is not found.

type MapTranslator Uses

type MapTranslator struct {
    // contains filtered or unexported fields
}

MapTranslator is an in-memory implementation of Translator using maps.

func NewMapTranslator Uses

func NewMapTranslator() *MapTranslator

NewMapTranslator creates a new MapTranslator.

func (*MapTranslator) Get Uses

func (m *MapTranslator) Get(field string, id uint64) (interface{}, error)

Get returns the value mapped to the given id in the given field.

func (*MapTranslator) GetID Uses

func (m *MapTranslator) GetID(field string, val interface{}) (id uint64, err error)

GetID returns the integer id associated with the given value in the given field. It allocates a new ID if the value is not found.

type Mapper Uses

type Mapper interface {
    ID(...interface{}) ([]int64, error)
}

Mapper represents a single method for mapping a specific data type to a slice of row IDs. A data type might be composed of multiple fields (e.g. a 2D point). A data type might use multiple mappers.

type MonthMapper Uses

type MonthMapper struct {
}

MonthMapper is a Mapper for timestamps, mapping the month only

func (MonthMapper) ID Uses

func (m MonthMapper) ID(ti ...interface{}) (rowIDs []int64, err error)

ID maps a timestamp to a month bucket (1-12)

type NamedReadCloser Uses

type NamedReadCloser interface {
    io.ReadCloser
    Name() string
    Meta() map[string]interface{} // not sure if we need this
}

NamedReadCloser adds the ability to associate a name and other arbitrary metdata with a ReadCloser. This is used by RawSource for e.g. a directory reading source to associate the file name of each reader it returns. This might be needed to generate the identifier for each record within the file, or to help build a recovery mechanism in the case of process crash (tracking which files have and have not been read yet).

type Nexter Uses

type Nexter struct {
    // contains filtered or unexported fields
}

Nexter is a threadsafe monotonic unique id generator

func NewNexter Uses

func NewNexter(opts ...NexterOption) *Nexter

NewNexter creates a new id generator starting at 0 - can be modified by options.

func (*Nexter) Last Uses

func (n *Nexter) Last() (lastID uint64)

Last returns the most recently generated id

func (*Nexter) Next Uses

func (n *Nexter) Next() (nextID uint64)

Next generates a new id and returns it

type NexterFrameTranslator Uses

type NexterFrameTranslator struct {
    // contains filtered or unexported fields
}

NexterFrameTranslator satisfies the FieldTranslator interface, but simply allocates a new contiguous id every time GetID(val) is called. It does not store any mapping and Get(id) always returns an error. Pilosa requires column ids regardless of whether we actually require tracking what each individual column represents, and the NexterFrameTranslator is useful in the case that we don't.

func NewNexterFieldTranslator Uses

func NewNexterFieldTranslator() *NexterFrameTranslator

NewNexterFieldTranslator creates a new NexterFrameTranslator

func (*NexterFrameTranslator) Get Uses

func (n *NexterFrameTranslator) Get(id uint64) (interface{}, error)

Get always returns nil, and a non-nil error for the NexterFrameTranslator.

func (*NexterFrameTranslator) GetID Uses

func (n *NexterFrameTranslator) GetID(val interface{}) (id uint64, err error)

GetID for the NexterFrameTranslator increments the internal id counter atomically and returns the next id - it ignores the val argument entirely.

type NexterOption Uses

type NexterOption func(n *Nexter)

NexterOption can be passed to NewNexter to modify the Nexter's behavior.

type NopLogger Uses

type NopLogger struct{}

NopLogger logs nothing.

func (NopLogger) Debugf Uses

func (NopLogger) Debugf(format string, v ...interface{})

Debugf does nothing.

func (NopLogger) Printf Uses

func (NopLogger) Printf(format string, v ...interface{})

Printf does nothing.

type NopStatter Uses

type NopStatter struct{}

NopStatter does nothing.

func (NopStatter) Count Uses

func (NopStatter) Count(name string, value int64, rate float64, tags ...string)

Count does nothing.

func (NopStatter) Gauge Uses

func (NopStatter) Gauge(name string, value float64, rate float64, tags ...string)

Gauge does nothing.

func (NopStatter) Histogram Uses

func (NopStatter) Histogram(name string, value float64, rate float64, tags ...string)

Histogram does nothing.

func (NopStatter) Set Uses

func (NopStatter) Set(name string, value string, rate float64, tags ...string)

Set does nothing.

func (NopStatter) Timing Uses

func (NopStatter) Timing(name string, value time.Duration, rate float64, tags ...string)

Timing does nothing.

type Object Uses

type Object interface {
    // contains filtered or unexported methods
}

Object is an interface satisfied by all things which may appear as objects in RDF triples. All literals are objects, but not all objects are literals.

type Objects Uses

type Objects []Object

type Parser Uses

type Parser interface {
    Parse(string) (interface{}, error)
}

Parser represents a single method for parsing a string field to a value

type Peeker Uses

type Peeker interface {
    Peek() (interface{}, error)
}

Peeker is an interface for peeking ahead at the next record to be returned by Source.Record().

type PeekingSource Uses

type PeekingSource struct {
    Source
    // contains filtered or unexported fields
}

PeekingSource is a wrapper for Source which implements the Peeker interface by reading the next record from Source and caching it for the next call to Record().

func NewPeekingSource Uses

func NewPeekingSource(source Source) *PeekingSource

NewPeekingSource returns a new peeking source.

func (*PeekingSource) Peek Uses

func (p *PeekingSource) Peek() (interface{}, error)

Peek returns a copy of the next record in the underlying source, without discarding it, so the following call to Record() will return the same data as if Peek had not been called.

func (*PeekingSource) Record Uses

func (p *PeekingSource) Record() (interface{}, error)

Record returns the next record in the underlying source, first checking if a cached record from Peek() has been set.

type PilosaKeyMapper Uses

type PilosaKeyMapper struct {
    // contains filtered or unexported fields
}

PilosaKeyMapper implements the KeyMapper interface.

func NewPilosaKeyMapper Uses

func NewPilosaKeyMapper(t Translator, colTranslator ...FieldTranslator) *PilosaKeyMapper

NewPilosaKeyMapper returns a PilosaKeyMapper.

func (*PilosaKeyMapper) MapRequest Uses

func (p *PilosaKeyMapper) MapRequest(body []byte) ([]byte, error)

MapRequest takes a request body and returns a mapped version of that body.

func (*PilosaKeyMapper) MapResult Uses

func (p *PilosaKeyMapper) MapResult(field string, res interface{}) (mappedRes interface{}, err error)

MapResult converts the result of a single top level query (one element of QueryResponse.Results) to its mapped counterpart.

type PilosaOption Uses

type PilosaOption func(opt *pilosaOptions) error

func OptPilosaClientOptions Uses

func OptPilosaClientOptions(options ...gopilosa.ClientOption) PilosaOption

func OptPilosaImportOptions Uses

func OptPilosaImportOptions(options ...gopilosa.ImportOption) PilosaOption

type PilosaRecord Uses

type PilosaRecord struct {
    Col  uint64OrString
    Rows []Row
    Vals []Val
}

PilosaRecord represents a number of set columns and values in a single Column in Pilosa.

func (*PilosaRecord) AddRow Uses

func (pr *PilosaRecord) AddRow(field string, idOrKey uint64OrString)

AddRow adds a new column to be set to the PilosaRecord.

func (*PilosaRecord) AddRowTime Uses

func (pr *PilosaRecord) AddRowTime(field string, id uint64, ts time.Time)

AddRowTime adds a new column to be set with a timestamp to the PilosaRecord.

func (*PilosaRecord) AddVal Uses

func (pr *PilosaRecord) AddVal(field string, value int64)

AddVal adds a new value to be range encoded into the given field to the PilosaRecord.

type Point Uses

type Point struct {
    X   float64
    Y   float64
}

Point is a point in a 2D space

type Properteer Uses

type Properteer interface {
    Property() Property
}

Properteer is the interface which should be implemented by types which want to explicitly define how they should be interpreted as a string for use as a Property when they are used as a map key.

type Property Uses

type Property string

Property represents a Predicate, and can be turned into a Predicate IRI by a context

type Proxy Uses

type Proxy interface {
    ProxyRequest(orig *http.Request, origbody []byte) (*http.Response, error)
}

Proxy describes the functionality for proxying requests.

type RawSource Uses

type RawSource interface {
    NextReader() (NamedReadCloser, error)
}

RawSource is the interface for getting actual raw data from a data source. The returned ReadCloser may have multiple individual records—how it's interpreted will depend on the adapter which converts the bytes read into Go objects. As an example, a RawSource for S3 might return each object in an S3 bucket as a reader. Each object might be a CSV file with many individual records which a separate CSV parser could then turn into a pdk.Source. Alternatively, an optimized implementation might read the CSV and convert it directly to PilosaRecord objects.

type RecordMapper Uses

type RecordMapper interface {
    Map(record *Entity) (PilosaRecord, error)
}

RecordMapper is the interface for taking parsed records from the Parser and figuring out what bits and values to set in Pilosa. RecordMappers usually have a Translator and a Nexter for converting arbitrary values to monotonic integer ids and generating column ids respectively. Implementations should be thread safe.

type RecordParser Uses

type RecordParser interface {
    Parse(data interface{}) (*Entity, error)
}

RecordParser is the interface for turning raw records from Source into Go objects. Implementations of Parser should be thread safe.

type Region Uses

type Region struct {
    Vertices []Point
}

Region is a simple polygonal region of R2 space

type RegionMapper Uses

type RegionMapper struct {
    Regions []Region
    // contains filtered or unexported fields
}

RegionMapper is a Mapper for a set of geometric regions (e.g. neighborhoods or states) TODO: generate regions by reading shapefile

type Row Uses

type Row struct {
    Field string
    ID    uint64OrString

    // Time is the timestamp for the column in Pilosa which is the intersection of
    // this row and the Column in the PilosaRecord which holds this row.
    Time time.Time
}

Row represents a column to set in Pilosa sans column id (which is held by the PilosaRecord containing the Row).

type S Uses

type S string

type Source Uses

type Source interface {
    Record() (interface{}, error)
}

Source is the interface for getting raw data one record at a time. Implementations of Source should be thread safe.

type SparseIntMapper Uses

type SparseIntMapper struct {
    Min int64
    Max int64
    Map map[int64]int64
    // contains filtered or unexported fields
}

SparseIntMapper is a Mapper for integer types, mapping only relevant ints

func (SparseIntMapper) ID Uses

func (m SparseIntMapper) ID(ii ...interface{}) (rowIDs []int64, err error)

ID maps arbitrary ints to a rowID range

type Statter Uses

type Statter interface {
    Count(name string, value int64, rate float64, tags ...string)
    Gauge(name string, value float64, rate float64, tags ...string)
    Histogram(name string, value float64, rate float64, tags ...string)
    Set(name string, value string, rate float64, tags ...string)
    Timing(name string, value time.Duration, rate float64, tags ...string)
}

Statter is the interface that stats collectors must implement to get stats out of the PDK.

type StdLogger Uses

type StdLogger struct {
    *log.Logger
}

StdLogger only prints on Printf.

func (StdLogger) Debugf Uses

func (StdLogger) Debugf(format string, v ...interface{})

Debugf implements Logger interface, but prints nothing.

func (StdLogger) Printf Uses

func (s StdLogger) Printf(format string, v ...interface{})

Printf implements Logger interface.

type StringContainsMapper Uses

type StringContainsMapper struct {
    Matches []string // slice of strings to check for containment
}

StringContainsMapper is a Mapper for string types...

type StringMatchesMapper Uses

type StringMatchesMapper struct {
    Matches []string // slice of strings to check for match
}

StringMatchesMapper is a Mapper for string types...

type StringParser Uses

type StringParser struct {
}

StringParser is a parser for string types

func (StringParser) Parse Uses

func (p StringParser) Parse(field string) (result interface{}, err error)

Parse is an identity parser for strings

type SubjectFunc Uses

type SubjectFunc func(d interface{}) (string, error)

SubjectFunc is a wrapper like http.HandlerFunc which allows you to use a bare func as a Subjecter.

func (SubjectFunc) Subject Uses

func (s SubjectFunc) Subject(d interface{}) (string, error)

Subject implements Subjecter.

type SubjectPath Uses

type SubjectPath []string

SubjectPath is an EntitySubjecter which extracts a subject by walking the Entity properties denoted by the strings in SubjectPath.

func (SubjectPath) Subject Uses

func (p SubjectPath) Subject(e *Entity) (string, error)

Subject implements EntitySubjecter.

type Subjecter Uses

type Subjecter interface {
    Subject(d interface{}) (string, error)
}

Subjecter is an interface for getting the Subject of a record.

type Time Uses

type Time time.Time

type TimeOfDayMapper Uses

type TimeOfDayMapper struct {
    Res int64
}

TimeOfDayMapper is a Mapper for timestamps, mapping the time component only TODO: consider putting all time buckets in same field pros: single field cons: would have to abandon the simple ID interface. also single field may not be a good thing

func (TimeOfDayMapper) ID Uses

func (m TimeOfDayMapper) ID(ti ...interface{}) (rowIDs []int64, err error)

ID maps a timestamp to a time of day bucket

type TimeParser Uses

type TimeParser struct {
    Layout string
}

TimeParser is a parser for timestamps

func (TimeParser) Parse Uses

func (p TimeParser) Parse(field string) (result interface{}, err error)

Parse parses a timestamp string to a time.Time value

type Transformer Uses

type Transformer interface {
    Transform(e *Entity) error
}

Transformer is an interface for something which performs an in-place transformation on an Entity. It might enrich the entity by adding new fields, delete existing fields that don't need to be indexed, or change fields.

type TransformerFunc Uses

type TransformerFunc func(*Entity) error

TransformerFunc can be wrapped around a function to make it implement the Transformer interface. Similar to http.HandlerFunc.

func (TransformerFunc) Transform Uses

func (t TransformerFunc) Transform(e *Entity) error

Transform implements Transformer for TransformerFunc

type Translator Uses

type Translator interface {
    Get(field string, id uint64) (interface{}, error)
    GetID(field string, val interface{}) (uint64, error)
}

Translator describes the functionality for mapping arbitrary values in a given Pilosa field to row ids and back. Implementations should be threadsafe and generate ids monotonically.

type U Uses

type U uint

func (U) MarshalJSON Uses

func (U U) MarshalJSON() ([]byte, error)

type U16 Uses

type U16 uint16

func (U16) MarshalJSON Uses

func (U U16) MarshalJSON() ([]byte, error)

type U32 Uses

type U32 uint32

func (U32) MarshalJSON Uses

func (U U32) MarshalJSON() ([]byte, error)

type U64 Uses

type U64 uint64

func (U64) MarshalJSON Uses

func (U U64) MarshalJSON() ([]byte, error)

type U8 Uses

type U8 uint8

func (U8) MarshalJSON Uses

func (U U8) MarshalJSON() ([]byte, error)

type Val Uses

type Val struct {
    Field string
    Value int64
}

Val represents a BSI value to set in a Pilosa field sans column id (which is held by the PilosaRecord containing the Val).

type VerboseLogger Uses

type VerboseLogger struct {
    *log.Logger
}

VerboseLogger prints on both Printf and Debugf.

func (VerboseLogger) Debugf Uses

func (s VerboseLogger) Debugf(format string, v ...interface{})

Debugf implements Logger interface.

func (VerboseLogger) Printf Uses

func (s VerboseLogger) Printf(format string, v ...interface{})

Printf implements Logger interface.

type YearMapper Uses

type YearMapper struct {
    MinYear int64 // TODO? use this to eliminate empty rows for year < 2000 or whatever
}

YearMapper is a Mapper for timestamps, mapping the year only

func (YearMapper) ID Uses

func (m YearMapper) ID(ti ...interface{}) (rowIDs []int64, err error)

ID maps a timestamp to a year bucket

Directories

PathSynopsis
aws/s3
boltdbPackage boltdb provides a pdk.Translator implementation using boltdb.
cmd
cmd/pdk
csv
csv2
fake
fake/gen
file
geohash
http
json
kafka
kafka/datagen
kafkagen
leveldb
mock
termstatPackage termstat provides a stats implementation which periodically logs the statistics to the given writer.
test
usecase/fakeusers
usecase/gen
usecase/ssb
usecase/taxi
usecase/userid
usecase/weather

Package pdk imports 22 packages (graph) and is imported by 14 packages. Updated 2019-08-07. Refresh now. Tools for package owners.