pilosa: github.com/pilosa/pilosa Index | Files | Directories

package pilosa

import "github.com/pilosa/pilosa"

Package pilosa implements the core of the Pilosa distributed bitmap index. It contains all the domain objects, interfaces, and logic that defines pilosa.

Index

Package Files

api.go apimethod_string.go attr.go broadcast.go cache.go client.go cluster.go cmd.go diagnostics.go doc.go event.go executor.go field.go fragment.go gc.go handler.go holder.go index.go iterator.go pilosa.go row.go server.go time.go translate.go uri.go version.go view.go

Constants

const (

    // ClusterState represents the state returned in the /status endpoint.
    ClusterStateStarting = "STARTING"
    ClusterStateDegraded = "DEGRADED" // cluster is running but we've lost some # of hosts >0 but < replicaN
    ClusterStateNormal   = "NORMAL"
    ClusterStateResizing = "RESIZING"
)
const (
    DefaultFieldType = FieldTypeSet

    DefaultCacheType = CacheTypeRanked

    // Default ranked field cache
    DefaultCacheSize = 50000
)

Default field settings.

const (
    FieldTypeSet   = "set"
    FieldTypeInt   = "int"
    FieldTypeTime  = "time"
    FieldTypeMutex = "mutex"
    FieldTypeBool  = "bool"
)

Field types.

const (
    CacheTypeLRU    = "lru"
    CacheTypeRanked = "ranked"
    CacheTypeNone   = "none"
)

Cache types.

const (
    // ShardWidth is the number of column IDs in a shard. It must be a power of 2 greater than or equal to 16.
    // shardWidthExponent = 20 // set in shardwidthNN.go files
    ShardWidth = 1 << shardwidth.Exponent

    // HashBlockSize is the number of rows in a merkle hash block.
    HashBlockSize = 100
)
const TimeFormat = "2006-01-02T15:04"

TimeFormat is the go-style time format used to parse string dates.

Variables

var (
    ErrHostRequired = errors.New("host required")

    ErrIndexRequired = errors.New("index required")
    ErrIndexExists   = errors.New("index already exists")
    ErrIndexNotFound = errors.New("index not found")

    // ErrFieldRequired is returned when no field is specified.
    ErrFieldRequired = errors.New("field required")
    ErrFieldExists   = errors.New("field already exists")
    ErrFieldNotFound = errors.New("field not found")

    ErrBSIGroupNotFound         = errors.New("bsigroup not found")
    ErrBSIGroupExists           = errors.New("bsigroup already exists")
    ErrBSIGroupNameRequired     = errors.New("bsigroup name required")
    ErrInvalidBSIGroupType      = errors.New("invalid bsigroup type")
    ErrInvalidBSIGroupRange     = errors.New("invalid bsigroup range")
    ErrInvalidBSIGroupValueType = errors.New("invalid bsigroup value type")
    ErrBSIGroupValueTooLow      = errors.New("bsigroup value too low")
    ErrBSIGroupValueTooHigh     = errors.New("bsigroup value too high")
    ErrInvalidRangeOperation    = errors.New("invalid range operation")
    ErrInvalidBetweenValue      = errors.New("invalid value for between operation")

    ErrInvalidView      = errors.New("invalid view")
    ErrInvalidCacheType = errors.New("invalid cache type")

    ErrName  = errors.New("invalid index or field name, must match [a-z][a-z0-9_-]* and contain at most 64 characters")
    ErrLabel = errors.New("invalid row or column label, must match [A-Za-z0-9_-]")

    // ErrFragmentNotFound is returned when a fragment does not exist.
    ErrFragmentNotFound = errors.New("fragment not found")
    ErrQueryRequired    = errors.New("query required")
    ErrQueryCancelled   = errors.New("query cancelled")
    ErrQueryTimeout     = errors.New("query timeout")
    ErrTooManyWrites    = errors.New("too many write commands")

    // TODO(2.0) poorly named - used when a *node* doesn't own a shard. Probably
    // we won't need this error at all by 2.0 though.
    ErrClusterDoesNotOwnShard = errors.New("node does not own shard")

    ErrNodeIDNotExists    = errors.New("node with provided ID does not exist")
    ErrNodeNotCoordinator = errors.New("node is not the coordinator")
    ErrResizeNotRunning   = errors.New("no resize job currently running")

    ErrNotImplemented            = errors.New("not implemented")
    ErrFieldsArgumentRequired    = errors.New("fields argument required")
    ErrExpectedFieldListArgument = errors.New("expected field list argument")
)

System errors.

var (
    ErrTranslateStoreClosed       = errors.New("translate store closed")
    ErrTranslateStoreReaderClosed = errors.New("translate store reader closed")
    ErrReplicationNotSupported    = errors.New("replication not supported")
    ErrTranslateStoreReadOnly     = errors.New("translate store could not find or create key, translate store read only")
    ErrCannotOpenV1TranslateFile  = errors.New("cannot open v1 translate .keys file")
)

Translate store errors.

var BuildTime = "not recorded"
var Enterprise = "0"
var EnterpriseEnabled = false
var ErrInvalidTimeQuantum = errors.New("invalid time quantum")

ErrInvalidTimeQuantum is returned when parsing a time quantum.

var NopBroadcaster broadcaster = &nopBroadcaster{}

NopBroadcaster represents a Broadcaster that doesn't do anything.

var Version = "v0.0.0"

func DecodeAttrs Uses

func DecodeAttrs(v []byte) (map[string]interface{}, error)

DecodeAttrs decodes a byte slice into an attribute map.

func EncodeAttrs Uses

func EncodeAttrs(attr map[string]interface{}) ([]byte, error)

EncodeAttrs encodes an attribute map into a byte slice.

func MarshalInternalMessage Uses

func MarshalInternalMessage(m Message, s Serializer) ([]byte, error)

MarshalInternalMessage serializes the pilosa message and adds pilosa internal type info which is used by the internal messaging stuff.

func NewRankCache Uses

func NewRankCache(maxEntries uint32) *rankCache

NewRankCache returns a new instance of RankCache.

func OptAPIImportWorkerPoolSize Uses

func OptAPIImportWorkerPoolSize(size int) apiOption

func OptAPIServer Uses

func OptAPIServer(s *Server) apiOption

type API Uses

type API struct {
    Serializer Serializer
    // contains filtered or unexported fields
}

API provides the top level programmatic interface to Pilosa. It is usually wrapped by a handler which provides an external interface (e.g. HTTP).

func NewAPI Uses

func NewAPI(opts ...apiOption) (*API, error)

NewAPI returns a new API instance.

func (*API) ApplySchema Uses

func (api *API) ApplySchema(ctx context.Context, s *Schema, remote bool) error

ApplySchema takes the given schema and applies it across the cluster (if remote is false), or just to this node (if remote is true). This is designed for the use case of replicating a schema from one Pilosa cluster to another which is initially empty. It is not officially supported in other scenarios and may produce surprising results.

func (*API) AvailableShardsByIndex Uses

func (api *API) AvailableShardsByIndex(ctx context.Context) map[string]*roaring.Bitmap

AvailableShardsByIndex returns bitmaps of shards with available by index name.

func (*API) Close Uses

func (api *API) Close() error

Close closes the api and waits for it to shutdown.

func (*API) ClusterMessage Uses

func (api *API) ClusterMessage(ctx context.Context, reqBody io.Reader) error

ClusterMessage is for internal use. It decodes a protobuf message out of the body and forwards it to the BroadcastHandler.

func (*API) CreateField Uses

func (api *API) CreateField(ctx context.Context, indexName string, fieldName string, opts ...FieldOption) (*Field, error)

CreateField makes the named field in the named index with the given options. This method currently only takes a single functional option, but that may be changed in the future to support multiple options.

func (*API) CreateIndex Uses

func (api *API) CreateIndex(ctx context.Context, indexName string, options IndexOptions) (*Index, error)

CreateIndex makes a new Pilosa index.

func (*API) DeleteAvailableShard Uses

func (api *API) DeleteAvailableShard(_ context.Context, indexName, fieldName string, shardID uint64) error

DeleteAvailableShard a shard ID from the available shard set cache.

func (*API) DeleteField Uses

func (api *API) DeleteField(ctx context.Context, indexName string, fieldName string) error

DeleteField removes the named field from the named index. If the index is not found, an error is returned. If the field is not found, it is ignored and no action is taken.

func (*API) DeleteIndex Uses

func (api *API) DeleteIndex(ctx context.Context, indexName string) error

DeleteIndex removes the named index. If the index is not found it does nothing and returns no error.

func (*API) DeleteView Uses

func (api *API) DeleteView(ctx context.Context, indexName string, fieldName string, viewName string) error

DeleteView removes the given view.

func (*API) ExportCSV Uses

func (api *API) ExportCSV(ctx context.Context, indexName string, fieldName string, shard uint64, w io.Writer) error

ExportCSV encodes the fragment designated by the index,field,shard as CSV of the form <row>,<col>

func (*API) Field Uses

func (api *API) Field(ctx context.Context, indexName, fieldName string) (*Field, error)

Field retrieves the named field.

func (*API) FieldAttrDiff Uses

func (api *API) FieldAttrDiff(ctx context.Context, indexName string, fieldName string, blocks []AttrBlock) (map[uint64]map[string]interface{}, error)

FieldAttrDiff determines the local row attribute data blocks which differ from those provided.

func (*API) FragmentBlockData Uses

func (api *API) FragmentBlockData(ctx context.Context, body io.Reader) ([]byte, error)

FragmentBlockData is an endpoint for internal usage. It is not guaranteed to return anything useful. Currently it returns protobuf encoded row and column ids from a "block" which is a subdivision of a fragment.

func (*API) FragmentBlocks Uses

func (api *API) FragmentBlocks(ctx context.Context, indexName, fieldName, viewName string, shard uint64) ([]FragmentBlock, error)

FragmentBlocks returns the checksums and block ids for all blocks in the specified fragment.

func (*API) FragmentData Uses

func (api *API) FragmentData(ctx context.Context, indexName, fieldName, viewName string, shard uint64) (io.WriterTo, error)

FragmentData returns all data in the specified fragment.

func (*API) GetTranslateEntryReader Uses

func (api *API) GetTranslateEntryReader(ctx context.Context, offsets TranslateOffsetMap) (TranslateEntryReader, error)

GetTranslateEntryReader provides an entry reader for key translation logs starting at offset.

func (*API) Hosts Uses

func (api *API) Hosts(ctx context.Context) []*Node

Hosts returns a list of the hosts in the cluster including their ID, URL, and which is the coordinator.

func (*API) Import Uses

func (api *API) Import(ctx context.Context, req *ImportRequest, opts ...ImportOption) error

Import bulk imports data into a particular index,field,shard.

func (*API) ImportRoaring Uses

func (api *API) ImportRoaring(ctx context.Context, indexName, fieldName string, shard uint64, remote bool, req *ImportRoaringRequest) (err error)

ImportRoaring is a low level interface for importing data to Pilosa when extremely high throughput is desired. The data must be encoded in a particular way which may be unintuitive (discussed below). The data is merged with existing data.

It takes as input a roaring bitmap which it uses as the data for the indicated index, field, and shard. The bitmap may be encoded according to the official roaring spec (https://github.com/RoaringBitmap/RoaringFormatSpec), or to the pilosa roaring spec which supports 64 bit integers (https://www.pilosa.com/docs/latest/architecture/#roaring-bitmap-storage-format).

The data should be encoded the same way that Pilosa stores fragments internally. A bit "i" being set in the input bitmap indicates that the bit is set in Pilosa row "i/ShardWidth", and in column (shard*ShardWidth)+(i%ShardWidth). That is to say that "data" represents all of the rows in this shard of this field concatenated together in one long bitmap.

func (*API) ImportValue Uses

func (api *API) ImportValue(ctx context.Context, req *ImportValueRequest, opts ...ImportOption) error

ImportValue bulk imports values into a particular field.

func (*API) Index Uses

func (api *API) Index(ctx context.Context, indexName string) (*Index, error)

Index retrieves the named index.

func (*API) IndexAttrDiff Uses

func (api *API) IndexAttrDiff(ctx context.Context, indexName string, blocks []AttrBlock) (map[uint64]map[string]interface{}, error)

IndexAttrDiff determines the local column attribute data blocks which differ from those provided.

func (*API) Info Uses

func (api *API) Info() serverInfo

Info returns information about this server instance.

func (*API) LongQueryTime Uses

func (api *API) LongQueryTime() time.Duration

LongQueryTime returns the configured threshold for logging/statting long running queries.

func (*API) MaxShards Uses

func (api *API) MaxShards(ctx context.Context) map[string]uint64

MaxShards returns the maximum shard number for each index in a map. TODO (2.0): This method has been deprecated. Instead, use AvailableShardsByIndex.

func (*API) Node Uses

func (api *API) Node() *Node

Node gets the ID, URI and coordinator status for this particular node.

func (*API) PrimaryReplicaNodeURL Uses

func (api *API) PrimaryReplicaNodeURL() url.URL

PrimaryReplicaNodeURL returns the URL of the cluster's primary replica.

func (*API) Query Uses

func (api *API) Query(ctx context.Context, req *QueryRequest) (QueryResponse, error)

Query parses a PQL query out of the request and executes it.

func (*API) RecalculateCaches Uses

func (api *API) RecalculateCaches(ctx context.Context) error

RecalculateCaches forces all TopN caches to be updated. Used mainly for integration tests.

func (*API) RemoveNode Uses

func (api *API) RemoveNode(id string) (*Node, error)

RemoveNode puts the cluster into the "RESIZING" state and begins the job of removing the given node.

func (*API) ResizeAbort Uses

func (api *API) ResizeAbort() error

ResizeAbort stops the current resize job.

func (*API) Schema Uses

func (api *API) Schema(ctx context.Context) []*IndexInfo

Schema returns information about each index in Pilosa including which fields they contain.

func (*API) SetCoordinator Uses

func (api *API) SetCoordinator(ctx context.Context, id string) (oldNode, newNode *Node, err error)

SetCoordinator makes a new Node the cluster coordinator.

func (*API) ShardNodes Uses

func (api *API) ShardNodes(ctx context.Context, indexName string, shard uint64) ([]*Node, error)

ShardNodes returns the node and all replicas which should contain a shard's data.

func (*API) State Uses

func (api *API) State() string

State returns the cluster state which is usually "NORMAL", but could be "STARTING", "RESIZING", or potentially others. See cluster.go for more details.

func (*API) StatsWithTags Uses

func (api *API) StatsWithTags(tags []string) stats.StatsClient

StatsWithTags returns an instance of whatever implementation of StatsClient pilosa is using with the given tags.

func (*API) TranslateKeys Uses

func (api *API) TranslateKeys(r io.Reader) ([]byte, error)

TranslateKeys handles a TranslateKeyRequest.

func (*API) Version Uses

func (api *API) Version() string

Version returns the Pilosa version.

func (*API) Views Uses

func (api *API) Views(ctx context.Context, indexName string, fieldName string) ([]*view, error)

Views returns the views in the given field.

type AttrBlock Uses

type AttrBlock struct {
    ID       uint64 `json:"id"`
    Checksum []byte `json:"checksum"`
}

AttrBlock represents a checksummed block of the attribute store.

type AttrStore Uses

type AttrStore interface {
    Path() string
    Open() error
    Close() error
    Attrs(id uint64) (m map[string]interface{}, err error)
    SetAttrs(id uint64, m map[string]interface{}) error
    SetBulkAttrs(m map[uint64]map[string]interface{}) error
    Blocks() ([]AttrBlock, error)
    BlockData(i uint64) (map[uint64]map[string]interface{}, error)
}

AttrStore represents an interface for handling row/column attributes.

type BadRequestError Uses

type BadRequestError struct {
    // contains filtered or unexported fields
}

BadRequestError wraps an error value to signify that a request could not be read, decoded, or parsed such that in an HTTP scenario, http.StatusBadRequest would be returned.

func NewBadRequestError Uses

func NewBadRequestError(err error) BadRequestError

NewBadRequestError returns err wrapped in a BadRequestError.

type Bit Uses

type Bit struct {
    RowID     uint64
    ColumnID  uint64
    RowKey    string
    ColumnKey string
    Timestamp int64
}

Bit represents the intersection of a row and a column. It can be specified by integer ids or string keys.

type BlockDataRequest Uses

type BlockDataRequest struct {
    Index string
    Field string
    View  string
    Shard uint64
    Block uint64
}

BlockDataRequest describes the structure of a request for fragment block data.

type BlockDataResponse Uses

type BlockDataResponse struct {
    RowIDs    []uint64
    ColumnIDs []uint64
}

BlockDataResponse is the structured response of a block data request.

type ClusterStatus Uses

type ClusterStatus struct {
    ClusterID string
    State     string
    Nodes     []*Node
}

ClusterStatus describes the status of the cluster including its state and node topology.

type CmdIO Uses

type CmdIO struct {
    Stdin  io.Reader
    Stdout io.Writer
    Stderr io.Writer
    // contains filtered or unexported fields
}

CmdIO holds standard unix inputs and outputs.

func NewCmdIO Uses

func NewCmdIO(stdin io.Reader, stdout, stderr io.Writer) *CmdIO

NewCmdIO returns a new instance of CmdIO with inputs and outputs set to the arguments.

func (*CmdIO) Logger Uses

func (c *CmdIO) Logger() *log.Logger

type ColumnAttrSet Uses

type ColumnAttrSet struct {
    ID    uint64                 `json:"id"`
    Key   string                 `json:"key,omitempty"`
    Attrs map[string]interface{} `json:"attrs,omitempty"`
}

ColumnAttrSet represents a set of attributes for a vertical column in an index. Can have a set of attributes attached to it.

func (ColumnAttrSet) MarshalJSON Uses

func (cas ColumnAttrSet) MarshalJSON() ([]byte, error)

MarshalJSON marshals the ColumnAttrSet to JSON such that either a Key or an ID is included.

type ConflictError Uses

type ConflictError struct {
    // contains filtered or unexported fields
}

ConflictError wraps an error value to signify that a conflict with an existing resource occurred such that in an HTTP scenario, http.StatusConflict would be returned.

type CreateFieldMessage Uses

type CreateFieldMessage struct {
    Index string
    Field string
    Meta  *FieldOptions
}

CreateFieldMessage is an internal message indicating field creation.

type CreateIndexMessage Uses

type CreateIndexMessage struct {
    Index string
    Meta  *IndexOptions
}

CreateIndexMessage is an internal message indicating index creation.

type CreateShardMessage Uses

type CreateShardMessage struct {
    Index string
    Field string
    Shard uint64
}

CreateShardMessage is an internal message indicating shard creation.

type CreateViewMessage Uses

type CreateViewMessage struct {
    Index string
    Field string
    View  string
}

CreateViewMessage is an internal message indicating view creation.

type DeleteAvailableShardMessage Uses

type DeleteAvailableShardMessage struct {
    Index   string
    Field   string
    ShardID uint64
}

DeleteAvailableShardMessage is an internal message indicating available shard deletion.

type DeleteFieldMessage Uses

type DeleteFieldMessage struct {
    Index string
    Field string
}

DeleteFieldMessage is an internal message indicating field deletion.

type DeleteIndexMessage Uses

type DeleteIndexMessage struct {
    Index string
}

DeleteIndexMessage is an internal message indicating index deletion.

type DeleteViewMessage Uses

type DeleteViewMessage struct {
    Index string
    Field string
    View  string
}

DeleteViewMessage is an internal message indicating view deletion.

type Field Uses

type Field struct {
    Stats stats.StatsClient

    // Instantiates new translation store on open.
    OpenTranslateStore OpenTranslateStoreFunc
    // contains filtered or unexported fields
}

Field represents a container for views.

func NewField Uses

func NewField(path, index, name string, opts FieldOption) (*Field, error)

NewField returns a new instance of field.

func (*Field) AddRemoteAvailableShards Uses

func (f *Field) AddRemoteAvailableShards(b *roaring.Bitmap) error

AddRemoteAvailableShards merges the set of available shards into the current known set and saves the set to a file.

func (*Field) AvailableShards Uses

func (f *Field) AvailableShards() *roaring.Bitmap

AvailableShards returns a bitmap of shards that contain data.

func (*Field) CacheSize Uses

func (f *Field) CacheSize() uint32

CacheSize returns the ranked field cache size.

func (*Field) ClearBit Uses

func (f *Field) ClearBit(rowID, colID uint64) (changed bool, err error)

ClearBit clears a bit within the field.

func (*Field) Close Uses

func (f *Field) Close() error

Close closes the field and its views.

func (*Field) Import Uses

func (f *Field) Import(rowIDs, columnIDs []uint64, timestamps []*time.Time, opts ...ImportOption) error

Import bulk imports data.

func (*Field) Index Uses

func (f *Field) Index() string

Index returns the index name the field was initialized with.

func (*Field) Max Uses

func (f *Field) Max(filter *Row, name string) (max, count int64, err error)

Max returns the max for a field. An optional filtering row can be provided.

func (*Field) Min Uses

func (f *Field) Min(filter *Row, name string) (min, count int64, err error)

Min returns the min for a field. An optional filtering row can be provided.

func (*Field) Name Uses

func (f *Field) Name() string

Name returns the name the field was initialized with.

func (*Field) Open Uses

func (f *Field) Open() error

Open opens and initializes the field.

func (*Field) Options Uses

func (f *Field) Options() FieldOptions

Options returns all options for this field.

func (*Field) Path Uses

func (f *Field) Path() string

Path returns the path the field was initialized with.

func (*Field) Range Uses

func (f *Field) Range(name string, op pql.Token, predicate int64) (*Row, error)

Range performs a conditional operation on Field.

func (*Field) RemoveAvailableShard Uses

func (f *Field) RemoveAvailableShard(v uint64) error

RemoveAvailableShard removes a shard from the bitmap cache.

NOTE: This can be overridden on the next sync so all nodes should be updated.

func (*Field) Row Uses

func (f *Field) Row(rowID uint64) (*Row, error)

Row returns a row of the standard view. It seems this method is only being used by the test package, and the fact that it's only allowed on `set` fields is odd. This may be considered for deprecation in a future version.

func (*Field) RowAttrStore Uses

func (f *Field) RowAttrStore() AttrStore

RowAttrStore returns the attribute storage.

func (*Field) RowTime Uses

func (f *Field) RowTime(rowID uint64, time time.Time, quantum string) (*Row, error)

RowTime gets the row at the particular time with the granularity specified by the quantum.

func (*Field) SetBit Uses

func (f *Field) SetBit(rowID, colID uint64, t *time.Time) (changed bool, err error)

SetBit sets a bit on a view within the field.

func (*Field) SetCacheSize Uses

func (f *Field) SetCacheSize(v uint32) error

SetCacheSize sets the cache size for ranked fames. Persists to meta file on update. defaults to DefaultCacheSize 50000

func (*Field) SetValue Uses

func (f *Field) SetValue(columnID uint64, value int64) (changed bool, err error)

SetValue sets a field value for a column.

func (*Field) Sum Uses

func (f *Field) Sum(filter *Row, name string) (sum, count int64, err error)

Sum returns the sum and count for a field. An optional filtering row can be provided.

func (*Field) TimeQuantum Uses

func (f *Field) TimeQuantum() TimeQuantum

TimeQuantum returns the time quantum for the field.

func (*Field) TranslateStore Uses

func (f *Field) TranslateStore() TranslateStore

TranslateStore returns the underlying translation store for the field.

func (*Field) Type Uses

func (f *Field) Type() string

Type returns the field type.

func (*Field) Value Uses

func (f *Field) Value(columnID uint64) (value int64, exists bool, err error)

Value reads a field value for a column.

type FieldInfo Uses

type FieldInfo struct {
    Name    string       `json:"name"`
    Options FieldOptions `json:"options"`
    Views   []*ViewInfo  `json:"views,omitempty"`
}

FieldInfo represents schema information for a field.

type FieldOption Uses

type FieldOption func(fo *FieldOptions) error

FieldOption is a functional option type for pilosa.fieldOptions.

func OptFieldKeys Uses

func OptFieldKeys() FieldOption

OptFieldKeys is a functional option on FieldOptions used to specify whether keys are used for this field.

func OptFieldTypeBool Uses

func OptFieldTypeBool() FieldOption

OptFieldTypeBool is a functional option on FieldOptions used to specify the field as being type `bool` and to provide any respective configuration values.

func OptFieldTypeDefault Uses

func OptFieldTypeDefault() FieldOption

OptFieldTypeDefault is a functional option on FieldOptions used to set the field type and cache setting to the default values.

func OptFieldTypeInt Uses

func OptFieldTypeInt(min, max int64) FieldOption

OptFieldTypeInt is a functional option on FieldOptions used to specify the field as being type `int` and to provide any respective configuration values.

func OptFieldTypeMutex Uses

func OptFieldTypeMutex(cacheType string, cacheSize uint32) FieldOption

OptFieldTypeMutex is a functional option on FieldOptions used to specify the field as being type `mutex` and to provide any respective configuration values.

func OptFieldTypeSet Uses

func OptFieldTypeSet(cacheType string, cacheSize uint32) FieldOption

OptFieldTypeSet is a functional option on FieldOptions used to specify the field as being type `set` and to provide any respective configuration values.

func OptFieldTypeTime Uses

func OptFieldTypeTime(timeQuantum TimeQuantum, opt ...bool) FieldOption

OptFieldTypeTime is a functional option on FieldOptions used to specify the field as being type `time` and to provide any respective configuration values. Pass true to skip creation of the standard view.

type FieldOptions Uses

type FieldOptions struct {
    Base           int64       `json:"base,omitempty"`
    BitDepth       uint        `json:"bitDepth,omitempty"`
    Min            int64       `json:"min,omitempty"`
    Max            int64       `json:"max,omitempty"`
    Keys           bool        `json:"keys"`
    NoStandardView bool        `json:"noStandardView,omitempty"`
    CacheSize      uint32      `json:"cacheSize,omitempty"`
    CacheType      string      `json:"cacheType,omitempty"`
    Type           string      `json:"type,omitempty"`
    TimeQuantum    TimeQuantum `json:"timeQuantum,omitempty"`
}

FieldOptions represents options to set when initializing a field.

func (*FieldOptions) MarshalJSON Uses

func (o *FieldOptions) MarshalJSON() ([]byte, error)

MarshalJSON marshals FieldOptions to JSON such that only those attributes associated to the field type are included.

type FieldRow Uses

type FieldRow struct {
    Field  string `json:"field"`
    RowID  uint64 `json:"rowID"`
    RowKey string `json:"rowKey,omitempty"`
}

FieldRow is used to distinguish rows in a group by result.

func (FieldRow) MarshalJSON Uses

func (fr FieldRow) MarshalJSON() ([]byte, error)

MarshalJSON marshals FieldRow to JSON such that either a Key or an ID is included.

func (FieldRow) String Uses

func (fr FieldRow) String() string

String is the FieldRow stringer.

type FieldStatus Uses

type FieldStatus struct {
    Name            string
    AvailableShards *roaring.Bitmap
}

FieldStatus is an internal message representing the contents of a field.

type FieldValue Uses

type FieldValue struct {
    ColumnID  uint64
    ColumnKey string
    Value     int64
}

FieldValue represents the value for a column within a range-encoded field.

type FragmentBlock Uses

type FragmentBlock struct {
    ID       int    `json:"id"`
    Checksum []byte `json:"checksum"`
}

FragmentBlock represents info about a subsection of the rows in a block. This is used for comparing data in remote blocks for active anti-entropy.

type GCNotifier Uses

type GCNotifier interface {
    Close()
    AfterGC() <-chan struct{}
}

GCNotifier represents an interface for garbage collection notificationss.

var NopGCNotifier GCNotifier = &nopGCNotifier{}

NopGCNotifier represents a GCNotifier that doesn't do anything.

type GroupCount Uses

type GroupCount struct {
    Group []FieldRow `json:"group"`
    Count uint64     `json:"count"`
}

GroupCount represents a result item for a group by query.

func (GroupCount) Compare Uses

func (g GroupCount) Compare(o GroupCount) int

Compare is used in ordering two GroupCount objects.

type Handler Uses

type Handler interface {
    Serve() error
    Close() error
}

Handler is the interface for the data handler, a wrapper around Pilosa's data store.

var NopHandler Handler = nopHandler{}

NopHandler is a no-op implementation of the Handler interface.

type Hasher Uses

type Hasher interface {
    // Hashes the key into a number between [0,N).
    Hash(key uint64, n int) int
}

Hasher represents an interface to hash integers into buckets.

type Holder Uses

type Holder struct {
    NewAttrStore func(string) AttrStore

    // Stats
    Stats stats.StatsClient

    // Data directory path.
    Path string

    Logger logger.Logger

    // Instantiates new translation stores for indexes & fields.
    OpenTranslateStore  OpenTranslateStoreFunc  // local store
    OpenTranslateReader OpenTranslateReaderFunc // replication
    // contains filtered or unexported fields
}

Holder represents a container for indexes.

func NewHolder Uses

func NewHolder() *Holder

NewHolder returns a new instance of Holder.

func (*Holder) Close Uses

func (h *Holder) Close() error

Close closes all open fragments.

func (*Holder) CreateIndex Uses

func (h *Holder) CreateIndex(name string, opt IndexOptions) (*Index, error)

CreateIndex creates an index. An error is returned if the index already exists.

func (*Holder) CreateIndexIfNotExists Uses

func (h *Holder) CreateIndexIfNotExists(name string, opt IndexOptions) (*Index, error)

CreateIndexIfNotExists returns an index by name. The index is created if it does not already exist.

func (*Holder) DeleteIndex Uses

func (h *Holder) DeleteIndex(name string) error

DeleteIndex removes an index from the holder.

func (*Holder) Field Uses

func (h *Holder) Field(index, name string) *Field

Field returns the field for an index and name.

func (*Holder) HasData Uses

func (h *Holder) HasData() (bool, error)

HasData returns true if Holder contains at least one index. This is used to determine if the rebalancing of data is necessary when a node joins the cluster.

func (*Holder) Index Uses

func (h *Holder) Index(name string) *Index

Index returns the index by name.

func (*Holder) IndexPath Uses

func (h *Holder) IndexPath(name string) string

IndexPath returns the path where a given index is stored.

func (*Holder) Indexes Uses

func (h *Holder) Indexes() []*Index

Indexes returns a list of all indexes in the holder.

func (*Holder) Open Uses

func (h *Holder) Open() error

Open initializes the root data directory for the holder.

func (*Holder) Schema Uses

func (h *Holder) Schema() []*IndexInfo

Schema returns schema information for all indexes, fields, and views.

func (*Holder) TranslateEntryReader Uses

func (h *Holder) TranslateEntryReader(ctx context.Context, offsets TranslateOffsetMap) (_ TranslateEntryReader, err error)

TranslateEntryReader returns a reader that merges all index & field reader that are specified in the offsets map.

func (*Holder) TranslateOffsetMap Uses

func (h *Holder) TranslateOffsetMap() (TranslateOffsetMap, error)

TranslateOffsetMap returns a map of offsets for all indexes & fields.

func (*Holder) TranslateStore Uses

func (h *Holder) TranslateStore(index, field string) (TranslateStore, error)

TranslateStore returns store for the given index or field.

type ImportOption Uses

type ImportOption func(*ImportOptions) error

ImportOption is a functional option type for API.Import.

func OptImportOptionsClear Uses

func OptImportOptionsClear(c bool) ImportOption

OptImportOptionsClear is a functional option on ImportOption used to specify whether the import is a set or clear operation.

func OptImportOptionsIgnoreKeyCheck Uses

func OptImportOptionsIgnoreKeyCheck(b bool) ImportOption

OptImportOptionsIgnoreKeyCheck is a functional option on ImportOption used to specify whether key check should be ignored.

type ImportOptions Uses

type ImportOptions struct {
    Clear          bool
    IgnoreKeyCheck bool
}

ImportOptions holds the options for the API.Import method.

type ImportRequest Uses

type ImportRequest struct {
    Index      string
    Field      string
    Shard      uint64
    RowIDs     []uint64
    ColumnIDs  []uint64
    RowKeys    []string
    ColumnKeys []string
    Timestamps []int64
}

ImportRequest describes the import request structure for an import.

type ImportResponse Uses

type ImportResponse struct {
    Err string
}

ImportResponse is the structured response of an import.

type ImportRoaringRequest Uses

type ImportRoaringRequest struct {
    Clear bool
    Views map[string][]byte
}

ImportRoaringRequest describes the import request structure for an import containing roaring-encoded data.

type ImportValueRequest Uses

type ImportValueRequest struct {
    Index      string
    Field      string
    Shard      uint64
    ColumnIDs  []uint64
    ColumnKeys []string
    Values     []int64
}

ImportValueRequest describes the import request structure for a value (BSI) import.

type InMemTranslateStore Uses

type InMemTranslateStore struct {
    // contains filtered or unexported fields
}

InMemTranslateStore is an in-memory storage engine for mapping keys to int values.

func NewInMemTranslateStore Uses

func NewInMemTranslateStore(index, field string) *InMemTranslateStore

NewInMemTranslateStore returns a new instance of InMemTranslateStore.

func (*InMemTranslateStore) Close Uses

func (s *InMemTranslateStore) Close() error

func (*InMemTranslateStore) EntryReader Uses

func (s *InMemTranslateStore) EntryReader(ctx context.Context, offset uint64) (TranslateEntryReader, error)

EntryReader returns an error. Replication is not supported.

func (*InMemTranslateStore) ForceSet Uses

func (s *InMemTranslateStore) ForceSet(id uint64, key string) error

ForceSet writes the id/key pair to the db. Used by replication.

func (*InMemTranslateStore) MaxID Uses

func (s *InMemTranslateStore) MaxID() (uint64, error)

MaxID returns the highest identifier in the store.

func (*InMemTranslateStore) ReadOnly Uses

func (s *InMemTranslateStore) ReadOnly() bool

ReadOnly returns true if the store is in read-only mode.

func (*InMemTranslateStore) SetReadOnly Uses

func (s *InMemTranslateStore) SetReadOnly(v bool)

SetReadOnly toggles the read-only mode of the store.

func (*InMemTranslateStore) TranslateID Uses

func (s *InMemTranslateStore) TranslateID(id uint64) (string, error)

TranslateID converts an integer ID to a string key. Returns a blank string if ID does not exist.

func (*InMemTranslateStore) TranslateIDs Uses

func (s *InMemTranslateStore) TranslateIDs(ids []uint64) ([]string, error)

TranslateIDs converts a list of integer IDs to a list of string keys.

func (*InMemTranslateStore) TranslateKey Uses

func (s *InMemTranslateStore) TranslateKey(key string) (uint64, error)

TranslateKeys converts a string key to an integer ID. If key does not have an associated id then one is created.

func (*InMemTranslateStore) TranslateKeys Uses

func (s *InMemTranslateStore) TranslateKeys(keys []string) ([]uint64, error)

TranslateKeys converts a string key to an integer ID. If key does not have an associated id then one is created.

func (*InMemTranslateStore) WriteNotify Uses

func (s *InMemTranslateStore) WriteNotify() <-chan struct{}

WriteNotify returns a channel that is closed when a new entry is written.

type Index Uses

type Index struct {
    Stats stats.StatsClient

    // Instantiates new translation stores for fields.
    OpenTranslateStore OpenTranslateStoreFunc
    // contains filtered or unexported fields
}

Index represents a container for fields.

func NewIndex Uses

func NewIndex(path, name string) (*Index, error)

NewIndex returns a new instance of Index.

func (*Index) AvailableShards Uses

func (i *Index) AvailableShards() *roaring.Bitmap

AvailableShards returns a bitmap of all shards with data in the index.

func (*Index) Close Uses

func (i *Index) Close() error

Close closes the index and its fields.

func (*Index) ColumnAttrStore Uses

func (i *Index) ColumnAttrStore() AttrStore

ColumnAttrStore returns the storage for column attributes.

func (*Index) CreateField Uses

func (i *Index) CreateField(name string, opts ...FieldOption) (*Field, error)

CreateField creates a field.

func (*Index) CreateFieldIfNotExists Uses

func (i *Index) CreateFieldIfNotExists(name string, opts ...FieldOption) (*Field, error)

CreateFieldIfNotExists creates a field with the given options if it doesn't exist.

func (*Index) DeleteField Uses

func (i *Index) DeleteField(name string) error

DeleteField removes a field from the index.

func (*Index) Field Uses

func (i *Index) Field(name string) *Field

Field returns a field in the index by name.

func (*Index) Fields Uses

func (i *Index) Fields() []*Field

Fields returns a list of all fields in the index.

func (*Index) Keys Uses

func (i *Index) Keys() bool

Keys returns true if the index uses string keys.

func (*Index) Name Uses

func (i *Index) Name() string

Name returns name of the index.

func (*Index) Open Uses

func (i *Index) Open() (err error)

Open opens and initializes the index.

func (*Index) Options Uses

func (i *Index) Options() IndexOptions

Options returns all options for this index.

func (*Index) Path Uses

func (i *Index) Path() string

Path returns the path the index was initialized with.

func (*Index) TranslateStore Uses

func (i *Index) TranslateStore() TranslateStore

TranslateStore returns the underlying translation store for the index.

type IndexInfo Uses

type IndexInfo struct {
    Name       string       `json:"name"`
    Options    IndexOptions `json:"options"`
    Fields     []*FieldInfo `json:"fields"`
    ShardWidth uint64       `json:"shardWidth"`
}

IndexInfo represents schema information for an index.

type IndexOptions Uses

type IndexOptions struct {
    Keys           bool `json:"keys"`
    TrackExistence bool `json:"trackExistence"`
}

IndexOptions represents options to set when initializing an index.

type IndexStatus Uses

type IndexStatus struct {
    Name   string
    Fields []*FieldStatus
}

IndexStatus is an internal message representing the contents of an index.

type InternalClient Uses

type InternalClient interface {
    MaxShardByIndex(ctx context.Context) (map[string]uint64, error)
    Schema(ctx context.Context) ([]*IndexInfo, error)
    PostSchema(ctx context.Context, uri *URI, s *Schema, remote bool) error
    CreateIndex(ctx context.Context, index string, opt IndexOptions) error
    FragmentNodes(ctx context.Context, index string, shard uint64) ([]*Node, error)
    Nodes(ctx context.Context) ([]*Node, error)
    Query(ctx context.Context, index string, queryRequest *QueryRequest) (*QueryResponse, error)
    QueryNode(ctx context.Context, uri *URI, index string, queryRequest *QueryRequest) (*QueryResponse, error)
    Import(ctx context.Context, index, field string, shard uint64, bits []Bit, opts ...ImportOption) error
    ImportK(ctx context.Context, index, field string, bits []Bit, opts ...ImportOption) error
    EnsureIndex(ctx context.Context, name string, options IndexOptions) error
    EnsureField(ctx context.Context, indexName string, fieldName string) error
    EnsureFieldWithOptions(ctx context.Context, index, field string, opt FieldOptions) error
    ImportValue(ctx context.Context, index, field string, shard uint64, vals []FieldValue, opts ...ImportOption) error
    ImportValueK(ctx context.Context, index, field string, vals []FieldValue, opts ...ImportOption) error
    ExportCSV(ctx context.Context, index, field string, shard uint64, w io.Writer) error
    CreateField(ctx context.Context, index, field string) error
    CreateFieldWithOptions(ctx context.Context, index, field string, opt FieldOptions) error
    FragmentBlocks(ctx context.Context, uri *URI, index, field, view string, shard uint64) ([]FragmentBlock, error)
    BlockData(ctx context.Context, uri *URI, index, field, view string, shard uint64, block int) ([]uint64, []uint64, error)
    ColumnAttrDiff(ctx context.Context, uri *URI, index string, blks []AttrBlock) (map[uint64]map[string]interface{}, error)
    RowAttrDiff(ctx context.Context, uri *URI, index, field string, blks []AttrBlock) (map[uint64]map[string]interface{}, error)
    SendMessage(ctx context.Context, uri *URI, msg []byte) error
    RetrieveShardFromURI(ctx context.Context, index, field, view string, shard uint64, uri URI) (io.ReadCloser, error)
    ImportRoaring(ctx context.Context, uri *URI, index, field string, shard uint64, remote bool, req *ImportRoaringRequest) error
}

InternalClient should be implemented by any struct that enables any transport between nodes TODO: Refactor Note from Travis: Typically an interface containing more than two or three methods is an indication that something hasn't been architected correctly. While I understand that putting the entire Client behind an interface might require this many methods, I don't want to let it go unquestioned.

type InternalQueryClient Uses

type InternalQueryClient interface {
    QueryNode(ctx context.Context, uri *URI, index string, queryRequest *QueryRequest) (*QueryResponse, error)
}

InternalQueryClient is the internal interface for querying a node.

type Message Uses

type Message interface{}

Message is the interface implemented by all core pilosa types which can be serialized to messages. TODO add at least a single "isMessage()" method.

type MultiTranslateEntryReader Uses

type MultiTranslateEntryReader struct {
    // contains filtered or unexported fields
}

MultiTranslateEntryReader reads from multiple TranslateEntryReader instances and merges them into a single reader.

func NewMultiTranslateEntryReader Uses

func NewMultiTranslateEntryReader(ctx context.Context, readers []TranslateEntryReader) *MultiTranslateEntryReader

NewMultiTranslateEntryReader returns a new instance of MultiTranslateEntryReader.

func (*MultiTranslateEntryReader) Close Uses

func (r *MultiTranslateEntryReader) Close() error

Close stops the reader & child readers and waits for all goroutines to stop.

func (*MultiTranslateEntryReader) ReadEntry Uses

func (r *MultiTranslateEntryReader) ReadEntry(entry *TranslateEntry) error

ReadEntry reads the next available entry into entry. Returns an error if any of the child readers error. Returns io.EOF if reader is closed.

type Node Uses

type Node struct {
    ID            string `json:"id"`
    URI           URI    `json:"uri"`
    IsCoordinator bool   `json:"isCoordinator"`
    State         string `json:"state"`
}

Node represents a node in the cluster.

func (*Node) Clone Uses

func (n *Node) Clone() *Node

func (Node) String Uses

func (n Node) String() string

type NodeEvent Uses

type NodeEvent struct {
    Event NodeEventType
    Node  *Node
}

NodeEvent is a single event related to node activity in the cluster.

type NodeEventType Uses

type NodeEventType int

NodeEventType are the types of node events.

const (
    NodeJoin NodeEventType = iota
    NodeLeave
    NodeUpdate
)

Constant node event types.

type NodeStateMessage Uses

type NodeStateMessage struct {
    NodeID string `protobuf:"bytes,1,opt,name=NodeID,proto3" json:"NodeID,omitempty"`
    State  string `protobuf:"bytes,2,opt,name=State,proto3" json:"State,omitempty"`
}

NodeStateMessage is an internal message for broadcasting a node's state.

type NodeStatus Uses

type NodeStatus struct {
    Node    *Node
    Indexes []*IndexStatus
    Schema  *Schema
}

NodeStatus is an internal message representing the contents of a node.

type Nodes Uses

type Nodes []*Node

Nodes represents a list of nodes.

func (Nodes) Clone Uses

func (a Nodes) Clone() []*Node

Clone returns a shallow copy of nodes.

func (Nodes) Contains Uses

func (a Nodes) Contains(n *Node) bool

Contains returns true if a node exists in the list.

func (Nodes) ContainsID Uses

func (a Nodes) ContainsID(id string) bool

ContainsID returns true if host matches one of the node's id.

func (Nodes) Filter Uses

func (a Nodes) Filter(n *Node) []*Node

Filter returns a new list of nodes with node removed.

func (Nodes) FilterID Uses

func (a Nodes) FilterID(id string) []*Node

FilterID returns a new list of nodes with ID removed.

func (Nodes) FilterURI Uses

func (a Nodes) FilterURI(uri URI) []*Node

FilterURI returns a new list of nodes with URI removed.

func (Nodes) IDs Uses

func (a Nodes) IDs() []string

IDs returns a list of all node IDs.

func (Nodes) URIs Uses

func (a Nodes) URIs() []URI

URIs returns a list of all uris.

type NotFoundError Uses

type NotFoundError struct {
    // contains filtered or unexported fields
}

NotFoundError wraps an error value to signify that a resource was not found such that in an HTTP scenario, http.StatusNotFound would be returned.

type OpenTranslateReaderFunc Uses

type OpenTranslateReaderFunc func(ctx context.Context, nodeURL string, offsets TranslateOffsetMap) (TranslateEntryReader, error)

OpenTranslateReaderFunc represents a function for instantiating and opening a TranslateStore.

type OpenTranslateStoreFunc Uses

type OpenTranslateStoreFunc func(path, index, field string) (TranslateStore, error)

OpenTranslateStoreFunc represents a function for instantiating and opening a TranslateStore.

type Pair Uses

type Pair struct {
    ID    uint64 `json:"id"`
    Key   string `json:"key,omitempty"`
    Count uint64 `json:"count"`
}

Pair holds an id/count pair.

type Pairs Uses

type Pairs []Pair

Pairs is a sortable slice of Pair objects.

func (Pairs) Add Uses

func (p Pairs) Add(other []Pair) []Pair

Add merges other into p and returns a new slice.

func (Pairs) Keys Uses

func (p Pairs) Keys() []uint64

Keys returns a slice of all keys in p.

func (Pairs) Len Uses

func (p Pairs) Len() int

func (Pairs) Less Uses

func (p Pairs) Less(i, j int) bool

func (*Pairs) Pop Uses

func (p *Pairs) Pop() interface{}

Pop removes the minimum element from the Pair slice.

func (*Pairs) Push Uses

func (p *Pairs) Push(x interface{})

Push appends the element onto the Pair slice.

func (Pairs) String Uses

func (p Pairs) String() string

func (Pairs) Swap Uses

func (p Pairs) Swap(i, j int)

type QueryRequest Uses

type QueryRequest struct {
    // Index to execute query against.
    Index string

    // The query string to parse and execute.
    Query string

    // The shards to include in the query execution.
    // If empty, all shards are included.
    Shards []uint64

    // Return column attributes, if true.
    ColumnAttrs bool

    // Do not return row attributes, if true.
    ExcludeRowAttrs bool

    // Do not return columns, if true.
    ExcludeColumns bool

    // If true, indicates that query is part of a larger distributed query.
    // If false, this request is on the originating node.
    Remote bool
}

QueryRequest represent a request to process a query.

type QueryResponse Uses

type QueryResponse struct {
    // Result for each top-level query call.
    // Can be a Bitmap, Pairs, or uint64.
    Results []interface{}

    // Set of column attribute objects matching IDs returned in Result.
    ColumnAttrSets []*ColumnAttrSet

    // Error during parsing or execution.
    Err error
}

QueryResponse represent a response from a processed query.

func (*QueryResponse) MarshalJSON Uses

func (resp *QueryResponse) MarshalJSON() ([]byte, error)

MarshalJSON marshals QueryResponse into a JSON-encoded byte slice

type RecalculateCaches Uses

type RecalculateCaches struct{}

RecalculateCaches is an internal message for recalculating all caches within a holder.

type ResizeInstruction Uses

type ResizeInstruction struct {
    JobID         int64
    Node          *Node
    Coordinator   *Node
    Sources       []*ResizeSource
    NodeStatus    *NodeStatus
    ClusterStatus *ClusterStatus
}

ResizeInstruction contains the instruction provided to a node during a cluster resize operation.

type ResizeInstructionComplete Uses

type ResizeInstructionComplete struct {
    JobID int64
    Node  *Node
    Error string
}

ResizeInstructionComplete is an internal message to the coordinator indicating that the resize instructions performed on a single node have completed.

type ResizeSource Uses

type ResizeSource struct {
    Node  *Node  `protobuf:"bytes,1,opt,name=Node" json:"Node,omitempty"`
    Index string `protobuf:"bytes,2,opt,name=Index,proto3" json:"Index,omitempty"`
    Field string `protobuf:"bytes,3,opt,name=Field,proto3" json:"Field,omitempty"`
    View  string `protobuf:"bytes,4,opt,name=View,proto3" json:"View,omitempty"`
    Shard uint64 `protobuf:"varint,5,opt,name=Shard,proto3" json:"Shard,omitempty"`
}

ResizeSource is the source of data for a node acting on a ResizeInstruction.

type Row Uses

type Row struct {

    // String keys translated to/from segment columns.
    Keys []string

    // Attributes associated with the row.
    Attrs map[string]interface{}
    // contains filtered or unexported fields
}

Row is a set of integers (the associated columns), and attributes which are arbitrary key/value pairs storing metadata about what the row represents.

func NewRow Uses

func NewRow(columns ...uint64) *Row

NewRow returns a new instance of Row.

func (*Row) Any Uses

func (r *Row) Any() bool

Any returns true if row contains any bits.

func (*Row) Columns Uses

func (r *Row) Columns() []uint64

Columns returns the columns in r as a slice of ints.

func (*Row) Count Uses

func (r *Row) Count() uint64

Count returns the number of columns in the row.

func (*Row) Difference Uses

func (r *Row) Difference(other *Row) *Row

Difference returns the diff of r and other.

func (*Row) Freeze Uses

func (r *Row) Freeze()

func (*Row) Intersect Uses

func (r *Row) Intersect(other *Row) *Row

Intersect returns the itersection of r and other.

func (*Row) IsEmpty Uses

func (r *Row) IsEmpty() bool

IsEmpty returns true if the row doesn't contain any set bits.

func (*Row) MarshalJSON Uses

func (r *Row) MarshalJSON() ([]byte, error)

MarshalJSON returns a JSON-encoded byte slice of r.

func (*Row) Merge Uses

func (r *Row) Merge(other *Row)

Merge merges data from other into r.

func (*Row) Segments Uses

func (r *Row) Segments() []rowSegment

Segments returns a list of all segments in the row.

func (*Row) SetBit Uses

func (r *Row) SetBit(i uint64) (changed bool)

SetBit sets the i-th column of the row.

func (*Row) Shift Uses

func (r *Row) Shift(n int64) (*Row, error)

Shift returns the bitwise shift of r by n bits. Currently only positive shift values are supported.

func (*Row) Union Uses

func (r *Row) Union(others ...*Row) *Row

Union returns the bitwise union of r and other.

func (*Row) Xor Uses

func (r *Row) Xor(other *Row) *Row

Xor returns the xor of r and other.

type RowIDs Uses

type RowIDs []uint64

RowIDs is a query return type for just uint64 row ids. It should only be used internally (since RowIdentifiers is the external return type), but it is exported because the proto package needs access to it.

type RowIdentifiers Uses

type RowIdentifiers struct {
    Rows []uint64 `json:"rows"`
    Keys []string `json:"keys,omitempty"`
}

RowIdentifiers is a return type for a list of row ids or row keys. The names `Rows` and `Keys` are meant to follow the same convention as the Row query which returns `Columns` and `Keys`. TODO: Rename this to something better. Anything.

type Schema Uses

type Schema struct {
    Indexes []*IndexInfo
}

Schema contains information about indexes and their configuration.

type Serializer Uses

type Serializer interface {
    Marshal(Message) ([]byte, error)
    Unmarshal([]byte, Message) error
}

Serializer is an interface for serializing pilosa types to bytes and back.

type Server Uses

type Server struct {
    // contains filtered or unexported fields
}

Server represents a holder wrapped by a running HTTP server.

func NewServer Uses

func NewServer(opts ...ServerOption) (*Server, error)

NewServer returns a new instance of Server.

func (*Server) Close Uses

func (s *Server) Close() error

Close closes the server and waits for it to shutdown.

func (*Server) Holder Uses

func (s *Server) Holder() *Holder

Holder returns the holder for server. TODO: have this return an interface for Holder instead of concrete object?

func (*Server) InternalClient Uses

func (s *Server) InternalClient() InternalClient

func (*Server) NodeID Uses

func (s *Server) NodeID() string

NodeID returns the server's node id.

func (*Server) Open Uses

func (s *Server) Open() error

Open opens and initializes the server.

func (*Server) SendAsync Uses

func (s *Server) SendAsync(m Message) error

SendAsync represents an implementation of Broadcaster.

func (*Server) SendSync Uses

func (s *Server) SendSync(m Message) error

SendSync represents an implementation of Broadcaster.

func (*Server) SendTo Uses

func (s *Server) SendTo(to *Node, m Message) error

SendTo represents an implementation of Broadcaster.

func (*Server) SyncData Uses

func (s *Server) SyncData() error

SyncData manually invokes the anti entropy process which makes sure that this node has the data from all replicas across the cluster.

func (*Server) UpAndDown Uses

func (s *Server) UpAndDown() error

UpAndDown brings the server up minimally and shuts it down again; basically, it exists for testing holder open and close.

type ServerOption Uses

type ServerOption func(s *Server) error

ServerOption is a functional option type for pilosa.Server

func OptServerAntiEntropyInterval Uses

func OptServerAntiEntropyInterval(interval time.Duration) ServerOption

OptServerAntiEntropyInterval is a functional option on Server used to set the anti-entropy interval.

func OptServerAttrStoreFunc Uses

func OptServerAttrStoreFunc(af func(string) AttrStore) ServerOption

OptServerAttrStoreFunc is a functional option on Server used to provide the function to use to generate a new attribute store.

func OptServerClusterDisabled Uses

func OptServerClusterDisabled(disabled bool, hosts []string) ServerOption

OptServerClusterDisabled tells the server whether to use a static cluster with the defined hosts. Mostly used for testing.

func OptServerClusterHasher Uses

func OptServerClusterHasher(h Hasher) ServerOption

OptServerClusterHasher is a functional option on Server used to specify the consistent hash algorithm for data location within the cluster.

func OptServerDataDir Uses

func OptServerDataDir(dir string) ServerOption

OptServerDataDir is a functional option on Server used to set the data directory.

func OptServerDiagnosticsInterval Uses

func OptServerDiagnosticsInterval(dur time.Duration) ServerOption

OptServerDiagnosticsInterval is a functional option on Server used to specify the duration between diagnostic checks.

func OptServerExecutorPoolSize Uses

func OptServerExecutorPoolSize(size int) ServerOption

func OptServerGCNotifier Uses

func OptServerGCNotifier(gcn GCNotifier) ServerOption

OptServerGCNotifier is a functional option on Server used to set the garbage collection notification source.

func OptServerInternalClient Uses

func OptServerInternalClient(c InternalClient) ServerOption

OptServerInternalClient is a functional option on Server used to set the implementation of InternalClient.

func OptServerIsCoordinator Uses

func OptServerIsCoordinator(is bool) ServerOption

OptServerIsCoordinator is a functional option on Server used to specify whether or not this server is the coordinator.

func OptServerLogger Uses

func OptServerLogger(l logger.Logger) ServerOption

OptServerLogger is a functional option on Server used to set the logger.

func OptServerLongQueryTime Uses

func OptServerLongQueryTime(dur time.Duration) ServerOption

OptServerLongQueryTime is a functional option on Server used to set long query duration.

func OptServerMaxWritesPerRequest Uses

func OptServerMaxWritesPerRequest(n int) ServerOption

OptServerMaxWritesPerRequest is a functional option on Server used to set the maximum number of writes allowed per request.

func OptServerMetricInterval Uses

func OptServerMetricInterval(dur time.Duration) ServerOption

OptServerMetricInterval is a functional option on Server used to set the interval between metric samples.

func OptServerNodeID Uses

func OptServerNodeID(nodeID string) ServerOption

OptServerNodeID is a functional option on Server used to set the server node ID.

func OptServerOpenTranslateReader Uses

func OptServerOpenTranslateReader(fn OpenTranslateReaderFunc) ServerOption

OptServerOpenTranslateReader is a functional option on Server used to specify the remote translation data reader.

func OptServerOpenTranslateStore Uses

func OptServerOpenTranslateStore(fn OpenTranslateStoreFunc) ServerOption

OptServerOpenTranslateStore is a functional option on Server used to specify the translation data store type.

func OptServerPrimaryTranslateStore Uses

func OptServerPrimaryTranslateStore(store TranslateStore) ServerOption

OptServerPrimaryTranslateStore has been deprecated.

func OptServerReplicaN Uses

func OptServerReplicaN(n int) ServerOption

OptServerReplicaN is a functional option on Server used to set the number of replicas.

func OptServerSerializer Uses

func OptServerSerializer(ser Serializer) ServerOption

OptServerSerializer is a functional option on Server used to set the serializer.

func OptServerStatsClient Uses

func OptServerStatsClient(sc stats.StatsClient) ServerOption

OptServerStatsClient is a functional option on Server used to specify the stats client.

func OptServerSystemInfo Uses

func OptServerSystemInfo(si SystemInfo) ServerOption

OptServerSystemInfo is a functional option on Server used to set the system information source.

func OptServerURI Uses

func OptServerURI(uri *URI) ServerOption

OptServerURI is a functional option on Server used to set the server URI.

type SetCoordinatorMessage Uses

type SetCoordinatorMessage struct {
    New *Node
}

SetCoordinatorMessage is an internal message instructing nodes to honor a new coordinator.

type SystemInfo Uses

type SystemInfo interface {
    Uptime() (uint64, error)
    Platform() (string, error)
    Family() (string, error)
    OSVersion() (string, error)
    KernelVersion() (string, error)
    MemFree() (uint64, error)
    MemTotal() (uint64, error)
    MemUsed() (uint64, error)
    CPUModel() string
    CPUCores() (physical int, logical int, err error)
    CPUMHz() (int, error)
    CPUArch() string
}

SystemInfo collects information about the host OS.

type TimeQuantum Uses

type TimeQuantum string

TimeQuantum represents a time granularity for time-based bitmaps.

func (TimeQuantum) HasDay Uses

func (q TimeQuantum) HasDay() bool

HasDay returns true if the quantum contains a 'D' unit.

func (TimeQuantum) HasHour Uses

func (q TimeQuantum) HasHour() bool

HasHour returns true if the quantum contains a 'H' unit.

func (TimeQuantum) HasMonth Uses

func (q TimeQuantum) HasMonth() bool

HasMonth returns true if the quantum contains a 'M' unit.

func (TimeQuantum) HasYear Uses

func (q TimeQuantum) HasYear() bool

HasYear returns true if the quantum contains a 'Y' unit.

func (*TimeQuantum) Set Uses

func (q *TimeQuantum) Set(value string) error

Set sets the time quantum value.

func (TimeQuantum) String Uses

func (q TimeQuantum) String() string

func (TimeQuantum) Type Uses

func (q TimeQuantum) Type() string

Type returns the type of a time quantum value.

func (TimeQuantum) Valid Uses

func (q TimeQuantum) Valid() bool

Valid returns true if q is a valid time quantum value.

type Topology Uses

type Topology struct {
    // contains filtered or unexported fields
}

Topology represents the list of hosts in the cluster.

func (*Topology) ContainsID Uses

func (t *Topology) ContainsID(id string) bool

ContainsID returns true if id matches one of the topology's IDs.

type TranslateEntry Uses

type TranslateEntry struct {
    Index string `json:"index,omitempty"`
    Field string `json:"field,omitempty"`
    ID    uint64 `json:"id,omitempty"`
    Key   string `json:"key,omitempty"`
}

TranslateEntry represents a key/ID pair from a TranslateStore.

type TranslateEntryReader Uses

type TranslateEntryReader interface {
    io.Closer
    ReadEntry(entry *TranslateEntry) error
}

TranslateEntryReader represents a stream of translation entries.

type TranslateKeysRequest Uses

type TranslateKeysRequest struct {
    Index string
    Field string
    Keys  []string
}

TranslateKeysRequest describes the structure of a request for a batch of key translations.

type TranslateKeysResponse Uses

type TranslateKeysResponse struct {
    IDs []uint64
}

TranslateKeysResponse is the structured response of a key translation request.

type TranslateOffsetMap Uses

type TranslateOffsetMap map[string]map[string]uint64

TranslateOffsetMap maintains a set of offsets for both indexes & fields.

func (TranslateOffsetMap) FieldOffset Uses

func (m TranslateOffsetMap) FieldOffset(index, name string) uint64

FieldOffset returns the offset for the given field.

func (TranslateOffsetMap) IndexOffset Uses

func (m TranslateOffsetMap) IndexOffset(name string) uint64

IndexOffset returns the offset for the given index.

func (TranslateOffsetMap) SetFieldOffset Uses

func (m TranslateOffsetMap) SetFieldOffset(index, name string, offset uint64)

SetFieldOffset sets the offset for the given field.

func (TranslateOffsetMap) SetIndexOffset Uses

func (m TranslateOffsetMap) SetIndexOffset(name string, offset uint64)

SetIndexOffset sets the offset for the given index.

type TranslateStore Uses

type TranslateStore interface {
    io.Closer

    // Returns the maximum ID set on the store.
    MaxID() (uint64, error)

    // Sets & retrieves whether the store is read-only.
    ReadOnly() bool
    SetReadOnly(v bool)

    // Converts a string key to its autoincrementing integer ID value.
    TranslateKey(key string) (uint64, error)
    TranslateKeys(key []string) ([]uint64, error)

    // Converts an integer ID to its associated string key.
    TranslateID(id uint64) (string, error)
    TranslateIDs(id []uint64) ([]string, error)

    // Forces the write of a key/id pair, even if read only. Used by replication.
    ForceSet(id uint64, key string) error

    // Returns a reader from the given ID offset.
    EntryReader(ctx context.Context, offset uint64) (TranslateEntryReader, error)
}

TranslateStore is the storage for translation string-to-uint64 values.

func OpenInMemTranslateStore Uses

func OpenInMemTranslateStore(rawurl, index, field string) (TranslateStore, error)

OpenInMemTranslateStore returns a new instance of InMemTranslateStore. Implements OpenTranslateStoreFunc.

type URI Uses

type URI struct {
    Scheme string `json:"scheme"`
    Host   string `json:"host"`
    Port   uint16 `json:"port"`
}

URI represents a Pilosa URI. A Pilosa URI consists of three parts: 1) Scheme: Protocol of the URI. Default: http. 2) Host: Hostname or IP URI. Default: localhost. IPv6 addresses should be written in brackets, e.g., `[fd42:4201:f86b:7e09:216:3eff:fefa:ed80]`. 3) Port: Port of the URI. Default: 10101.

All parts of the URI are optional. The following are equivalent:

http://localhost:10101
http://localhost
http://:10101
localhost:10101
localhost
:10101

func AddressWithDefaults Uses

func AddressWithDefaults(addr string) (*URI, error)

AddressWithDefaults converts addr into a valid address, using defaults when necessary.

func NewURIFromAddress Uses

func NewURIFromAddress(address string) (*URI, error)

NewURIFromAddress parses the passed address and returns a URI.

func NewURIFromHostPort Uses

func NewURIFromHostPort(host string, port uint16) (*URI, error)

NewURIFromHostPort returns a URI with specified host and port.

func (*URI) HostPort Uses

func (u *URI) HostPort() string

HostPort returns `Host:Port`

func (*URI) MarshalJSON Uses

func (u *URI) MarshalJSON() ([]byte, error)

MarshalJSON marshals URI into a JSON-encoded byte slice.

func (*URI) Path Uses

func (u *URI) Path(path string) string

Path returns URI with path

func (*URI) Set Uses

func (u *URI) Set(value string) error

Set sets the uri value.

func (*URI) SetPort Uses

func (u *URI) SetPort(port uint16)

SetPort sets the port of this URI.

func (URI) String Uses

func (u URI) String() string

String returns the address as a string.

func (URI) Type Uses

func (u URI) Type() string

Type returns the type of a uri.

func (*URI) URL Uses

func (u *URI) URL() url.URL

URL returns a url.URL representation of the URI.

func (*URI) UnmarshalJSON Uses

func (u *URI) UnmarshalJSON(b []byte) error

UnmarshalJSON unmarshals a byte slice to a URI.

type URIs Uses

type URIs []URI

URIs is a convenience type representing a slice of URI.

func (URIs) HostPortStrings Uses

func (u URIs) HostPortStrings() []string

HostPortStrings returns a slice of host:port strings based on the slice of URI.

type UpdateCoordinatorMessage Uses

type UpdateCoordinatorMessage struct {
    New *Node
}

UpdateCoordinatorMessage is an internal message for reassigning the coordinator.

type ValCount Uses

type ValCount struct {
    Val   int64 `json:"value"`
    Count int64 `json:"count"`
}

ValCount represents a grouping of sum & count for Sum() and Average() calls.

type ViewInfo Uses

type ViewInfo struct {
    Name string `json:"name"`
}

ViewInfo represents schema information for a view.

Directories

PathSynopsis
boltdb
cmdPackage cmd contains all the pilosa subcommand definitions (1 per file).
cmd/pilosaThis is the entrypoint for the Pilosa binary.
ctlpackage ctl contains all pilosa subcommands other than 'server'.
encoding/proto
enterprisePackage enterprise is now deprecated, and the functionality under enterprise/b has been copied into roaring/.
gcnotify
gopsutil
gossip
http
internalPackage internal is a generated protocol buffer package.
internal/test
logger
lruPackage lru implements an LRU cache.
mock
pqlPackage pql defines the Pilosa Query Language.
prometheus
roaringPackage roaring implements roaring bitmaps with support for incremental changes.
server
shardwidth
stats
statsd
syswrapPackage syswrap wraps syscalls (just mmap right now) in order to impose a global in-process limit on the maximum number of active mmaps.
test
toml
tracing
tracing/opentracing

Package pilosa imports 47 packages (graph) and is imported by 38 packages. Updated 2019-11-15. Refresh now. Tools for package owners.