core: go.gazette.dev/core/consumer/protocol Index | Files

package protocol

import "go.gazette.dev/core/consumer/protocol"

Package protocol defines the consumer datamodel, validation behaviors, and gRPC APIs which are shared across clients and consumer application servers. Datamodel types and APIs are implemented as generated protobuf messages and stubs, typically extended with additional parsing, validation, and shared implementation behaviors. A central goal of this package to be highly exacting in defining allowed "shapes" that types & messages may take (through implementations of the Validator interface), providing strong guarantees to consumers and clients that messages are well-formed without need for additional ad-hoc, repetitive checks (which often become a maintenance burden).

By convention, this package is usually imported as `pc`, short for "Protocol of Consumer", due to it's ubiquity and to distinguish it from package go.gazette.dev/core/broker/protocol (imported as `pb`). Eg,

import pc "go.gazette.dev/core/consumer/protocol"

Index

Package Files

checkpoint_extensions.go doc.go protocol.pb.go rpc_extensions.go shard_spec_extensions.go

Variables

var (
    ErrInvalidLengthProtocol        = fmt.Errorf("proto: negative length found during unmarshaling")
    ErrIntOverflowProtocol          = fmt.Errorf("proto: integer overflow")
    ErrUnexpectedEndOfGroupProtocol = fmt.Errorf("proto: unexpected end of group")
)
var ReplicaStatus_Code_name = map[int32]string{
    0:   "IDLE",
    100: "BACKFILL",
    200: "STANDBY",
    300: "PRIMARY",
    400: "FAILED",
}
var ReplicaStatus_Code_value = map[string]int32{
    "IDLE":     0,
    "BACKFILL": 100,
    "STANDBY":  200,
    "PRIMARY":  300,
    "FAILED":   400,
}
var Status_name = map[int32]string{
    0:  "OK",
    1:  "SHARD_NOT_FOUND",
    2:  "NO_SHARD_PRIMARY",
    3:  "NOT_SHARD_PRIMARY",
    4:  "ETCD_TRANSACTION_FAILED",
}
var Status_value = map[string]int32{
    "OK":                      0,
    "SHARD_NOT_FOUND":         1,
    "NO_SHARD_PRIMARY":        2,
    "NOT_SHARD_PRIMARY":       3,
    "ETCD_TRANSACTION_FAILED": 4,
}

func ExtractShardSpecMetaLabels Uses

func ExtractShardSpecMetaLabels(spec *ShardSpec, out pb.LabelSet) pb.LabelSet

ExtractShardSpecMetaLabels returns meta-labels of the ShardSpec, using |out| as a buffer.

func FlattenProducerStates Uses

func FlattenProducerStates(cp Checkpoint) []message.ProducerState

FlattenProducerStates returns a []ProducerState drawn from the Checkpoint.

func FlattenReadThrough Uses

func FlattenReadThrough(cp Checkpoint) pb.Offsets

FlattenReadThrough returns Offsets drawn from the Checkpoint.

func RegisterShardServer Uses

func RegisterShardServer(s *grpc.Server, srv ShardServer)

type ApplyRequest Uses

type ApplyRequest struct {
    Changes []ApplyRequest_Change `protobuf:"bytes,1,rep,name=changes,proto3" json:"changes"`
}

func (*ApplyRequest) Descriptor Uses

func (*ApplyRequest) Descriptor() ([]byte, []int)

func (*ApplyRequest) Marshal Uses

func (m *ApplyRequest) Marshal() (dAtA []byte, err error)

func (*ApplyRequest) MarshalTo Uses

func (m *ApplyRequest) MarshalTo(dAtA []byte) (int, error)

func (*ApplyRequest) MarshalToSizedBuffer Uses

func (m *ApplyRequest) MarshalToSizedBuffer(dAtA []byte) (int, error)

func (*ApplyRequest) ProtoMessage Uses

func (*ApplyRequest) ProtoMessage()

func (*ApplyRequest) ProtoSize Uses

func (m *ApplyRequest) ProtoSize() (n int)

func (*ApplyRequest) Reset Uses

func (m *ApplyRequest) Reset()

func (*ApplyRequest) String Uses

func (m *ApplyRequest) String() string

func (*ApplyRequest) Unmarshal Uses

func (m *ApplyRequest) Unmarshal(dAtA []byte) error

func (*ApplyRequest) Validate Uses

func (m *ApplyRequest) Validate() error

Validate returns an error if the ApplyRequest is not well-formed.

func (*ApplyRequest) XXX_DiscardUnknown Uses

func (m *ApplyRequest) XXX_DiscardUnknown()

func (*ApplyRequest) XXX_Marshal Uses

func (m *ApplyRequest) XXX_Marshal(b []byte, deterministic bool) ([]byte, error)

func (*ApplyRequest) XXX_Merge Uses

func (m *ApplyRequest) XXX_Merge(src proto.Message)

func (*ApplyRequest) XXX_Size Uses

func (m *ApplyRequest) XXX_Size() int

func (*ApplyRequest) XXX_Unmarshal Uses

func (m *ApplyRequest) XXX_Unmarshal(b []byte) error

type ApplyRequest_Change Uses

type ApplyRequest_Change struct {
    // Expected ModRevision of the current ShardSpec. If the shard is being
    // created, expect_mod_revision is zero.
    ExpectModRevision int64 `protobuf:"varint,1,opt,name=expect_mod_revision,json=expectModRevision,proto3" json:"expect_mod_revision,omitempty"`
    // ShardSpec to be updated (if expect_mod_revision > 0) or created
    // (if expect_mod_revision == 0).
    Upsert *ShardSpec `protobuf:"bytes,2,opt,name=upsert,proto3" json:"upsert,omitempty"`
    // Shard to be deleted. expect_mod_revision must not be zero.
    Delete ShardID `protobuf:"bytes,3,opt,name=delete,proto3,casttype=ShardID" json:"delete,omitempty"`
}

Change defines an insertion, update, or deletion to be applied to the set of ShardSpecs. Exactly one of |upsert| or |delete| must be set.

func (*ApplyRequest_Change) Descriptor Uses

func (*ApplyRequest_Change) Descriptor() ([]byte, []int)

func (*ApplyRequest_Change) Marshal Uses

func (m *ApplyRequest_Change) Marshal() (dAtA []byte, err error)

func (*ApplyRequest_Change) MarshalTo Uses

func (m *ApplyRequest_Change) MarshalTo(dAtA []byte) (int, error)

func (*ApplyRequest_Change) MarshalToSizedBuffer Uses

func (m *ApplyRequest_Change) MarshalToSizedBuffer(dAtA []byte) (int, error)

func (*ApplyRequest_Change) ProtoMessage Uses

func (*ApplyRequest_Change) ProtoMessage()

func (*ApplyRequest_Change) ProtoSize Uses

func (m *ApplyRequest_Change) ProtoSize() (n int)

func (*ApplyRequest_Change) Reset Uses

func (m *ApplyRequest_Change) Reset()

func (*ApplyRequest_Change) String Uses

func (m *ApplyRequest_Change) String() string

func (*ApplyRequest_Change) Unmarshal Uses

func (m *ApplyRequest_Change) Unmarshal(dAtA []byte) error

func (*ApplyRequest_Change) Validate Uses

func (m *ApplyRequest_Change) Validate() error

Validate returns an error if the ApplyRequest_Change is not well-formed.

func (*ApplyRequest_Change) XXX_DiscardUnknown Uses

func (m *ApplyRequest_Change) XXX_DiscardUnknown()

func (*ApplyRequest_Change) XXX_Marshal Uses

func (m *ApplyRequest_Change) XXX_Marshal(b []byte, deterministic bool) ([]byte, error)

func (*ApplyRequest_Change) XXX_Merge Uses

func (m *ApplyRequest_Change) XXX_Merge(src proto.Message)

func (*ApplyRequest_Change) XXX_Size Uses

func (m *ApplyRequest_Change) XXX_Size() int

func (*ApplyRequest_Change) XXX_Unmarshal Uses

func (m *ApplyRequest_Change) XXX_Unmarshal(b []byte) error

type ApplyResponse Uses

type ApplyResponse struct {
    // Status of the Apply RPC.
    Status Status `protobuf:"varint,1,opt,name=status,proto3,enum=consumer.Status" json:"status,omitempty"`
    // Header of the response.
    Header protocol.Header `protobuf:"bytes,2,opt,name=header,proto3" json:"header"`
}

func (*ApplyResponse) Descriptor Uses

func (*ApplyResponse) Descriptor() ([]byte, []int)

func (*ApplyResponse) Marshal Uses

func (m *ApplyResponse) Marshal() (dAtA []byte, err error)

func (*ApplyResponse) MarshalTo Uses

func (m *ApplyResponse) MarshalTo(dAtA []byte) (int, error)

func (*ApplyResponse) MarshalToSizedBuffer Uses

func (m *ApplyResponse) MarshalToSizedBuffer(dAtA []byte) (int, error)

func (*ApplyResponse) ProtoMessage Uses

func (*ApplyResponse) ProtoMessage()

func (*ApplyResponse) ProtoSize Uses

func (m *ApplyResponse) ProtoSize() (n int)

func (*ApplyResponse) Reset Uses

func (m *ApplyResponse) Reset()

func (*ApplyResponse) String Uses

func (m *ApplyResponse) String() string

func (*ApplyResponse) Unmarshal Uses

func (m *ApplyResponse) Unmarshal(dAtA []byte) error

func (*ApplyResponse) Validate Uses

func (m *ApplyResponse) Validate() error

Validate returns an error if the ApplyResponse is not well-formed.

func (*ApplyResponse) XXX_DiscardUnknown Uses

func (m *ApplyResponse) XXX_DiscardUnknown()

func (*ApplyResponse) XXX_Marshal Uses

func (m *ApplyResponse) XXX_Marshal(b []byte, deterministic bool) ([]byte, error)

func (*ApplyResponse) XXX_Merge Uses

func (m *ApplyResponse) XXX_Merge(src proto.Message)

func (*ApplyResponse) XXX_Size Uses

func (m *ApplyResponse) XXX_Size() int

func (*ApplyResponse) XXX_Unmarshal Uses

func (m *ApplyResponse) XXX_Unmarshal(b []byte) error

type BuildCheckpointArgs Uses

type BuildCheckpointArgs struct {
    ReadThrough    pb.Offsets
    ProducerStates []message.ProducerState
    AckIntents     []message.AckIntent
}

BuildCheckpointArgs are arguments of BuildCheckpoint.

type Checkpoint Uses

type Checkpoint struct {
    // Sources is metadata of journals consumed by the shard.
    Sources map[go_gazette_dev_core_broker_protocol.Journal]Checkpoint_Source `protobuf:"bytes,1,rep,name=sources,proto3,castkey=go.gazette.dev/core/broker/protocol.Journal" json:"sources" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"`
    // AckIntents is acknowledgement intents to be written to journals to which
    // uncommitted messages were published during the transaction which produced
    // this Checkpoint.
    AckIntents map[go_gazette_dev_core_broker_protocol.Journal][]byte `protobuf:"bytes,2,rep,name=ack_intents,json=ackIntents,proto3,castkey=go.gazette.dev/core/broker/protocol.Journal" json:"ack_intents,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"`
}

Checkpoint is processing metadata of a consumer shard which allows for its recovery on fault.

func BuildCheckpoint Uses

func BuildCheckpoint(args BuildCheckpointArgs) Checkpoint

BuildCheckpoint builds a Checkpoint message instance from the arguments.

func (*Checkpoint) Descriptor Uses

func (*Checkpoint) Descriptor() ([]byte, []int)

func (*Checkpoint) Marshal Uses

func (m *Checkpoint) Marshal() (dAtA []byte, err error)

func (*Checkpoint) MarshalTo Uses

func (m *Checkpoint) MarshalTo(dAtA []byte) (int, error)

func (*Checkpoint) MarshalToSizedBuffer Uses

func (m *Checkpoint) MarshalToSizedBuffer(dAtA []byte) (int, error)

func (*Checkpoint) ProtoMessage Uses

func (*Checkpoint) ProtoMessage()

func (*Checkpoint) ProtoSize Uses

func (m *Checkpoint) ProtoSize() (n int)

func (*Checkpoint) Reset Uses

func (m *Checkpoint) Reset()

func (*Checkpoint) String Uses

func (m *Checkpoint) String() string

func (*Checkpoint) Unmarshal Uses

func (m *Checkpoint) Unmarshal(dAtA []byte) error

func (*Checkpoint) XXX_DiscardUnknown Uses

func (m *Checkpoint) XXX_DiscardUnknown()

func (*Checkpoint) XXX_Marshal Uses

func (m *Checkpoint) XXX_Marshal(b []byte, deterministic bool) ([]byte, error)

func (*Checkpoint) XXX_Merge Uses

func (m *Checkpoint) XXX_Merge(src proto.Message)

func (*Checkpoint) XXX_Size Uses

func (m *Checkpoint) XXX_Size() int

func (*Checkpoint) XXX_Unmarshal Uses

func (m *Checkpoint) XXX_Unmarshal(b []byte) error

type Checkpoint_ProducerState Uses

type Checkpoint_ProducerState struct {
    // LastAck is the last acknowledged Clock of this producer.
    LastAck go_gazette_dev_core_message.Clock `protobuf:"fixed64,1,opt,name=last_ack,json=lastAck,proto3,casttype=go.gazette.dev/core/message.Clock" json:"last_ack,omitempty"`
    // Begin is the offset of the first message byte having CONTINUE_TXN that's
    // larger than LastAck. Eg, it's the offset which opens the next
    // transaction. If there is no such message, Begin is -1.
    Begin go_gazette_dev_core_broker_protocol.Offset `protobuf:"varint,2,opt,name=begin,proto3,casttype=go.gazette.dev/core/broker/protocol.Offset" json:"begin,omitempty"`
}

ProducerState is metadata of a producer as-of a read-through journal offset.

func (*Checkpoint_ProducerState) Descriptor Uses

func (*Checkpoint_ProducerState) Descriptor() ([]byte, []int)

func (*Checkpoint_ProducerState) Marshal Uses

func (m *Checkpoint_ProducerState) Marshal() (dAtA []byte, err error)

func (*Checkpoint_ProducerState) MarshalTo Uses

func (m *Checkpoint_ProducerState) MarshalTo(dAtA []byte) (int, error)

func (*Checkpoint_ProducerState) MarshalToSizedBuffer Uses

func (m *Checkpoint_ProducerState) MarshalToSizedBuffer(dAtA []byte) (int, error)

func (*Checkpoint_ProducerState) ProtoMessage Uses

func (*Checkpoint_ProducerState) ProtoMessage()

func (*Checkpoint_ProducerState) ProtoSize Uses

func (m *Checkpoint_ProducerState) ProtoSize() (n int)

func (*Checkpoint_ProducerState) Reset Uses

func (m *Checkpoint_ProducerState) Reset()

func (*Checkpoint_ProducerState) String Uses

func (m *Checkpoint_ProducerState) String() string

func (*Checkpoint_ProducerState) Unmarshal Uses

func (m *Checkpoint_ProducerState) Unmarshal(dAtA []byte) error

func (*Checkpoint_ProducerState) XXX_DiscardUnknown Uses

func (m *Checkpoint_ProducerState) XXX_DiscardUnknown()

func (*Checkpoint_ProducerState) XXX_Marshal Uses

func (m *Checkpoint_ProducerState) XXX_Marshal(b []byte, deterministic bool) ([]byte, error)

func (*Checkpoint_ProducerState) XXX_Merge Uses

func (m *Checkpoint_ProducerState) XXX_Merge(src proto.Message)

func (*Checkpoint_ProducerState) XXX_Size Uses

func (m *Checkpoint_ProducerState) XXX_Size() int

func (*Checkpoint_ProducerState) XXX_Unmarshal Uses

func (m *Checkpoint_ProducerState) XXX_Unmarshal(b []byte) error

type Checkpoint_Source Uses

type Checkpoint_Source struct {
    // Offset of the journal which has been read-through.
    ReadThrough go_gazette_dev_core_broker_protocol.Offset `protobuf:"varint,1,opt,name=read_through,json=readThrough,proto3,casttype=go.gazette.dev/core/broker/protocol.Offset" json:"read_through,omitempty"`
    // States of journal producers. Producer keys are 6-byte,
    // RFC 4122 v1 node identifiers (see message.ProducerID).
    Producers map[string]Checkpoint_ProducerState `protobuf:"bytes,2,rep,name=producers,proto3" json:"producers" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"`
}

Source is metadata of a consumed source journal.

func (*Checkpoint_Source) Descriptor Uses

func (*Checkpoint_Source) Descriptor() ([]byte, []int)

func (*Checkpoint_Source) Marshal Uses

func (m *Checkpoint_Source) Marshal() (dAtA []byte, err error)

func (*Checkpoint_Source) MarshalTo Uses

func (m *Checkpoint_Source) MarshalTo(dAtA []byte) (int, error)

func (*Checkpoint_Source) MarshalToSizedBuffer Uses

func (m *Checkpoint_Source) MarshalToSizedBuffer(dAtA []byte) (int, error)

func (*Checkpoint_Source) ProtoMessage Uses

func (*Checkpoint_Source) ProtoMessage()

func (*Checkpoint_Source) ProtoSize Uses

func (m *Checkpoint_Source) ProtoSize() (n int)

func (*Checkpoint_Source) Reset Uses

func (m *Checkpoint_Source) Reset()

func (*Checkpoint_Source) String Uses

func (m *Checkpoint_Source) String() string

func (*Checkpoint_Source) Unmarshal Uses

func (m *Checkpoint_Source) Unmarshal(dAtA []byte) error

func (*Checkpoint_Source) XXX_DiscardUnknown Uses

func (m *Checkpoint_Source) XXX_DiscardUnknown()

func (*Checkpoint_Source) XXX_Marshal Uses

func (m *Checkpoint_Source) XXX_Marshal(b []byte, deterministic bool) ([]byte, error)

func (*Checkpoint_Source) XXX_Merge Uses

func (m *Checkpoint_Source) XXX_Merge(src proto.Message)

func (*Checkpoint_Source) XXX_Size Uses

func (m *Checkpoint_Source) XXX_Size() int

func (*Checkpoint_Source) XXX_Unmarshal Uses

func (m *Checkpoint_Source) XXX_Unmarshal(b []byte) error

type ConsumerSpec Uses

type ConsumerSpec struct {
    // ProcessSpec of the consumer.
    protocol.ProcessSpec `protobuf:"bytes,1,opt,name=process_spec,json=processSpec,proto3,embedded=process_spec" json:"process_spec" yaml:",inline"`
    // Maximum number of assigned Shards.
    ShardLimit uint32 `protobuf:"varint,2,opt,name=shard_limit,json=shardLimit,proto3" json:"shard_limit,omitempty"`
}

ConsumerSpec describes a Consumer process instance and its configuration. It serves as a allocator MemberValue.

func (*ConsumerSpec) Descriptor Uses

func (*ConsumerSpec) Descriptor() ([]byte, []int)

func (*ConsumerSpec) ItemLimit Uses

func (m *ConsumerSpec) ItemLimit() int

ItemLimit is the maximum number of shards this consumer may process. allocator.MemberValue implementation.

func (*ConsumerSpec) Marshal Uses

func (m *ConsumerSpec) Marshal() (dAtA []byte, err error)

func (*ConsumerSpec) MarshalString Uses

func (m *ConsumerSpec) MarshalString() string

MarshalString returns the marshaled encoding of the ConsumerSpec as a string.

func (*ConsumerSpec) MarshalTo Uses

func (m *ConsumerSpec) MarshalTo(dAtA []byte) (int, error)

func (*ConsumerSpec) MarshalToSizedBuffer Uses

func (m *ConsumerSpec) MarshalToSizedBuffer(dAtA []byte) (int, error)

func (*ConsumerSpec) ProtoMessage Uses

func (*ConsumerSpec) ProtoMessage()

func (*ConsumerSpec) ProtoSize Uses

func (m *ConsumerSpec) ProtoSize() (n int)

func (*ConsumerSpec) Reset Uses

func (m *ConsumerSpec) Reset()

func (*ConsumerSpec) String Uses

func (m *ConsumerSpec) String() string

func (*ConsumerSpec) Unmarshal Uses

func (m *ConsumerSpec) Unmarshal(dAtA []byte) error

func (*ConsumerSpec) Validate Uses

func (m *ConsumerSpec) Validate() error

Validate returns an error if the ConsumerSpec is not well-formed.

func (*ConsumerSpec) XXX_DiscardUnknown Uses

func (m *ConsumerSpec) XXX_DiscardUnknown()

func (*ConsumerSpec) XXX_Marshal Uses

func (m *ConsumerSpec) XXX_Marshal(b []byte, deterministic bool) ([]byte, error)

func (*ConsumerSpec) XXX_Merge Uses

func (m *ConsumerSpec) XXX_Merge(src proto.Message)

func (*ConsumerSpec) XXX_Size Uses

func (m *ConsumerSpec) XXX_Size() int

func (*ConsumerSpec) XXX_Unmarshal Uses

func (m *ConsumerSpec) XXX_Unmarshal(b []byte) error

func (*ConsumerSpec) ZeroLimit Uses

func (m *ConsumerSpec) ZeroLimit()

ZeroLimit zeros the ConsumerSpec ShardLimit.

type GetHintsRequest Uses

type GetHintsRequest struct {
    // Shard to fetch hints for.
    Shard ShardID `protobuf:"bytes,1,opt,name=shard,proto3,casttype=ShardID" json:"shard,omitempty"`
}

func (*GetHintsRequest) Descriptor Uses

func (*GetHintsRequest) Descriptor() ([]byte, []int)

func (*GetHintsRequest) Marshal Uses

func (m *GetHintsRequest) Marshal() (dAtA []byte, err error)

func (*GetHintsRequest) MarshalTo Uses

func (m *GetHintsRequest) MarshalTo(dAtA []byte) (int, error)

func (*GetHintsRequest) MarshalToSizedBuffer Uses

func (m *GetHintsRequest) MarshalToSizedBuffer(dAtA []byte) (int, error)

func (*GetHintsRequest) ProtoMessage Uses

func (*GetHintsRequest) ProtoMessage()

func (*GetHintsRequest) ProtoSize Uses

func (m *GetHintsRequest) ProtoSize() (n int)

func (*GetHintsRequest) Reset Uses

func (m *GetHintsRequest) Reset()

func (*GetHintsRequest) String Uses

func (m *GetHintsRequest) String() string

func (*GetHintsRequest) Unmarshal Uses

func (m *GetHintsRequest) Unmarshal(dAtA []byte) error

func (*GetHintsRequest) Validate Uses

func (m *GetHintsRequest) Validate() error

Validate returns an error if the HintsRequest is not well-formed.

func (*GetHintsRequest) XXX_DiscardUnknown Uses

func (m *GetHintsRequest) XXX_DiscardUnknown()

func (*GetHintsRequest) XXX_Marshal Uses

func (m *GetHintsRequest) XXX_Marshal(b []byte, deterministic bool) ([]byte, error)

func (*GetHintsRequest) XXX_Merge Uses

func (m *GetHintsRequest) XXX_Merge(src proto.Message)

func (*GetHintsRequest) XXX_Size Uses

func (m *GetHintsRequest) XXX_Size() int

func (*GetHintsRequest) XXX_Unmarshal Uses

func (m *GetHintsRequest) XXX_Unmarshal(b []byte) error

type GetHintsResponse Uses

type GetHintsResponse struct {
    // Status of the Hints RPC.
    Status Status `protobuf:"varint,1,opt,name=status,proto3,enum=consumer.Status" json:"status,omitempty"`
    // Header of the response.
    Header protocol.Header `protobuf:"bytes,2,opt,name=header,proto3" json:"header"`
    // Primary hints for the shard.
    PrimaryHints GetHintsResponse_ResponseHints `protobuf:"bytes,3,opt,name=primary_hints,json=primaryHints,proto3" json:"primary_hints"`
    // List of backup hints for a shard. The most recent recovery log hints will
    // be first, any subsequent hints are for historical backup. If there is no
    // value for a hint key the value corresponding hints will be nil.
    BackupHints []GetHintsResponse_ResponseHints `protobuf:"bytes,4,rep,name=backup_hints,json=backupHints,proto3" json:"backup_hints"`
}

func (*GetHintsResponse) Descriptor Uses

func (*GetHintsResponse) Descriptor() ([]byte, []int)

func (*GetHintsResponse) Marshal Uses

func (m *GetHintsResponse) Marshal() (dAtA []byte, err error)

func (*GetHintsResponse) MarshalTo Uses

func (m *GetHintsResponse) MarshalTo(dAtA []byte) (int, error)

func (*GetHintsResponse) MarshalToSizedBuffer Uses

func (m *GetHintsResponse) MarshalToSizedBuffer(dAtA []byte) (int, error)

func (*GetHintsResponse) ProtoMessage Uses

func (*GetHintsResponse) ProtoMessage()

func (*GetHintsResponse) ProtoSize Uses

func (m *GetHintsResponse) ProtoSize() (n int)

func (*GetHintsResponse) Reset Uses

func (m *GetHintsResponse) Reset()

func (*GetHintsResponse) String Uses

func (m *GetHintsResponse) String() string

func (*GetHintsResponse) Unmarshal Uses

func (m *GetHintsResponse) Unmarshal(dAtA []byte) error

func (*GetHintsResponse) Validate Uses

func (m *GetHintsResponse) Validate() error

Validate returns an error if the HintsResponse is not well-formed.

func (*GetHintsResponse) XXX_DiscardUnknown Uses

func (m *GetHintsResponse) XXX_DiscardUnknown()

func (*GetHintsResponse) XXX_Marshal Uses

func (m *GetHintsResponse) XXX_Marshal(b []byte, deterministic bool) ([]byte, error)

func (*GetHintsResponse) XXX_Merge Uses

func (m *GetHintsResponse) XXX_Merge(src proto.Message)

func (*GetHintsResponse) XXX_Size Uses

func (m *GetHintsResponse) XXX_Size() int

func (*GetHintsResponse) XXX_Unmarshal Uses

func (m *GetHintsResponse) XXX_Unmarshal(b []byte) error

type GetHintsResponse_ResponseHints Uses

type GetHintsResponse_ResponseHints struct {
    // If the hints value does not exist Hints will be nil.
    Hints *recoverylog.FSMHints `protobuf:"bytes,1,opt,name=hints,proto3" json:"hints,omitempty"`
}

func (*GetHintsResponse_ResponseHints) Descriptor Uses

func (*GetHintsResponse_ResponseHints) Descriptor() ([]byte, []int)

func (*GetHintsResponse_ResponseHints) Marshal Uses

func (m *GetHintsResponse_ResponseHints) Marshal() (dAtA []byte, err error)

func (*GetHintsResponse_ResponseHints) MarshalTo Uses

func (m *GetHintsResponse_ResponseHints) MarshalTo(dAtA []byte) (int, error)

func (*GetHintsResponse_ResponseHints) MarshalToSizedBuffer Uses

func (m *GetHintsResponse_ResponseHints) MarshalToSizedBuffer(dAtA []byte) (int, error)

func (*GetHintsResponse_ResponseHints) ProtoMessage Uses

func (*GetHintsResponse_ResponseHints) ProtoMessage()

func (*GetHintsResponse_ResponseHints) ProtoSize Uses

func (m *GetHintsResponse_ResponseHints) ProtoSize() (n int)

func (*GetHintsResponse_ResponseHints) Reset Uses

func (m *GetHintsResponse_ResponseHints) Reset()

func (*GetHintsResponse_ResponseHints) String Uses

func (m *GetHintsResponse_ResponseHints) String() string

func (*GetHintsResponse_ResponseHints) Unmarshal Uses

func (m *GetHintsResponse_ResponseHints) Unmarshal(dAtA []byte) error

func (GetHintsResponse_ResponseHints) Validate Uses

func (m GetHintsResponse_ResponseHints) Validate() error

Validate returns an error if the GetHintsResponse_ResponseHints is not well-formed.

func (*GetHintsResponse_ResponseHints) XXX_DiscardUnknown Uses

func (m *GetHintsResponse_ResponseHints) XXX_DiscardUnknown()

func (*GetHintsResponse_ResponseHints) XXX_Marshal Uses

func (m *GetHintsResponse_ResponseHints) XXX_Marshal(b []byte, deterministic bool) ([]byte, error)

func (*GetHintsResponse_ResponseHints) XXX_Merge Uses

func (m *GetHintsResponse_ResponseHints) XXX_Merge(src proto.Message)

func (*GetHintsResponse_ResponseHints) XXX_Size Uses

func (m *GetHintsResponse_ResponseHints) XXX_Size() int

func (*GetHintsResponse_ResponseHints) XXX_Unmarshal Uses

func (m *GetHintsResponse_ResponseHints) XXX_Unmarshal(b []byte) error

type ListRequest Uses

type ListRequest struct {
    // Selector optionally refines the set of shards which will be enumerated.
    // If zero-valued, all shards are returned. Otherwise, only ShardSpecs
    // matching the LabelSelector will be returned. One meta-label "id" is
    // additionally supported by the selector, where "id=example-shard-ID"
    // will match a ShardSpec with ID "example-shard-ID".
    Selector protocol.LabelSelector `protobuf:"bytes,1,opt,name=selector,proto3" json:"selector"`
}

func (*ListRequest) Descriptor Uses

func (*ListRequest) Descriptor() ([]byte, []int)

func (*ListRequest) Marshal Uses

func (m *ListRequest) Marshal() (dAtA []byte, err error)

func (*ListRequest) MarshalTo Uses

func (m *ListRequest) MarshalTo(dAtA []byte) (int, error)

func (*ListRequest) MarshalToSizedBuffer Uses

func (m *ListRequest) MarshalToSizedBuffer(dAtA []byte) (int, error)

func (*ListRequest) ProtoMessage Uses

func (*ListRequest) ProtoMessage()

func (*ListRequest) ProtoSize Uses

func (m *ListRequest) ProtoSize() (n int)

func (*ListRequest) Reset Uses

func (m *ListRequest) Reset()

func (*ListRequest) String Uses

func (m *ListRequest) String() string

func (*ListRequest) Unmarshal Uses

func (m *ListRequest) Unmarshal(dAtA []byte) error

func (*ListRequest) Validate Uses

func (m *ListRequest) Validate() error

Validate returns an error if the ListRequest is not well-formed.

func (*ListRequest) XXX_DiscardUnknown Uses

func (m *ListRequest) XXX_DiscardUnknown()

func (*ListRequest) XXX_Marshal Uses

func (m *ListRequest) XXX_Marshal(b []byte, deterministic bool) ([]byte, error)

func (*ListRequest) XXX_Merge Uses

func (m *ListRequest) XXX_Merge(src proto.Message)

func (*ListRequest) XXX_Size Uses

func (m *ListRequest) XXX_Size() int

func (*ListRequest) XXX_Unmarshal Uses

func (m *ListRequest) XXX_Unmarshal(b []byte) error

type ListResponse Uses

type ListResponse struct {
    // Status of the List RPC.
    Status Status `protobuf:"varint,1,opt,name=status,proto3,enum=consumer.Status" json:"status,omitempty"`
    // Header of the response.
    Header protocol.Header      `protobuf:"bytes,2,opt,name=header,proto3" json:"header"`
    Shards []ListResponse_Shard `protobuf:"bytes,3,rep,name=shards,proto3" json:"shards"`
}

func (*ListResponse) Descriptor Uses

func (*ListResponse) Descriptor() ([]byte, []int)

func (*ListResponse) Marshal Uses

func (m *ListResponse) Marshal() (dAtA []byte, err error)

func (*ListResponse) MarshalTo Uses

func (m *ListResponse) MarshalTo(dAtA []byte) (int, error)

func (*ListResponse) MarshalToSizedBuffer Uses

func (m *ListResponse) MarshalToSizedBuffer(dAtA []byte) (int, error)

func (*ListResponse) ProtoMessage Uses

func (*ListResponse) ProtoMessage()

func (*ListResponse) ProtoSize Uses

func (m *ListResponse) ProtoSize() (n int)

func (*ListResponse) Reset Uses

func (m *ListResponse) Reset()

func (*ListResponse) String Uses

func (m *ListResponse) String() string

func (*ListResponse) Unmarshal Uses

func (m *ListResponse) Unmarshal(dAtA []byte) error

func (*ListResponse) Validate Uses

func (m *ListResponse) Validate() error

Validate returns an error if the ListResponse is not well-formed.

func (*ListResponse) XXX_DiscardUnknown Uses

func (m *ListResponse) XXX_DiscardUnknown()

func (*ListResponse) XXX_Marshal Uses

func (m *ListResponse) XXX_Marshal(b []byte, deterministic bool) ([]byte, error)

func (*ListResponse) XXX_Merge Uses

func (m *ListResponse) XXX_Merge(src proto.Message)

func (*ListResponse) XXX_Size Uses

func (m *ListResponse) XXX_Size() int

func (*ListResponse) XXX_Unmarshal Uses

func (m *ListResponse) XXX_Unmarshal(b []byte) error

type ListResponse_Shard Uses

type ListResponse_Shard struct {
    Spec ShardSpec `protobuf:"bytes,1,opt,name=spec,proto3" json:"spec"`
    // Current ModRevision of the ShardSpec.
    ModRevision int64 `protobuf:"varint,2,opt,name=mod_revision,json=modRevision,proto3" json:"mod_revision,omitempty"`
    // Route of the shard, including endpoints.
    Route protocol.Route `protobuf:"bytes,3,opt,name=route,proto3" json:"route"`
    // Status of each replica. Cardinality and ordering matches |route|.
    Status []ReplicaStatus `protobuf:"bytes,4,rep,name=status,proto3" json:"status"`
}

Shards of the response.

func (*ListResponse_Shard) Descriptor Uses

func (*ListResponse_Shard) Descriptor() ([]byte, []int)

func (*ListResponse_Shard) Marshal Uses

func (m *ListResponse_Shard) Marshal() (dAtA []byte, err error)

func (*ListResponse_Shard) MarshalTo Uses

func (m *ListResponse_Shard) MarshalTo(dAtA []byte) (int, error)

func (*ListResponse_Shard) MarshalToSizedBuffer Uses

func (m *ListResponse_Shard) MarshalToSizedBuffer(dAtA []byte) (int, error)

func (*ListResponse_Shard) ProtoMessage Uses

func (*ListResponse_Shard) ProtoMessage()

func (*ListResponse_Shard) ProtoSize Uses

func (m *ListResponse_Shard) ProtoSize() (n int)

func (*ListResponse_Shard) Reset Uses

func (m *ListResponse_Shard) Reset()

func (*ListResponse_Shard) String Uses

func (m *ListResponse_Shard) String() string

func (*ListResponse_Shard) Unmarshal Uses

func (m *ListResponse_Shard) Unmarshal(dAtA []byte) error

func (*ListResponse_Shard) Validate Uses

func (m *ListResponse_Shard) Validate() error

Validate returns an error if the ListResponse_Shard is not well-formed.

func (*ListResponse_Shard) XXX_DiscardUnknown Uses

func (m *ListResponse_Shard) XXX_DiscardUnknown()

func (*ListResponse_Shard) XXX_Marshal Uses

func (m *ListResponse_Shard) XXX_Marshal(b []byte, deterministic bool) ([]byte, error)

func (*ListResponse_Shard) XXX_Merge Uses

func (m *ListResponse_Shard) XXX_Merge(src proto.Message)

func (*ListResponse_Shard) XXX_Size Uses

func (m *ListResponse_Shard) XXX_Size() int

func (*ListResponse_Shard) XXX_Unmarshal Uses

func (m *ListResponse_Shard) XXX_Unmarshal(b []byte) error

type ReplicaStatus Uses

type ReplicaStatus struct {
    Code ReplicaStatus_Code `protobuf:"varint,1,opt,name=code,proto3,enum=consumer.ReplicaStatus_Code" json:"code,omitempty"`
    // Errors encountered during replica processing. Set iff |code| is FAILED.
    Errors []string `protobuf:"bytes,2,rep,name=errors,proto3" json:"errors,omitempty"`
}

ReplicaStatus is the status of a ShardSpec assigned to a ConsumerSpec. It serves as an allocator AssignmentValue. ReplicaStatus is reduced by taking the maximum enum value among statuses. Eg, if a primary is PRIMARY, one replica is BACKFILL and the other STANDBY, then the status is PRIMARY. If one of the replicas transitioned to FAILED, than the status is FAILED. This reduction behavior is used to summarize status across all replicas.

func (*ReplicaStatus) Descriptor Uses

func (*ReplicaStatus) Descriptor() ([]byte, []int)

func (*ReplicaStatus) Marshal Uses

func (m *ReplicaStatus) Marshal() (dAtA []byte, err error)

func (*ReplicaStatus) MarshalString Uses

func (m *ReplicaStatus) MarshalString() string

MarshalString returns the marshaled encoding of the ReplicaStatus as a string.

func (*ReplicaStatus) MarshalTo Uses

func (m *ReplicaStatus) MarshalTo(dAtA []byte) (int, error)

func (*ReplicaStatus) MarshalToSizedBuffer Uses

func (m *ReplicaStatus) MarshalToSizedBuffer(dAtA []byte) (int, error)

func (*ReplicaStatus) ProtoMessage Uses

func (*ReplicaStatus) ProtoMessage()

func (*ReplicaStatus) ProtoSize Uses

func (m *ReplicaStatus) ProtoSize() (n int)

func (*ReplicaStatus) Reduce Uses

func (m *ReplicaStatus) Reduce(other *ReplicaStatus)

Reduce folds another ReplicaStatus into this one.

func (*ReplicaStatus) Reset Uses

func (m *ReplicaStatus) Reset()

func (*ReplicaStatus) String Uses

func (m *ReplicaStatus) String() string

func (*ReplicaStatus) Unmarshal Uses

func (m *ReplicaStatus) Unmarshal(dAtA []byte) error

func (*ReplicaStatus) Validate Uses

func (m *ReplicaStatus) Validate() error

Validate returns an error if the ReplicaStatus is not well-formed.

func (*ReplicaStatus) XXX_DiscardUnknown Uses

func (m *ReplicaStatus) XXX_DiscardUnknown()

func (*ReplicaStatus) XXX_Marshal Uses

func (m *ReplicaStatus) XXX_Marshal(b []byte, deterministic bool) ([]byte, error)

func (*ReplicaStatus) XXX_Merge Uses

func (m *ReplicaStatus) XXX_Merge(src proto.Message)

func (*ReplicaStatus) XXX_Size Uses

func (m *ReplicaStatus) XXX_Size() int

func (*ReplicaStatus) XXX_Unmarshal Uses

func (m *ReplicaStatus) XXX_Unmarshal(b []byte) error

type ReplicaStatus_Code Uses

type ReplicaStatus_Code int32
const (
    ReplicaStatus_IDLE ReplicaStatus_Code = 0
    // The replica is actively playing the historical recovery log.
    ReplicaStatus_BACKFILL ReplicaStatus_Code = 100
    // The replica has finished playing the historical recovery log and is
    // live-tailing it to locally mirror recorded operations as they are
    // produced. It can take over as primary at any time.
    //
    // Shards not having recovery logs immediately transition to STANDBY.
    ReplicaStatus_STANDBY ReplicaStatus_Code = 200
    // The replica is actively serving as primary.
    ReplicaStatus_PRIMARY ReplicaStatus_Code = 300
    // The replica has encountered an unrecoverable error.
    ReplicaStatus_FAILED ReplicaStatus_Code = 400
)

func (ReplicaStatus_Code) EnumDescriptor Uses

func (ReplicaStatus_Code) EnumDescriptor() ([]byte, []int)

func (ReplicaStatus_Code) String Uses

func (x ReplicaStatus_Code) String() string

func (ReplicaStatus_Code) Validate Uses

func (x ReplicaStatus_Code) Validate() error

Validate returns an error if the ReplicaStatus_Code is not well-formed.

type RoutedShardClient Uses

type RoutedShardClient interface {
    ShardClient
    pb.DispatchRouter
}

RoutedShardClient composes a ShardClient and DispatchRouter.

func NewRoutedShardClient Uses

func NewRoutedShardClient(sc ShardClient, dr pb.DispatchRouter) RoutedShardClient

NewRoutedShardClient composes a ShardClient and DispatchRouter.

type ShardClient Uses

type ShardClient interface {
    // Stat returns detailed status of a given Shard.
    Stat(ctx context.Context, in *StatRequest, opts ...grpc.CallOption) (*StatResponse, error)
    // List Shards, their ShardSpecs and their processing status.
    List(ctx context.Context, in *ListRequest, opts ...grpc.CallOption) (*ListResponse, error)
    // Apply changes to the collection of Shards managed by the consumer.
    Apply(ctx context.Context, in *ApplyRequest, opts ...grpc.CallOption) (*ApplyResponse, error)
    // GetHints fetches hints for a shard.
    GetHints(ctx context.Context, in *GetHintsRequest, opts ...grpc.CallOption) (*GetHintsResponse, error)
}

ShardClient is the client API for Shard service.

For semantics around ctx use and closing/ending streaming RPCs, please refer to https://godoc.org/google.golang.org/grpc#ClientConn.NewStream.

func NewShardClient Uses

func NewShardClient(cc *grpc.ClientConn) ShardClient

type ShardID Uses

type ShardID string

ShardID uniquely identifies a shard processed by a Gazette consumer.

func (ShardID) String Uses

func (id ShardID) String() string

String returns the Shard as a string.

func (ShardID) Validate Uses

func (id ShardID) Validate() error

Validate returns an error if the Shard is not well-formed.

type ShardServer Uses

type ShardServer interface {
    // Stat returns detailed status of a given Shard.
    Stat(context.Context, *StatRequest) (*StatResponse, error)
    // List Shards, their ShardSpecs and their processing status.
    List(context.Context, *ListRequest) (*ListResponse, error)
    // Apply changes to the collection of Shards managed by the consumer.
    Apply(context.Context, *ApplyRequest) (*ApplyResponse, error)
    // GetHints fetches hints for a shard.
    GetHints(context.Context, *GetHintsRequest) (*GetHintsResponse, error)
}

ShardServer is the server API for Shard service.

type ShardSpec Uses

type ShardSpec struct {
    // ID of the shard.
    Id  ShardID `protobuf:"bytes,1,opt,name=id,proto3,casttype=ShardID" json:"id,omitempty" yaml:",omitempty"`
    // Sources of the shard, uniquely ordered on Source journal.
    Sources []ShardSpec_Source `protobuf:"bytes,2,rep,name=sources,proto3" json:"sources" yaml:",omitempty"`
    // Prefix of the Journal into which the shard's recovery log will be recorded.
    // The complete Journal name is built as "{recovery_log_prefix}/{shard_id}".
    // If empty, the shard does not use a recovery log.
    RecoveryLogPrefix string `protobuf:"bytes,3,opt,name=recovery_log_prefix,json=recoveryLogPrefix,proto3" json:"recovery_log_prefix,omitempty" yaml:"recovery_log_prefix,omitempty"`
    // Prefix of Etcd keys into which recovery log FSMHints are written to and
    // read from. FSMHints allow readers of the recovery log to efficiently
    // determine the minimum fragments of log which must be read to fully recover
    // local store state. The complete hint key written by the shard primary is:
    //
    //   "{hint_prefix}/{shard_id}.primary"
    //
    // The primary will regularly produce updated hints into this key, and
    // players of the log will similarly utilize hints from this key.
    // If |recovery_log_prefix| is set, |hint_prefix| must be also.
    HintPrefix string `protobuf:"bytes,4,opt,name=hint_prefix,json=hintPrefix,proto3" json:"hint_prefix,omitempty" yaml:"hint_prefix,omitempty"`
    // Backups of verified recovery log FSMHints, retained as a disaster-recovery
    // mechanism. On completing playback, a player will write recovered hints to:
    //
    //   "{hints_prefix}/{shard_id}.backup.0".
    //
    // It also move hints previously stored under
    // "{hints_prefix/{shard_id}.backup.0" to
    // "{hints_prefix/{shard_id}.backup.1", and so on, keeping at most
    // |hint_backups| distinct sets of FSMHints.
    //
    // In the case of disaster or data-loss, these copied hints can be an
    // important fallback for recovering a consistent albeit older version of the
    // shard's store, with each relying on only progressively older portions of
    // the recovery log.
    //
    // When pruning the recovery log, log fragments which are older than (and no
    // longer required by) the *oldest* backup are discarded, ensuring that
    // all hints remain valid for playback.
    HintBackups int32 `protobuf:"varint,5,opt,name=hint_backups,json=hintBackups,proto3" json:"hint_backups,omitempty" yaml:"hint_backups,omitempty"`
    // Max duration of shard transactions. This duration upper-bounds the amount
    // of time during which a transaction may process messages before it must
    // flush and commit. It may run for less time if an input message stall occurs
    // (eg, no decoded journal message is ready without blocking). A typical value
    // would be `1s`: applications which perform extensive aggregation over
    // message streams exhibiting locality of "hot" keys may benefit from larger
    // values.
    MaxTxnDuration time.Duration `protobuf:"bytes,6,opt,name=max_txn_duration,json=maxTxnDuration,proto3,stdduration" json:"max_txn_duration" yaml:"max_txn_duration,omitempty"`
    // Min duration of shard transactions. This duration lower-bounds the amount
    // of time during which a transaction must process messages before it may
    // flush and commit. It may run for more time if additional messages are
    // available (eg, decoded journal messages are ready without blocking). Note
    // also that transactions are pipelined: a current transaction may process
    // messages while a prior transaction's recovery log writes flush to Gazette,
    // but it cannot prepare to commit until the prior transaction writes
    // complete. In other words even if |min_txn_quantum| is zero, some degree of
    // message batching is expected due to the network delay inherent in Gazette
    // writes. A typical value of would be `0s`: applications which perform
    // extensive aggregation may benefit from larger values.
    MinTxnDuration time.Duration `protobuf:"bytes,7,opt,name=min_txn_duration,json=minTxnDuration,proto3,stdduration" json:"min_txn_duration" yaml:"min_txn_duration,omitempty"`
    // Disable processing of the shard.
    Disable bool `protobuf:"varint,8,opt,name=disable,proto3" json:"disable,omitempty" yaml:",omitempty"`
    // Hot standbys is the desired number of consumer processes which should be
    // replicating the primary consumer's recovery log. Standbys are allocated in
    // a separate availability zone of the current primary, and tail the live log
    // to continuously mirror the primary's on-disk DB file structure. Should the
    // primary experience failure, one of the hot standbys will be assigned to
    // take over as the new shard primary, which is accomplished by simply opening
    // its local copy of the recovered store files.
    //
    // Note that under regular operation, shard hand-off is zero downtime even if
    // standbys are zero, as the current primary will not cede ownership until the
    // replacement process declares itself ready. However, without standbys a
    // process failure will leave the shard without an active primary while its
    // replacement starts and completes playback of its recovery log.
    HotStandbys uint32 `protobuf:"varint,9,opt,name=hot_standbys,json=hotStandbys,proto3" json:"hot_standbys,omitempty" yaml:"hot_standbys,omitempty"`
    // User-defined Labels of this ShardSpec. The label "id" is reserved and may
    // not be used with a ShardSpec's labels.
    protocol.LabelSet `protobuf:"bytes,10,opt,name=labels,proto3,embedded=labels" json:"labels" yaml:",omitempty,inline"`
    // Disable waiting for acknowledgements of pending message(s).
    //
    // If a consumer transaction reads uncommitted messages, it will by default
    // remain open (subject to the max duration) awaiting an acknowledgement of
    // those messages, in the hope that that acknowledgement will be quickly
    // forthcoming and, by remaining open, we can process all messages in this
    // transaction. Effectively we're trading a small amount of increased local
    // latency for a global reduction in end-to-end latency.
    //
    // This works well for acyclic message flows, but can introduce unnecessary
    // stalls if there are message cycles between shards. In the simplest case,
    // a transaction could block awaiting an ACK of a message that it itself
    // produced -- an ACK which can't arrive until the transaction closes.
    DisableWaitForAck bool `protobuf:"varint,11,opt,name=disable_wait_for_ack,json=disableWaitForAck,proto3" json:"disable_wait_for_ack,omitempty" yaml:"disable_wait_for_ack,omitempty"`
}

ShardSpec describes a shard and its configuration, and is the long-lived unit of work and scaling for a consumer application. Each shard is allocated to a one "primary" at-a-time selected from the current processes of a consumer application, and is re-assigned on process fault or exit.

ShardSpecs describe all configuration of the shard and its processing, including journals to consume, configuration for processing transactions, its recovery log, hot standbys, etc. ShardSpecs may be further extended with domain-specific labels & values to further define application behavior. ShardSpec is-a allocator.ItemValue.

func IntersectShardSpecs Uses

func IntersectShardSpecs(a, b ShardSpec) ShardSpec

IntersectShardSpecs returns a ShardSpec having a non-zero-valued field for each field value which is shared between |a| and |b|.

func SubtractShardSpecs Uses

func SubtractShardSpecs(a, b ShardSpec) ShardSpec

SubtractShardSpecs returns a ShardSpec derived from |a| but having a zero-valued field for each field which is matched by |b|.

func UnionShardSpecs Uses

func UnionShardSpecs(a, b ShardSpec) ShardSpec

UnionShardSpecs returns a ShardSpec combining all non-zero-valued fields across |a| and |b|. Where both |a| and |b| provide a non-zero value for a field, the value of |a| is retained.

func (*ShardSpec) Descriptor Uses

func (*ShardSpec) Descriptor() ([]byte, []int)

func (*ShardSpec) DesiredReplication Uses

func (m *ShardSpec) DesiredReplication() int

DesiredReplication is the desired number of shard replicas. allocator.ItemValue implementation.

func (*ShardSpec) HintBackupKeys Uses

func (m *ShardSpec) HintBackupKeys() []string

HintBackupKeys returns Etcd keys to which verified, disaster-recovery hints are written.

func (*ShardSpec) HintPrimaryKey Uses

func (m *ShardSpec) HintPrimaryKey() string

HintPrimaryKey returns the Etcd key to which recorded, primary hints are written.

func (*ShardSpec) Marshal Uses

func (m *ShardSpec) Marshal() (dAtA []byte, err error)

func (*ShardSpec) MarshalString Uses

func (m *ShardSpec) MarshalString() string

MarshalString returns the marshaled encoding of the ShardSpec as a string.

func (*ShardSpec) MarshalTo Uses

func (m *ShardSpec) MarshalTo(dAtA []byte) (int, error)

func (*ShardSpec) MarshalToSizedBuffer Uses

func (m *ShardSpec) MarshalToSizedBuffer(dAtA []byte) (int, error)

func (*ShardSpec) ProtoMessage Uses

func (*ShardSpec) ProtoMessage()

func (*ShardSpec) ProtoSize Uses

func (m *ShardSpec) ProtoSize() (n int)

func (*ShardSpec) RecoveryLog Uses

func (m *ShardSpec) RecoveryLog() pb.Journal

RecoveryLog returns the Journal to which the Shard's recovery log is recorded. IF the Shard has no recovery log, "" is returned..

func (*ShardSpec) Reset Uses

func (m *ShardSpec) Reset()

func (*ShardSpec) String Uses

func (m *ShardSpec) String() string

func (*ShardSpec) Unmarshal Uses

func (m *ShardSpec) Unmarshal(dAtA []byte) error

func (*ShardSpec) Validate Uses

func (m *ShardSpec) Validate() error

Validate returns an error if the ShardSpec is not well-formed.

func (*ShardSpec) XXX_DiscardUnknown Uses

func (m *ShardSpec) XXX_DiscardUnknown()

func (*ShardSpec) XXX_Marshal Uses

func (m *ShardSpec) XXX_Marshal(b []byte, deterministic bool) ([]byte, error)

func (*ShardSpec) XXX_Merge Uses

func (m *ShardSpec) XXX_Merge(src proto.Message)

func (*ShardSpec) XXX_Size Uses

func (m *ShardSpec) XXX_Size() int

func (*ShardSpec) XXX_Unmarshal Uses

func (m *ShardSpec) XXX_Unmarshal(b []byte) error

type ShardSpec_Source Uses

type ShardSpec_Source struct {
    // Journal which this shard is consuming.
    Journal go_gazette_dev_core_broker_protocol.Journal `protobuf:"bytes,1,opt,name=journal,proto3,casttype=go.gazette.dev/core/broker/protocol.Journal" json:"journal,omitempty"`
    // Minimum journal byte offset the shard should begin reading from.
    // Typically this should be zero, as read offsets are check-pointed and
    // restored from the shard's Store as it processes. |min_offset| can be
    // useful for shard initialization, directing it to skip over historical
    // portions of the journal not needed for the application's use case.
    MinOffset go_gazette_dev_core_broker_protocol.Offset `protobuf:"varint,3,opt,name=min_offset,json=minOffset,proto3,casttype=go.gazette.dev/core/broker/protocol.Offset" json:"min_offset,omitempty" yaml:"min_offset,omitempty"`
}

Sources define the set of journals which this shard consumes. At least one Source must be specified, and in many use cases only one will be needed. For use cases which can benefit, multiple sources may be specified to represent a "join" over messages of distinct journals.

Note the effective mapping of messages to each of the joined journals should align (eg, joining a journal of customer updates with one of orders, where both are mapped on customer ID). This typically means the partitioning of the two event "topics" must be the same.

Another powerful pattern is to shard on partitions of a high-volume event stream, and also have each shard join against all events of a low-volume stream. For example, a shard might ingest and index "viewed product" events, read a comparably low-volume "purchase" event stream, and on each purchase publish the bundle of its corresponding prior product views.

func (*ShardSpec_Source) Descriptor Uses

func (*ShardSpec_Source) Descriptor() ([]byte, []int)

func (*ShardSpec_Source) Marshal Uses

func (m *ShardSpec_Source) Marshal() (dAtA []byte, err error)

func (*ShardSpec_Source) MarshalTo Uses

func (m *ShardSpec_Source) MarshalTo(dAtA []byte) (int, error)

func (*ShardSpec_Source) MarshalToSizedBuffer Uses

func (m *ShardSpec_Source) MarshalToSizedBuffer(dAtA []byte) (int, error)

func (*ShardSpec_Source) ProtoMessage Uses

func (*ShardSpec_Source) ProtoMessage()

func (*ShardSpec_Source) ProtoSize Uses

func (m *ShardSpec_Source) ProtoSize() (n int)

func (*ShardSpec_Source) Reset Uses

func (m *ShardSpec_Source) Reset()

func (*ShardSpec_Source) String Uses

func (m *ShardSpec_Source) String() string

func (*ShardSpec_Source) Unmarshal Uses

func (m *ShardSpec_Source) Unmarshal(dAtA []byte) error

func (*ShardSpec_Source) Validate Uses

func (m *ShardSpec_Source) Validate() error

Validate returns an error if the ShardSpec_Source is not well-formed.

func (*ShardSpec_Source) XXX_DiscardUnknown Uses

func (m *ShardSpec_Source) XXX_DiscardUnknown()

func (*ShardSpec_Source) XXX_Marshal Uses

func (m *ShardSpec_Source) XXX_Marshal(b []byte, deterministic bool) ([]byte, error)

func (*ShardSpec_Source) XXX_Merge Uses

func (m *ShardSpec_Source) XXX_Merge(src proto.Message)

func (*ShardSpec_Source) XXX_Size Uses

func (m *ShardSpec_Source) XXX_Size() int

func (*ShardSpec_Source) XXX_Unmarshal Uses

func (m *ShardSpec_Source) XXX_Unmarshal(b []byte) error

type StatRequest Uses

type StatRequest struct {
    // Header may be attached by a proxying consumer peer.
    Header *protocol.Header `protobuf:"bytes,1,opt,name=header,proto3" json:"header,omitempty"`
    // Shard to Stat.
    Shard ShardID `protobuf:"bytes,2,opt,name=shard,proto3,casttype=ShardID" json:"shard,omitempty"`
    // Journals and offsets which must be reflected in a completed consumer
    // transaction before Stat returns, blocking if required. Offsets of journals
    // not read by this shard are ignored.
    ReadThrough map[go_gazette_dev_core_broker_protocol.Journal]go_gazette_dev_core_broker_protocol.Offset `protobuf:"bytes,3,rep,name=read_through,json=readThrough,proto3,castkey=go.gazette.dev/core/broker/protocol.Journal,castvalue=go.gazette.dev/core/broker/protocol.Offset" json:"read_through,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"varint,2,opt,name=value,proto3"`
}

func (*StatRequest) Descriptor Uses

func (*StatRequest) Descriptor() ([]byte, []int)

func (*StatRequest) Marshal Uses

func (m *StatRequest) Marshal() (dAtA []byte, err error)

func (*StatRequest) MarshalTo Uses

func (m *StatRequest) MarshalTo(dAtA []byte) (int, error)

func (*StatRequest) MarshalToSizedBuffer Uses

func (m *StatRequest) MarshalToSizedBuffer(dAtA []byte) (int, error)

func (*StatRequest) ProtoMessage Uses

func (*StatRequest) ProtoMessage()

func (*StatRequest) ProtoSize Uses

func (m *StatRequest) ProtoSize() (n int)

func (*StatRequest) Reset Uses

func (m *StatRequest) Reset()

func (*StatRequest) String Uses

func (m *StatRequest) String() string

func (*StatRequest) Unmarshal Uses

func (m *StatRequest) Unmarshal(dAtA []byte) error

func (*StatRequest) Validate Uses

func (m *StatRequest) Validate() error

Validate returns an error if the StatRequest is not well-formed.

func (*StatRequest) XXX_DiscardUnknown Uses

func (m *StatRequest) XXX_DiscardUnknown()

func (*StatRequest) XXX_Marshal Uses

func (m *StatRequest) XXX_Marshal(b []byte, deterministic bool) ([]byte, error)

func (*StatRequest) XXX_Merge Uses

func (m *StatRequest) XXX_Merge(src proto.Message)

func (*StatRequest) XXX_Size Uses

func (m *StatRequest) XXX_Size() int

func (*StatRequest) XXX_Unmarshal Uses

func (m *StatRequest) XXX_Unmarshal(b []byte) error

type StatResponse Uses

type StatResponse struct {
    // Status of the Stat RPC.
    Status Status `protobuf:"varint,1,opt,name=status,proto3,enum=consumer.Status" json:"status,omitempty"`
    // Header of the response.
    Header protocol.Header `protobuf:"bytes,2,opt,name=header,proto3" json:"header"`
    // Journals and offsets read through by the most recent completed consumer
    // transaction.
    ReadThrough map[go_gazette_dev_core_broker_protocol.Journal]go_gazette_dev_core_broker_protocol.Offset `protobuf:"bytes,3,rep,name=read_through,json=readThrough,proto3,castkey=go.gazette.dev/core/broker/protocol.Journal,castvalue=go.gazette.dev/core/broker/protocol.Offset" json:"read_through,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"varint,2,opt,name=value,proto3"`
    // Journals and offsets this shard has published through, including
    // acknowledgements, as-of the most recent completed consumer transaction.
    //
    // Formally, if an acknowledged message A results in this shard publishing
    // messages B, and A falls within |read_through|, then all messages B & their
    // acknowledgements fall within |publish_at|.
    //
    // The composition of |read_through| and |publish_at| allow CQRS applications
    // to provide read-your-writes consistency, even if written events pass
    // through multiple intermediate consumers and arbitrary transformations
    // before arriving at the materialized view which is ultimately queried.
    PublishAt map[go_gazette_dev_core_broker_protocol.Journal]go_gazette_dev_core_broker_protocol.Offset `protobuf:"bytes,4,rep,name=publish_at,json=publishAt,proto3,castkey=go.gazette.dev/core/broker/protocol.Journal,castvalue=go.gazette.dev/core/broker/protocol.Offset" json:"publish_at,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"varint,2,opt,name=value,proto3"`
}

func (*StatResponse) Descriptor Uses

func (*StatResponse) Descriptor() ([]byte, []int)

func (*StatResponse) Marshal Uses

func (m *StatResponse) Marshal() (dAtA []byte, err error)

func (*StatResponse) MarshalTo Uses

func (m *StatResponse) MarshalTo(dAtA []byte) (int, error)

func (*StatResponse) MarshalToSizedBuffer Uses

func (m *StatResponse) MarshalToSizedBuffer(dAtA []byte) (int, error)

func (*StatResponse) ProtoMessage Uses

func (*StatResponse) ProtoMessage()

func (*StatResponse) ProtoSize Uses

func (m *StatResponse) ProtoSize() (n int)

func (*StatResponse) Reset Uses

func (m *StatResponse) Reset()

func (*StatResponse) String Uses

func (m *StatResponse) String() string

func (*StatResponse) Unmarshal Uses

func (m *StatResponse) Unmarshal(dAtA []byte) error

func (*StatResponse) Validate Uses

func (m *StatResponse) Validate() error

Validate returns an error if the StatResponse is not well-formed.

func (*StatResponse) XXX_DiscardUnknown Uses

func (m *StatResponse) XXX_DiscardUnknown()

func (*StatResponse) XXX_Marshal Uses

func (m *StatResponse) XXX_Marshal(b []byte, deterministic bool) ([]byte, error)

func (*StatResponse) XXX_Merge Uses

func (m *StatResponse) XXX_Merge(src proto.Message)

func (*StatResponse) XXX_Size Uses

func (m *StatResponse) XXX_Size() int

func (*StatResponse) XXX_Unmarshal Uses

func (m *StatResponse) XXX_Unmarshal(b []byte) error

type Status Uses

type Status int32

Status is a response status code, used across Gazette Consumer RPC APIs.

const (
    Status_OK Status = 0
    // The named shard does not exist.
    Status_SHARD_NOT_FOUND Status = 1
    // There is no current primary consumer process for the shard. This is a
    // temporary condition which should quickly resolve, assuming sufficient
    // consumer capacity.
    Status_NO_SHARD_PRIMARY Status = 2
    // The present consumer process is not the assigned primary for the shard,
    // and was not instructed to proxy the request.
    Status_NOT_SHARD_PRIMARY Status = 3
    // The Etcd transaction failed. Returned by Update RPC when an
    // expect_mod_revision of the UpdateRequest differs from the current
    // ModRevision of the ShardSpec within the store.
    Status_ETCD_TRANSACTION_FAILED Status = 4
)

func (Status) EnumDescriptor Uses

func (Status) EnumDescriptor() ([]byte, []int)

func (Status) String Uses

func (x Status) String() string

func (Status) Validate Uses

func (x Status) Validate() error

Validate returns an error if the Status is not well-formed.

type UnimplementedShardServer Uses

type UnimplementedShardServer struct {
}

UnimplementedShardServer can be embedded to have forward compatible implementations.

func (*UnimplementedShardServer) Apply Uses

func (*UnimplementedShardServer) Apply(ctx context.Context, req *ApplyRequest) (*ApplyResponse, error)

func (*UnimplementedShardServer) GetHints Uses

func (*UnimplementedShardServer) GetHints(ctx context.Context, req *GetHintsRequest) (*GetHintsResponse, error)

func (*UnimplementedShardServer) List Uses

func (*UnimplementedShardServer) List(ctx context.Context, req *ListRequest) (*ListResponse, error)

func (*UnimplementedShardServer) Stat Uses

func (*UnimplementedShardServer) Stat(ctx context.Context, req *StatRequest) (*StatResponse, error)

Package protocol imports 18 packages (graph) and is imported by 31 packages. Updated 2020-06-18. Refresh now. Tools for package owners.