cockroach: github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/kvfeed Index | Files

package kvfeed

import "github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/kvfeed"

Package kvfeed provides an abstraction to stream kvs to a buffer.

The kvfeed coordinated performing logical backfills in the face of schema changes and then running rangefeeds.

Index

Package Files

buffer.go kv_feed.go metrics.go physical_kv_feed.go scanner.go

Variables

var MemBufferDefaultCapacity = envutil.EnvOrDefaultBytes(
    "COCKROACH_CHANGEFEED_BUFFER_CAPACITY", 1<<30) // 1GB

MemBufferDefaultCapacity is the default capacity for a memBuffer for a single changefeed.

TODO(dan): It would be better if all changefeeds shared a single capacity that was given by the operater at startup, like we do for RocksDB and SQL.

func Run Uses

func Run(ctx context.Context, cfg Config) error

Run will run the kvfeed. The feed runs synchronously and returns an error when it finishes.

type Config Uses

type Config struct {
    Settings           *cluster.Settings
    DB                 *kv.DB
    Clock              *hlc.Clock
    Gossip             gossip.OptionalGossip
    Spans              []roachpb.Span
    Targets            jobspb.ChangefeedTargets
    Sink               EventBufferWriter
    LeaseMgr           *lease.Manager
    Metrics            *Metrics
    MM                 *mon.BytesMonitor
    WithDiff           bool
    SchemaChangeEvents changefeedbase.SchemaChangeEventClass
    SchemaChangePolicy changefeedbase.SchemaChangePolicy

    // If true, the feed will begin with a dump of data at exactly the
    // InitialHighWater. This is a peculiar behavior. In general the
    // InitialHighWater is a point in time at which all data is known to have
    // been seen.
    NeedsInitialScan bool

    // InitialHighWater is the timestamp from which new events are guaranteed to
    // be produced.
    InitialHighWater hlc.Timestamp
}

Config configures a kvfeed.

type Event Uses

type Event struct {
    // contains filtered or unexported fields
}

Event represents an event emitted by a kvfeed. It is either a KV or a resolved timestamp.

func (*Event) BackfillTimestamp Uses

func (b *Event) BackfillTimestamp() hlc.Timestamp

BackfillTimestamp overrides the timestamp of the schema that should be used to interpret this KV. If set and prevVal is provided, the previous timestamp will be used to interpret the previous value.

If unset (zero-valued), the KV's timestamp will be used to interpret both of the current and previous values instead.

func (*Event) BufferGetTimestamp Uses

func (b *Event) BufferGetTimestamp() time.Time

BufferGetTimestamp is the time this event came out of the buffer.

func (*Event) KV Uses

func (b *Event) KV() roachpb.KeyValue

KV is populated if this event returns true for IsKV().

func (*Event) PrevValue Uses

func (b *Event) PrevValue() roachpb.Value

PrevValue returns the previous value for this event. PrevValue is non-zero if this is a KV event and the key had a non-tombstone value before the change and the before value of each change was requested (optDiff).

func (*Event) Resolved Uses

func (b *Event) Resolved() *jobspb.ResolvedSpan

Resolved will be non-nil if this is a resolved timestamp event (i.e. IsKV() returns false).

func (*Event) Timestamp Uses

func (b *Event) Timestamp() hlc.Timestamp

Timestamp returns the timestamp of the write if this is a KV event. If there is a non-zero BackfillTimestamp, that is returned. If this is a resolved timestamp event, the timestamp is the resolved timestamp.

func (*Event) Type Uses

func (b *Event) Type() EventType

Type returns the event's EventType.

type EventBuffer Uses

type EventBuffer interface {
    EventBufferReader
    EventBufferWriter
}

EventBuffer is an interface for communicating kvfeed entries between processors.

func MakeChanBuffer Uses

func MakeChanBuffer() EventBuffer

MakeChanBuffer returns an EventBuffer backed by an unbuffered channel.

TODO(ajwerner): Consider adding a buffer here. We know performance of the backfill is terrible. Probably some of that is due to every KV being sent on a channel. This should all get benchmarked and tuned.

type EventBufferReader Uses

type EventBufferReader interface {
    // Get retrieves an entry from the buffer.
    Get(ctx context.Context) (Event, error)
}

EventBufferReader is the read portion of the EventBuffer interface.

type EventBufferWriter Uses

type EventBufferWriter interface {
    AddKV(ctx context.Context, kv roachpb.KeyValue, prevVal roachpb.Value, backfillTimestamp hlc.Timestamp) error
    AddResolved(ctx context.Context, span roachpb.Span, ts hlc.Timestamp, boundaryReached bool) error
    Close(ctx context.Context)
}

EventBufferWriter is the write portion of the EventBuffer interface.

type EventType Uses

type EventType int

EventType indicates the type of the event. Different types indicate which methods will be meaningful. Events are implemented this way rather than as an interface to remove the need to box the events and allow for events to be used in slices directly.

const (
    // KVEvent indicates that the KV, PrevValue, and BackfillTimestamp methods
    // on the Event meaningful.
    KVEvent EventType = iota

    // ResolvedEvent indicates that the Resolved method on the Event will be
    // meaningful.
    ResolvedEvent
)

type Metrics Uses

type Metrics struct {
    BufferEntriesIn      *metric.Counter
    BufferEntriesOut     *metric.Counter
    PollRequestNanosHist *metric.Histogram
}

Metrics is a metric.Struct for kvfeed metrics.

TODO(ajwerner): Make these metrics more reasonable given the removal of the poller and polling in general.

func MakeMetrics Uses

func MakeMetrics(histogramWindow time.Duration) Metrics

MakeMetrics constructs a Metrics struct with the provided histogram window.

func (Metrics) MetricStruct Uses

func (m Metrics) MetricStruct()

MetricStruct makes Metrics a metric.Struct.

Package kvfeed imports 31 packages (graph) and is imported by 3 packages. Updated 2020-08-12. Refresh now. Tools for package owners.