cockroach: Index | Files

package bulk

import ""


Package Files

buffering_adder.go bulk_metrics.go kv_buf.go sst_batcher.go

func AddSSTable Uses

func AddSSTable(
    ctx context.Context,
    db SSTSender,
    start, end roachpb.Key,
    sstBytes []byte,
    disallowShadowing bool,
    ms enginepb.MVCCStats,
    settings *cluster.Settings,
) (int, error)

AddSSTable retries db.AddSSTable if retryable errors occur, including if the SST spans a split, in which case it is iterated and split into two SSTs, one for each side of the split in the error, and each are retried.

type BufferingAdder Uses

type BufferingAdder struct {
    // contains filtered or unexported fields

BufferingAdder is a wrapper for an SSTBatcher that allows out-of-order calls to Add, buffering them up and then sorting them before then passing them in order into an SSTBatcher

func MakeBulkAdder Uses

func MakeBulkAdder(
    ctx context.Context,
    db SSTSender,
    rangeCache *kvcoord.RangeDescriptorCache,
    settings *cluster.Settings,
    timestamp hlc.Timestamp,
    opts kvserverbase.BulkAdderOptions,
    bulkMon *mon.BytesMonitor,
) (*BufferingAdder, error)

MakeBulkAdder makes a kvserverbase.BulkAdder that buffers and sorts K/Vs passed to add into SSTs that are then ingested. rangeCache if set is consulted to avoid generating an SST that will span a range boundary and thus encounter an error and need to be split and retired to be applied.

func (*BufferingAdder) Add Uses

func (b *BufferingAdder) Add(ctx context.Context, key roachpb.Key, value []byte) error

Add adds a key to the buffer and checks if it needs to flush.

func (*BufferingAdder) Close Uses

func (b *BufferingAdder) Close(ctx context.Context)

Close closes the underlying SST builder.

func (*BufferingAdder) CurrentBufferFill Uses

func (b *BufferingAdder) CurrentBufferFill() float32

CurrentBufferFill returns the current buffer fill percentage.

func (*BufferingAdder) Flush Uses

func (b *BufferingAdder) Flush(ctx context.Context) error

Flush flushes any buffered kvs to the batcher.

func (*BufferingAdder) GetSummary Uses

func (b *BufferingAdder) GetSummary() roachpb.BulkOpSummary

GetSummary returns this batcher's total added rows/bytes/etc.

func (*BufferingAdder) IsEmpty Uses

func (b *BufferingAdder) IsEmpty() bool

IsEmpty returns true if the adder has no un-flushed data in its buffer.

func (*BufferingAdder) SetOnFlush Uses

func (b *BufferingAdder) SetOnFlush(fn func())

SetOnFlush sets a callback to run after the buffering adder flushes.

type Metrics Uses

type Metrics struct {
    MaxBytesHist  *metric.Histogram
    CurBytesCount *metric.Gauge

Metrics contains pointers to the metrics for monitoring bulk operations.

func MakeBulkMetrics Uses

func MakeBulkMetrics(histogramWindow time.Duration) Metrics

MakeBulkMetrics instantiates the metrics holder for bulk operation monitoring.

func (Metrics) MetricStruct Uses

func (Metrics) MetricStruct()

MetricStruct implements the metrics.Struct interface.

type SSTBatcher Uses

type SSTBatcher struct {
    // contains filtered or unexported fields

SSTBatcher is a helper for bulk-adding many KVs in chunks via AddSSTable. An SSTBatcher can be handed KVs repeatedly and will make them into SSTs that are added when they reach the configured size, tracking the total added rows, bytes, etc. If configured with a non-nil, populated range cache, it will use it to attempt to flush SSTs before they cross range boundaries to minimize expensive on-split retries.

func MakeSSTBatcher Uses

func MakeSSTBatcher(
    ctx context.Context, db SSTSender, settings *cluster.Settings, flushBytes func() int64,
) (*SSTBatcher, error)

MakeSSTBatcher makes a ready-to-use SSTBatcher.

func (*SSTBatcher) AddMVCCKey Uses

func (b *SSTBatcher) AddMVCCKey(ctx context.Context, key storage.MVCCKey, value []byte) error

AddMVCCKey adds a key+timestamp/value pair to the batch (flushing if needed). This is only for callers that want to control the timestamp on individual keys -- like RESTORE where we want the restored data to look the like backup. Keys must be added in order.

func (*SSTBatcher) Close Uses

func (b *SSTBatcher) Close()

Close closes the underlying SST builder.

func (*SSTBatcher) Flush Uses

func (b *SSTBatcher) Flush(ctx context.Context) error

Flush sends the current batch, if any.

func (*SSTBatcher) GetSummary Uses

func (b *SSTBatcher) GetSummary() roachpb.BulkOpSummary

GetSummary returns this batcher's total added rows/bytes/etc.

func (*SSTBatcher) Reset Uses

func (b *SSTBatcher) Reset(ctx context.Context) error

Reset clears all state in the batcher and prepares it for reuse.

type SSTSender Uses

type SSTSender interface {
        ctx context.Context, begin, end interface{}, data []byte, disallowShadowing bool, stats *enginepb.MVCCStats, ingestAsWrites bool,
    ) error
    SplitAndScatter(ctx context.Context, key roachpb.Key, expirationTime hlc.Timestamp) error

SSTSender is an interface to send SST data to an engine.

Package bulk imports 20 packages (graph) and is imported by 6 packages. Updated 2020-08-12. Refresh now. Tools for package owners.