cockroach: Index | Files

package bulk

import ""


Package Files

buffering_adder.go bulk_metrics.go kv_buf.go row_counter.go sst_batcher.go sst_writer.go

func AddSSTable Uses

func AddSSTable(
    ctx context.Context,
    db sender,
    start, end roachpb.Key,
    sstBytes []byte,
    disallowShadowing bool,
    ms enginepb.MVCCStats,
) (int, error)

AddSSTable retries db.AddSSTable if retryable errors occur, including if the SST spans a split, in which case it is iterated and split into two SSTs, one for each side of the split in the error, and each are retried.

type BufferingAdder Uses

type BufferingAdder struct {
    // contains filtered or unexported fields

BufferingAdder is a wrapper for an SSTBatcher that allows out-of-order calls to Add, buffering them up and then sorting them before then passing them in order into an SSTBatcher

func MakeBulkAdder Uses

func MakeBulkAdder(
    ctx context.Context,
    db sender,
    rangeCache *kv.RangeDescriptorCache,
    timestamp hlc.Timestamp,
    opts storagebase.BulkAdderOptions,
    bulkMon *mon.BytesMonitor,
) (*BufferingAdder, error)

MakeBulkAdder makes a storagebase.BulkAdder that buffers and sorts K/Vs passed to add into SSTs that are then ingested. rangeCache if set is consulted to avoid generating an SST that will span a range boundary and thus encounter an error and need to be split and retired to be applied.

func (*BufferingAdder) Add Uses

func (b *BufferingAdder) Add(ctx context.Context, key roachpb.Key, value []byte) error

Add adds a key to the buffer and checks if it needs to flush.

func (*BufferingAdder) Close Uses

func (b *BufferingAdder) Close(ctx context.Context)

Close closes the underlying SST builder.

func (*BufferingAdder) CurrentBufferFill Uses

func (b *BufferingAdder) CurrentBufferFill() float32

CurrentBufferFill returns the current buffer fill percentage.

func (*BufferingAdder) Flush Uses

func (b *BufferingAdder) Flush(ctx context.Context) error

Flush flushes any buffered kvs to the batcher.

func (*BufferingAdder) GetSummary Uses

func (b *BufferingAdder) GetSummary() roachpb.BulkOpSummary

GetSummary returns this batcher's total added rows/bytes/etc.

func (*BufferingAdder) IsEmpty Uses

func (b *BufferingAdder) IsEmpty() bool

IsEmpty returns true if the adder has no un-flushed data in its buffer.

func (*BufferingAdder) SetOnFlush Uses

func (b *BufferingAdder) SetOnFlush(fn func())

SetOnFlush sets a callback to run after the buffering adder flushes.

type Metrics Uses

type Metrics struct {
    MaxBytesHist  *metric.Histogram
    CurBytesCount *metric.Gauge

Metrics contains pointers to the metrics for monitoring bulk operations.

func MakeBulkMetrics Uses

func MakeBulkMetrics(histogramWindow time.Duration) Metrics

MakeBulkMetrics instantiates the metrics holder for bulk operation monitoring.

func (Metrics) MetricStruct Uses

func (Metrics) MetricStruct()

MetricStruct implements the metrics.Struct interface.

type RowCounter Uses

type RowCounter struct {
    // contains filtered or unexported fields

RowCounter is a helper that counts how many distinct rows appear in the KVs that is is shown via `Count`. Note: the `DataSize` field of the BulkOpSummary is *not* populated by this and should be set separately.

func (*RowCounter) Count Uses

func (r *RowCounter) Count(key roachpb.Key) error

Count examines each key passed to it and increments the running count when it sees a key that belongs to a new row.

type SSTBatcher Uses

type SSTBatcher struct {
    // contains filtered or unexported fields

SSTBatcher is a helper for bulk-adding many KVs in chunks via AddSSTable. An SSTBatcher can be handed KVs repeatedly and will make them into SSTs that are added when they reach the configured size, tracking the total added rows, bytes, etc. If configured with a non-nil, populated range cache, it will use it to attempt to flush SSTs before they cross range boundaries to minimize expensive on-split retries.

func MakeSSTBatcher Uses

func MakeSSTBatcher(ctx context.Context, db sender, flushBytes uint64) (*SSTBatcher, error)

MakeSSTBatcher makes a ready-to-use SSTBatcher.

func (*SSTBatcher) AddMVCCKey Uses

func (b *SSTBatcher) AddMVCCKey(ctx context.Context, key engine.MVCCKey, value []byte) error

AddMVCCKey adds a key+timestamp/value pair to the batch (flushing if needed). This is only for callers that want to control the timestamp on individual keys -- like RESTORE where we want the restored data to look the like backup. Keys must be added in order.

func (*SSTBatcher) Close Uses

func (b *SSTBatcher) Close()

Close closes the underlying SST builder.

func (*SSTBatcher) Flush Uses

func (b *SSTBatcher) Flush(ctx context.Context) error

Flush sends the current batch, if any.

func (*SSTBatcher) GetSummary Uses

func (b *SSTBatcher) GetSummary() roachpb.BulkOpSummary

GetSummary returns this batcher's total added rows/bytes/etc.

func (*SSTBatcher) Reset Uses

func (b *SSTBatcher) Reset() error

Reset clears all state in the batcher and prepares it for reuse.

type SSTWriter Uses

type SSTWriter struct {

    // DataSize tracks the total key and value bytes added so far.
    DataSize uint64
    // contains filtered or unexported fields

SSTWriter writes SSTables.

func MakeSSTWriter Uses

func MakeSSTWriter() SSTWriter

MakeSSTWriter creates a new SSTWriter.

func (*SSTWriter) Add Uses

func (fw *SSTWriter) Add(kv engine.MVCCKeyValue) error

Add puts a kv entry into the sstable being built. An error is returned if it is not greater than any previously added entry (according to the comparator configured during writer creation). `Close` cannot have been called.

func (*SSTWriter) Close Uses

func (fw *SSTWriter) Close()

Close finishes and frees memory and other resources. Close is idempotent.

func (*SSTWriter) Finish Uses

func (fw *SSTWriter) Finish() ([]byte, error)

Finish finalizes the writer and returns the constructed file's contents. At least one kv entry must have been added.

Package bulk imports 22 packages (graph) and is imported by 8 packages. Updated 2019-09-19. Refresh now. Tools for package owners.