pbq

package
v1.2.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: May 7, 2024 License: Apache-2.0 Imports: 16 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func DefaultOptions

func DefaultOptions() *options

Types

type Manager

type Manager struct {

	// we need lock to access pbqMap, since deregister will be called inside pbq
	// and each pbq will be inside a go routine, and also entire PBQ could be managed
	// through a go routine (depends on the orchestrator)
	sync.RWMutex
	// contains filtered or unexported fields
}

Manager helps in managing the lifecycle of PBQ instances

func NewManager

func NewManager(ctx context.Context, vertexName string, pipelineName string, vr int32, storeProvider wal.Manager, windowType window.Type, opts ...PBQOption) (*Manager, error)

NewManager returns new instance of manager We don't intend this to be called by multiple routines.

func (*Manager) CreateNewPBQ

func (m *Manager) CreateNewPBQ(ctx context.Context, partitionID partition.ID) (ReadWriteCloser, error)

CreateNewPBQ creates new pbq for a partition

func (*Manager) GetPBQ

func (m *Manager) GetPBQ(partitionID partition.ID) ReadWriteCloser

GetPBQ returns pbq for the given ID

func (*Manager) ListPartitions

func (m *Manager) ListPartitions() []*PBQ

ListPartitions returns all the pbq instances

func (*Manager) ShutDown

func (m *Manager) ShutDown(ctx context.Context)

ShutDown for clean shut down, flushes pending messages to store and closes the store

type PBQ

type PBQ struct {
	PartitionID partition.ID
	// contains filtered or unexported fields
}

PBQ Buffer queue which is backed with a persisted store, each partition will have a PBQ associated with it

func (*PBQ) Close

func (p *PBQ) Close() error

Close is used by the writer to indicate close of context we should flush pending messages to store

func (*PBQ) CloseOfBook

func (p *PBQ) CloseOfBook()

CloseOfBook closes output channel

func (*PBQ) GC

func (p *PBQ) GC() error

GC cleans up the PBQ and also the store associated with it. GC is invoked after the Reader (ProcessAndForward) has finished forwarding the output to ISB.

func (*PBQ) ReadCh

func (p *PBQ) ReadCh() <-chan *window.TimedWindowRequest

ReadCh exposes read channel to read the window requests from the PBQ close on read channel indicates COB

func (*PBQ) Write

func (p *PBQ) Write(ctx context.Context, request *window.TimedWindowRequest, persist bool) error

Write accepts a window request and writes it to the PBQ, only the isb message is written to the store. The other metadata like operation etc are recomputed from WAL. request can never be nil.

type PBQOption

type PBQOption func(options *options) error

func WithChannelBufferSize

func WithChannelBufferSize(size int64) PBQOption

WithChannelBufferSize sets buffer size option

func WithReadBatchSize

func WithReadBatchSize(size int64) PBQOption

WithReadBatchSize sets read batch size option

func WithReadTimeout

func WithReadTimeout(seconds time.Duration) PBQOption

WithReadTimeout sets read timeout option

type ReadWriteCloser

type ReadWriteCloser interface {
	Reader
	WriteCloser
}

ReadWriteCloser is an unified interface to PBQ read and write interfaces. Close is only for Writer.

type Reader

type Reader interface {
	// ReadCh exposes channel to read from PBQ
	ReadCh() <-chan *window.TimedWindowRequest
	// GC does garbage collection, it deletes all the persisted data from the store
	GC() error
}

Reader provides methods to read from PBQ.

type WriteCloser

type WriteCloser interface {
	// Write writes the TimedWindowRequest to PBQ
	// persist to decide if the data should be persisted or not
	// during replay persist will be false
	Write(ctx context.Context, msg *window.TimedWindowRequest, persist bool) error
	// CloseOfBook (cob) closes PBQ, no writes will be accepted after cob
	CloseOfBook()
	// Close to handle context close on writer
	// Any pending data can be flushed to the persistent store at this point.
	Close() error
}

WriteCloser provides methods to write data to the PQB and close the PBQ. No data can be written to PBQ after cob.

Directories

Path Synopsis
Package partition is a tuple containing (start, end) time and an optional slot.
Package partition is a tuple containing (start, end) time and an optional slot.
wal
aligned/fs
Package fs implements write-ahead-log on FileSystem.
Package fs implements write-ahead-log on FileSystem.
unaligned/fs
Package fs implements write-ahead-log on FileSystem.
Package fs implements write-ahead-log on FileSystem.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL