list

package
v0.0.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 5, 2019 License: Apache-2.0 Imports: 15 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	ListVectorPartyHeader uint32 = 0xFADEFACF
)

Variables

This section is empty.

Functions

func IsNilAddr 

func IsNilAddr(addr [2]uintptr) bool

func NewArchiveVectorParty 

func NewArchiveVectorParty(length int, dataType common.DataType,
	totalValueBytes int64, locker sync.Locker) common.ArchiveVectorParty

NewArchiveVectorParty returns a new ArchiveVectorParty. It should only be used during backfill or archiving when constructing a new list archiving vp. Length is the number of total rows and totalValueBytes is the total bytes used to store values and validities.

func NewLiveVectorParty 

func NewLiveVectorParty(length int, dataType common.DataType,
	hmm common.HostMemoryManager) common.LiveVectorParty

NewLiveVectorParty returns a LiveVectorParty pointer which implements ListVectorParty. It's safe to pass nil HostMemoryManager.

func ToArrayArchiveVectorParty 

func ToArrayArchiveVectorParty(vp memCom.VectorParty, locker sync.Locker) memCom.ArchiveVectorParty

func ToArrayLiveVectorParty 

func ToArrayLiveVectorParty(vp memCom.VectorParty) memCom.LiveVectorParty

func ToArrayVectorParty 

func ToArrayVectorParty(rvp *tests.RawVectorParty, forLiveVP bool) (vp memCom.VectorParty, err error)

Types

type ArchiveVectorParty 

type ArchiveVectorParty struct {
	common.Pinnable
	// contains filtered or unexported fields
}

ArchiveVectorParty is the representation of list data type vector party in archive store. It does not support random access update. Instead updates to archiveListVectorParty can only be done via appending to the tail during archiving and backfill. It use a single value vector to store the values and validities so that it has the same in memory representation except archiving vp does not have cap vector. The only update supported is same length value in place change

func (*ArchiveVectorParty) Allocate 

func (vp *ArchiveVectorParty) Allocate(hasCount bool)

Allocate allocate underlying storage for vector party. Note allocation for archive vp does not report host memory change. Memory reporting is done after switching to the new version of archive store. Before switching the memory managed by this vp is counted as unmanaged memory.

func (*ArchiveVectorParty) AsList 

func (vp *ArchiveVectorParty) AsList() common.ListVectorParty

AsList is the implementation from common.VectorParty

func (*ArchiveVectorParty) CopyOnWrite 

func (vp *ArchiveVectorParty) CopyOnWrite(batchSize int) common.ArchiveVectorParty

CopyOnWrite clone vector party for updates, the update can only for in-place change with same length for update row

func (*ArchiveVectorParty) Dump 

func (vp *ArchiveVectorParty) Dump(file *os.File)

Dump is for testing purpose

func (*ArchiveVectorParty) Equals 

func (vp *ArchiveVectorParty) Equals(other common.VectorParty) bool

Equals is the implementation from common.VectorParty

func (*ArchiveVectorParty) GetBytes 

func (vp *ArchiveVectorParty) GetBytes() int64

GetBytes returns the bytes this vp occupies.

func (*ArchiveVectorParty) GetCount 

func (vp *ArchiveVectorParty) GetCount(offset int) uint32

GetCount returns cumulative count on specified offset.

func (*ArchiveVectorParty) GetDataType 

func (vp *ArchiveVectorParty) GetDataType() common.DataType

GetDataType returns the element date type of this vp.

func (*ArchiveVectorParty) GetDataValue 

func (vp *ArchiveVectorParty) GetDataValue(row int) common.DataValue

GetDataValue is not implemented in baseVectorParty

func (*ArchiveVectorParty) GetDataValueByRow 

func (vp *ArchiveVectorParty) GetDataValueByRow(row int) common.DataValue

GetDataValueByRow just call GetDataValue

func (*ArchiveVectorParty) GetElemCount 

func (vp *ArchiveVectorParty) GetElemCount(row int) uint32

GetElemCount return the number of element for value in n-th row

func (*ArchiveVectorParty) GetLength 

func (vp *ArchiveVectorParty) GetLength() int

GetLength returns the length of the vp.

func (*ArchiveVectorParty) GetListValue 

func (vp *ArchiveVectorParty) GetListValue(row int) (unsafe.Pointer, bool)

SetListValue is the implentation of common.ListVecotrParty

func (*ArchiveVectorParty) GetNonDefaultValueCount 

func (vp *ArchiveVectorParty) GetNonDefaultValueCount() int

GetNonDefaultValueCount get Number of non-default values stored. Since we count all list values as valid values, it should be equal to the length of the vp. If in future we want to get a count of non default element value count, we may need to scan all the old element values when overwriting.

func (*ArchiveVectorParty) GetOffsetLength 

func (vp *ArchiveVectorParty) GetOffsetLength(row int) (offset uint32, length uint32)

GetOffsetLength returns the <offset, length> pair at ith row.

func (*ArchiveVectorParty) GetValidity 

func (vp *ArchiveVectorParty) GetValidity(row int) bool

GetValidity get validity of given offset.

func (*ArchiveVectorParty) IsList 

func (vp *ArchiveVectorParty) IsList() bool

IsList tells whether this vp is list vp. And user can later on cast it to proper interface.

func (*ArchiveVectorParty) LoadFromDisk 

func (vp *ArchiveVectorParty) LoadFromDisk(hostMemManager common.HostMemoryManager, diskStore diskstore.DiskStore,
	table string, shardID int, columnID, batchID int, batchVersion uint32, seqNum uint32)

LoadFromDisk load archive vector party from disk caller should lock archive batch before using

func (*ArchiveVectorParty) Prune 

func (vp *ArchiveVectorParty) Prune()

Prune prunes vector party based on column mode to clean memory if possible

func (*ArchiveVectorParty) Read 

func (vp *ArchiveVectorParty) Read(reader io.Reader, s common.VectorPartySerializer) error

Read reads a vector party from underlying reader. It first reads header from the reader and does several sanity checks. Then it reads vectors based on vector party mode.

func (*ArchiveVectorParty) SafeDestruct 

func (vp *ArchiveVectorParty) SafeDestruct()

SafeDestruct destructs vector party memory.

func (*ArchiveVectorParty) SetCount 

func (vp *ArchiveVectorParty) SetCount(offset int, count uint32)

SetCount is not supported by list vector party.

func (*ArchiveVectorParty) SetDataValue 

func (vp *ArchiveVectorParty) SetDataValue(row int, value common.DataValue,
	countsUpdateMode common.ValueCountsUpdateMode, counts ...uint32)

SetDataValue is the implentation of common.VecotrParty

func (*ArchiveVectorParty) SetListValue 

func (vp *ArchiveVectorParty) SetListValue(row int, val unsafe.Pointer, valid bool)

SetListValue is the implentation of common.ListVecotrParty

func (*ArchiveVectorParty) SetOffsetLength 

func (vp *ArchiveVectorParty) SetOffsetLength(row int, offset, length unsafe.Pointer)

SetOffsetLength update offset/length for nth fow

func (*ArchiveVectorParty) Slice 

func (vp *ArchiveVectorParty) Slice(startRow, numRows int) common.SlicedVector

Slice vector party into human readable SlicedVector format. For now just return an empty slice. TODO(lucafuji): implement slice vector on list vp.

func (*ArchiveVectorParty) SliceByValue 

func (vp *ArchiveVectorParty) SliceByValue(lowerBoundRow, upperBoundRow int, value unsafe.Pointer) (
	startRow int, endRow int, startIndex int, endIndex int)

SliceByValue is not supported by list vector party.

func (*ArchiveVectorParty) SliceIndex 

func (vp *ArchiveVectorParty) SliceIndex(lowerBoundRow, upperBoundRow int) (
	startIndex, endIndex int)

Slice vector party to get [startIndex, endIndex) based on [lowerBoundRow, upperBoundRow)

func (*ArchiveVectorParty) Write 

func (vp *ArchiveVectorParty) Write(writer io.Writer) error

Write is the implentation of common.VecotrParty

type Arena 

type Arena struct {
	// contains filtered or unexported fields
}

An Arena manages a set of slab classes and memoryOffset.

func NewArena 

func NewArena(startChunkSize int, slabSize int, growthFactor float64,
	mp NativeMemoryPool) *Arena

NewArena returns an Arena to manage byte slice memoryOffset based on a slab allocator approach.

The startChunkSize and slabSize should be > 0. The growthFactor should be > 1.0.

func (*Arena) AddRef 

func (s *Arena) AddRef(offsets [2]uintptr)

AddRef increase the ref count on a buf. The input buf must be from an Alloc() from the same Arena.

func (*Arena) Alloc 

func (s *Arena) Alloc(bufLen int) [2]uintptr

Alloc may return nil on errors, such as if no more free chunks are available and new slab memoryOffset was not allocatable (such as if malloc() returns nil). The returned buf may not be append()'ed to for growth. The returned buf must be DecRef()'ed for memoryOffset reuse.

func (*Arena) DecRef 

func (s *Arena) DecRef(buf [2]uintptr) bool

DecRef decreases the ref count on a buf. The input buf must be from an Alloc() from the same Arena. Once the buf's ref-count drops to 0, the Arena may reuse the buf. Returns true if this was the last DecRef() invocation (ref count reached 0).

func (*Arena) Owns 

func (s *Arena) Owns(offsets [2]uintptr) bool

Owns returns true if this Arena owns the buf.

func (*Arena) Stats 

func (s *Arena) Stats(m map[string]int64) map[string]int64

Stats fills an input map with runtime metrics about the Arena.

type HighLevelMemoryPool 

type HighLevelMemoryPool interface {
	// Allocate allocates size byte memory and return back to client.
	Allocate(size int) [2]uintptr
	// Reallocate reallocates memory according to the size of the old allocated
	// memory and size of new allocation requests. If the size is the same, it
	// does nothing and just return the old addr. Otherwise it will allocate a new
	// memory, copy the old content to it if oldSize is non-zero and returned back to
	// client. If the oldSize is not zero, it will also free the old memory.
	Reallocate(oldBuf [2]uintptr, oldSize int, newSize int) [2]uintptr
	// Return the memory back to memory pool.
	Free(buf [2]uintptr)
	// Return the actual memory address given offset.
	Interpret(offset uintptr) uintptr
	// Return the underlying native memory pool allocator.
	GetNativeMemoryAllocator() NativeMemoryPool
	// Release the underlying memory.
	Destroy()
}

HighLevelMemoryPool manages memory requests on pooled memory. It underlying uses a slab allocator to manage free memory chunks. When it no longer can satisfy memory allocation request from customer, it will request more memory from underlying NativeMemoryPool. All address returned back to client are an 2 element array of offset where the first offset is the memory allocated to the caller and second offset is the footer offset to current slab. Note that allocate a memory chunk larger than slabSize will fail. For more information related how slab allocator works, please refer to https://github.com/couchbase/go-slab.

func NewHighLevelMemoryPool 

func NewHighLevelMemoryPool(reporter HostMemoryChangeReporter) HighLevelMemoryPool

NewHighLevelMemoryPool returns a default implementation of HighLevelMemoryPool.

type HostMemoryChangeReporter 

type HostMemoryChangeReporter func(bytesChanged int64)

type LiveVectorParty 

type LiveVectorParty struct {
	sync.RWMutex
	// contains filtered or unexported fields
}

LiveVectorParty is the representation of list data type vector party in live store. It supports random access read and write. However, it does not support serialization into disk. It underlying uses a high level memory pool to store the list data. Therefore when this vector party is destructed, the underlying memory pool needs to be destroyed as well.

func (*LiveVectorParty) Allocate 

func (vp *LiveVectorParty) Allocate(hasCount bool)

Allocate allocate underlying storage for vector party

func (*LiveVectorParty) AsList 

func (vp *LiveVectorParty) AsList() common.ListVectorParty

AsList is the implementation from common.VectorParty

func (*LiveVectorParty) Dump 

func (vp *LiveVectorParty) Dump(file *os.File)

Dump is for testing purpose

func (*LiveVectorParty) Equals 

func (vp *LiveVectorParty) Equals(other common.VectorParty) bool

Equals is the implementation from common.VectorParty

func (*LiveVectorParty) GetBytes 

func (vp *LiveVectorParty) GetBytes() int64

GetBytes returns the bytes this vp occupies except memory pool

func (*LiveVectorParty) GetCap 

func (vp *LiveVectorParty) GetCap(row int) uint32

GetCap returns the cap at ith row. Only used for free a list element in live store.

func (*LiveVectorParty) GetDataType 

func (vp *LiveVectorParty) GetDataType() common.DataType

GetDataType returns the element date type of this vp.

func (*LiveVectorParty) GetDataValue 

func (vp *LiveVectorParty) GetDataValue(row int) common.DataValue

GetDataValue is not implemented in baseVectorParty

func (*LiveVectorParty) GetDataValueByRow 

func (vp *LiveVectorParty) GetDataValueByRow(row int) common.DataValue

GetDataValueByRow just call GetDataValue

func (*LiveVectorParty) GetElemCount 

func (vp *LiveVectorParty) GetElemCount(row int) uint32

GetElemCount return the number of element for value in n-th row

func (*LiveVectorParty) GetLength 

func (vp *LiveVectorParty) GetLength() int

GetLength returns the length of the vp.

func (*LiveVectorParty) GetListValue 

func (vp *LiveVectorParty) GetListValue(row int) (unsafe.Pointer, bool)

SetListValue is the implentation of common.ListVecotrParty

func (*LiveVectorParty) GetMinMaxValue 

func (vp *LiveVectorParty) GetMinMaxValue() (min, max uint32)

GetMinMaxValue is not supported by list vector party.

func (*LiveVectorParty) GetNonDefaultValueCount 

func (vp *LiveVectorParty) GetNonDefaultValueCount() int

GetNonDefaultValueCount get Number of non-default values stored. Since we count all list values as valid values, it should be equal to the length of the vp. If in future we want to get a count of non default element value count, we may need to scan all the old element values when overwriting.

func (*LiveVectorParty) GetOffsetLength 

func (vp *LiveVectorParty) GetOffsetLength(row int) (offset uint32, length uint32)

GetOffsetLength returns the <offset, length> pair at ith row.

func (*LiveVectorParty) GetTotalBytes 

func (vp *LiveVectorParty) GetTotalBytes() int64

GetTotalBytes return the bytes this vp occupies including memory pool

func (*LiveVectorParty) GetValidity 

func (vp *LiveVectorParty) GetValidity(row int) bool

GetValidity get validity of given offset.

func (*LiveVectorParty) GetValue 

func (vp *LiveVectorParty) GetValue(row int) (val unsafe.Pointer, validity bool)

GetValue is the implementation from common.VectorParty

func (*LiveVectorParty) IsList 

func (vp *LiveVectorParty) IsList() bool

IsList tells whether this vp is list vp. And user can later on cast it to proper interface.

func (*LiveVectorParty) Read 

func (vp *LiveVectorParty) Read(reader io.Reader, serializer common.VectorPartySerializer) (err error)

Read deserialize vector party

func (*LiveVectorParty) SafeDestruct 

func (vp *LiveVectorParty) SafeDestruct()

SafeDestruct destructs vector party memory.

func (*LiveVectorParty) SetBool 

func (vp *LiveVectorParty) SetBool(offset int, val bool, valid bool)

SetBool is not supported by list vector party.

func (*LiveVectorParty) SetDataValue 

func (vp *LiveVectorParty) SetDataValue(row int, value common.DataValue,
	countsUpdateMode common.ValueCountsUpdateMode, counts ...uint32)

SetDataValue

func (*LiveVectorParty) SetGoValue 

func (vp *LiveVectorParty) SetGoValue(offset int, val common.GoDataValue, valid bool)

SetGoValue is not supported by list vector party.

func (*LiveVectorParty) SetLength 

func (vp *LiveVectorParty) SetLength(length int)

SetLength is only for testing purpose, do NOT use this function in real code

func (*LiveVectorParty) SetListValue 

func (vp *LiveVectorParty) SetListValue(row int, val unsafe.Pointer, valid bool)

SetListValue is the implentation of common.ListVecotrParty

func (*LiveVectorParty) SetOffsetLength 

func (vp *LiveVectorParty) SetOffsetLength(row int, offset, length unsafe.Pointer)

SetOffsetLength update offset/length for nth fow

func (*LiveVectorParty) SetValue 

func (vp *LiveVectorParty) SetValue(row int, val unsafe.Pointer, valid bool)

SetValue is the implementation of common.LiveVectorParty

func (*LiveVectorParty) Slice 

func (vp *LiveVectorParty) Slice(startRow, numRows int) common.SlicedVector

Slice vector party into human readable SlicedVector format. For now just return an empty slice. TODO(lucafuji): implement slice vector on list vp.

func (*LiveVectorParty) Write 

func (vp *LiveVectorParty) Write(writer io.Writer) (err error)

Write serialize vector party.

type Loc 

type Loc struct {
	// contains filtered or unexported fields
}

An opaque reference to bytes managed by an Arena. See Arena.BufToLoc/LocToBuf(). A Loc struct is GC friendly in that a Loc does not have direct pointer fields into the Arena's memoryOffset that the GC's scanner must traverse.

func NilLoc 

func NilLoc() Loc

NilLoc returns a Loc where Loc.IsNil() is true.

func (Loc) IsNil 

func (cl Loc) IsNil() bool

IsNil returns true if the Loc came from NilLoc().

type NativeMemoryPool 

type NativeMemoryPool interface {
	// Malloc returns a byte slice to caller, will allocate more memoryOffset if no enough space.
	// returned addresses are not aligned.
	Malloc(size int) uintptr
	// Destroy frees the memory managed by this memory pool. After destroy, any malloc call's
	// behaviour will be undefined.
	Destroy()
	// GetBaseAddr returns the base address managed by this pool.
	GetBaseAddr() uintptr
	// GetTotalBytes returns the total bytes occupied by this memory pool.
	GetTotalBytes() int64
}

NativeMemoryPool is the interface to manage system memory to support high level memory pool allocation requests. All the pointer/address returned by this memory pool is relative to the base address fetched via GetBaseAddr.

func NewNativeMemoryPool 

func NewNativeMemoryPool(reporter HostMemoryChangeReporter) NativeMemoryPool

NewNativeMemoryPool returns a default implementation of NativeMemoryPool.

type TestFactoryT 

type TestFactoryT struct {
	tests.TestFactoryBase
}

TestFactoryT creates test objects from text file

func GetFactory 

func GetFactory() TestFactoryT

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.