arrowutils

func EnsureSameSchema ¶

func EnsureSameSchema(records []arrow.Record) ([]arrow.Record, error)

EnsureSameSchema ensures that all the records have the same schema. In cases where the schema is not equal, virtual null columns are inserted in the records with the missing column. When we have static schemas in the execution engine, steps like these should be unnecessary.

func GetGroupsAndOrderedSetRanges ¶

func GetGroupsAndOrderedSetRanges(
	firstGroup []any, arrs []arrow.Array,
) (*Int64Heap, *Int64Heap, []any, error)

GetGroupsAndOrderedSetRanges returns a min-heap of group ranges and ordered set ranges of the given arrow arrays in that order. For the given input with a single array: a a c d a b c This function will return [2, 3, 4, 5, 6] for the group ranges and [4] for the ordered set ranges. A group is a collection of values that are equal and an ordered set is a collection of groups that are in increasing order. The ranges are determined by iterating over the arrays and comparing the current group value for each column. The firstGroup to compare against must be provided (it can be initialized to the values at index 0 of each array). The last group found is returned.

func MakeNullArray ¶

func MakeNullArray(mem memory.Allocator, dt arrow.DataType, len int) arrow.Array

MakeNullArray makes a physical arrow.Array full of NULLs of the given DataType.

func MergeRecords ¶

func MergeRecords(
	mem memory.Allocator,
	records []arrow.Record,
	orderByCols []SortingColumn,
	limit uint64,
) (arrow.Record, error)

MergeRecords merges the given records. The records must all have the same schema. orderByCols is a slice of indexes into the columns that the records and resulting records are ordered by. While ordering the limit is checked before appending more rows. If limit is 0, no limit is applied. Note that the given records should already be ordered by the given columns. WARNING: Only ascending ordering is currently supported.

func SortRecord ¶

func SortRecord(r arrow.Record, columns []SortingColumn) (*array.Int32, error)

SortRecord sorts given arrow.Record by columns. Returns *array.Int32 of indices to sorted rows or record r.

Comparison is made sequentially by each column. When rows are equal in the first column we compare the rows om the second column and so on and so forth until rows that are not equal are found.

func Take ¶

func Take(ctx context.Context, r arrow.Record, indices *array.Int32) (arrow.Record, error)

Take uses indices which is an array of row index and returns a new record that only contains rows specified in indices.

Use compute.WithAllocator to pass a custom memory.Allocator.

func TakeColumn ¶

func TakeColumn(ctx context.Context, a arrow.Array, idx int, arr []arrow.Array, indices *array.Int32) error

func TakeDictColumn ¶

func TakeDictColumn(ctx context.Context, a *array.Dictionary, idx int, arr []arrow.Array, indices *array.Int32) error