bridge

package module
v0.0.0-...-196150a Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 23, 2019 License: Apache-2.0 Imports: 6 Imported by: 0

README

go-py-arrow-bridge

GoDoc CircleCI

A rudimentary bridge for Apache Arrow between Go and Python to facilitate zero-copy.

This Go module demonstrates in the tests how easy it is to create an Arrow Table in Python and use the same Arrow Table in Go without copying the underlying buffers.

Installation

Add the package to your go.mod file:

require github.com/nickpoorman/go-py-arrow-bridge master

Or, clone the repository:

git clone --branch master https://github.com/nickpoorman/go-py-arrow-bridge.git $GOPATH/src/github.com/nickpoorman/go-py-arrow-bridge

Example

See the example or clone down the repo an run it via make run.

Benchmarks

As you can see below, the amount of time to move data across the Python/Go language boundary stays constant as the number of elements increases.

However, as the number of chunks increase, the amount of time also increases. I believe this is due to the large number of CGO calls happening in loops. A future version might try to reduce the number of CGO calls by implementing the schema data gathering in C. In the meantime, a workaround could compress the table down to a single chunk before crossing the language boundary.

These results are from my Mid 2012 MacBook Air (1.8GHz i5 / 8 GB 1600 MHz DDR3).

(bullseye) ➜  go-py-arrow-bridge git:(master) ✗ PKG_CONFIG_PATH=/Users/nick/anaconda3/envs/bullseye/lib/pkgconfig LD_LIBRARY_PATH=/Users/nick/anaconda3/envs/bullseye/lib/python3.7:/Users/nick/anaconda3/envs/bullseye/lib PYTHONPATH=/Users/nick/anaconda3/envs/bullseye/lib/python3.7/site-packages:/Users/nick/projects/go-py-arrow-bridge/__python__ go test -bench=. -run=- -cpuprofile cpu.prof
goos: darwin
goarch: amd64
pkg: github.com/nickpoorman/go-py-arrow-bridge
BenchmarkAll/BenchmarkZeroCopyChunks_5-4                    3000            402994 ns/op
BenchmarkAll/BenchmarkZeroCopyChunks_7-4                    3000            463610 ns/op
BenchmarkAll/BenchmarkZeroCopyChunks_9-4                    2000            553570 ns/op
BenchmarkAll/BenchmarkZeroCopyChunks_1000-4                   30          67841714 ns/op
BenchmarkAll/BenchmarkZeroCopyChunks_1500-4                   20          72966442 ns/op
BenchmarkAll/BenchmarkZeroCopyChunks_2000-4                   20          91238313 ns/op
BenchmarkAll/BenchmarkZeroCopyChunks_2500-4                   10         120581479 ns/op
BenchmarkAll/BenchmarkZeroCopyChunks_3000-4                   10         149069387 ns/op
BenchmarkAll/BenchmarkZeroCopyChunks_3500-4                   10         168897623 ns/op
BenchmarkAll/BenchmarkZeroCopyChunks_4000-4                   10         187915637 ns/op
BenchmarkAll/BenchmarkZeroCopyChunks_4500-4                    5         209468675 ns/op
BenchmarkAll/BenchmarkZeroCopyChunks_5000-4                    5         232361156 ns/op
BenchmarkAll/BenchmarkZeroCopyChunks_5500-4                    5         249023617 ns/op
BenchmarkAll/BenchmarkZeroCopyChunks_6000-4                    5         274385207 ns/op
BenchmarkAll/BenchmarkZeroCopyChunks_6500-4                    5         305522949 ns/op
BenchmarkAll/BenchmarkZeroCopyChunks_7000-4                    5         324781757 ns/op
BenchmarkAll/BenchmarkZeroCopyChunks_7500-4                    3         349889266 ns/op
BenchmarkAll/BenchmarkZeroCopyChunks_8000-4                    3         372640132 ns/op
BenchmarkAll/BenchmarkZeroCopyChunks_8500-4                    3         394905472 ns/op
BenchmarkAll/BenchmarkZeroCopyChunks_9000-4                    3         413965959 ns/op
BenchmarkAll/BenchmarkZeroCopyChunks_9500-4                    3         440292768 ns/op
BenchmarkAll/BenchmarkZeroCopyChunks_10000-4                   3         461282623 ns/op
BenchmarkAll/BenchmarkZeroCopyElements_5-4                 10000            166585 ns/op
BenchmarkAll/BenchmarkZeroCopyElements_7-4                 10000            165732 ns/op
BenchmarkAll/BenchmarkZeroCopyElements_9-4                 10000            177193 ns/op
BenchmarkAll/BenchmarkZeroCopyElements_1000-4              10000            166462 ns/op
BenchmarkAll/BenchmarkZeroCopyElements_1500-4              10000            166774 ns/op
BenchmarkAll/BenchmarkZeroCopyElements_2000-4              10000            169948 ns/op
BenchmarkAll/BenchmarkZeroCopyElements_2500-4              10000            171018 ns/op
BenchmarkAll/BenchmarkZeroCopyElements_3000-4              10000            168100 ns/op
BenchmarkAll/BenchmarkZeroCopyElements_3500-4              10000            171136 ns/op
BenchmarkAll/BenchmarkZeroCopyElements_4000-4              10000            166941 ns/op
BenchmarkAll/BenchmarkZeroCopyElements_4500-4              10000            171599 ns/op
BenchmarkAll/BenchmarkZeroCopyElements_5000-4              10000            169485 ns/op
BenchmarkAll/BenchmarkZeroCopyElements_5500-4              10000            169657 ns/op
BenchmarkAll/BenchmarkZeroCopyElements_6000-4              10000            168274 ns/op
BenchmarkAll/BenchmarkZeroCopyElements_6500-4              10000            171372 ns/op
BenchmarkAll/BenchmarkZeroCopyElements_7000-4              10000            168484 ns/op
BenchmarkAll/BenchmarkZeroCopyElements_7500-4              10000            169056 ns/op
BenchmarkAll/BenchmarkZeroCopyElements_8000-4              10000            166486 ns/op
BenchmarkAll/BenchmarkZeroCopyElements_8500-4              10000            167760 ns/op
BenchmarkAll/BenchmarkZeroCopyElements_9000-4              10000            173118 ns/op
BenchmarkAll/BenchmarkZeroCopyElements_9500-4              10000            166797 ns/op
BenchmarkAll/BenchmarkZeroCopyElements_10000-4             10000            169281 ns/op
PASS
ok      github.com/nickpoorman/go-py-arrow-bridge       86.975s

License

(c) 2019 Nick Poorman. Licensed under the Apache License, Version 2.0.

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func CallPyFunc 

func CallPyFunc(obj *python3.PyObject, name string, args ...*python3.PyObject) *python3.PyObject

A helper for first fetching the function and then calling it

func GetFromType 

func GetFromType(t arrow.Type) (arrow.DataType, error)

GetFromType returns a arrow.DataType for a given arrow.Type

func GetIntAttr 

func GetIntAttr(obj *python3.PyObject, attr string) (int, bool)

func PyBufferToBytes 

func PyBufferToBytes(pyBuffer *python3.PyObject) ([]byte, error)

func PyBuffersGetBuffer 

func PyBuffersGetBuffer(pyBuffers *python3.PyObject, i int) (*memory.Buffer, error)

func PyBuffersToBuffers 

func PyBuffersToBuffers(pyBuffers *python3.PyObject) ([]*memory.Buffer, error)

func PyChunkGetBuffers 

func PyChunkGetBuffers(pyChunk *python3.PyObject) ([]*memory.Buffer, error)

func PyChunkGetLength 

func PyChunkGetLength(pyChunk *python3.PyObject) (int, error)

func PyChunkGetNullCount 

func PyChunkGetNullCount(pyChunk *python3.PyObject) (int, error)

func PyChunkGetOffset 

func PyChunkGetOffset(pyChunk *python3.PyObject) (int, error)

func PyChunkGetPyBuffers 

func PyChunkGetPyBuffers(pyChunk *python3.PyObject) (*python3.PyObject, error)

func PyChunkToChunk 

func PyChunkToChunk(pyChunk *python3.PyObject, dtype arrow.DataType) (array.Interface, error)

func PyChunkToData 

func PyChunkToData(pyChunk *python3.PyObject, dtype arrow.DataType) (*array.Data, error)

func PyChunkedGetPyChunks 

func PyChunkedGetPyChunks(pyChunked *python3.PyObject) (*python3.PyObject, error)

func PyChunkedToChunked 

func PyChunkedToChunked(pyChunked *python3.PyObject, dtype arrow.DataType) (*array.Chunked, error)

func PyChunkedToChunks 

func PyChunkedToChunks(pyChunked *python3.PyObject, dtype arrow.DataType) ([]array.Interface, error)

func PyChunksGetChunk 

func PyChunksGetChunk(pyChunks *python3.PyObject, i int, dtype arrow.DataType) (array.Interface, error)

func PyChunksGetPyChunk 

func PyChunksGetPyChunk(pyChunks *python3.PyObject, i int) (*python3.PyObject, error)

func PyColumnGetPyChunked 

func PyColumnGetPyChunked(pyColumn *python3.PyObject) (*python3.PyObject, error)

func PyColumnToChunkedWithField 

func PyColumnToChunkedWithField(pyColumn *python3.PyObject, field arrow.Field) (*array.Chunked, error)

func PyColumnToColumnWithField 

func PyColumnToColumnWithField(pyColumn *python3.PyObject, field arrow.Field) (*array.Column, error)

PyColumnToColumnWithField turns a PyColumn into a GoColumn

func PyDataTypeGetID 

func PyDataTypeGetID(pyDtype *python3.PyObject) (int, error)

func PyDataTypeToDataType 

func PyDataTypeToDataType(pyDtype *python3.PyObject) (arrow.DataType, error)

PyDataTypeToDataType returns the Go arrow DataType given the Python type.

func PyFieldToField 

func PyFieldToField(pyField *python3.PyObject) (*arrow.Field, error)

PyFieldToField given a Python field gets the Go Arrow field.

func PySchemaFromPyTable 

func PySchemaFromPyTable(pyTable *python3.PyObject) (*python3.PyObject, error)

PySchemaFromPyTable returns a pyarrow schema from a pyarrow Table.

func PySchemaToSchema 

func PySchemaToSchema(pySchema *python3.PyObject) (*arrow.Schema, error)

PySchemaToSchema given a Python schema gets the Go Arrow schema.

func PyTableGetPyColumn 

func PyTableGetPyColumn(pyTable *python3.PyObject, name string) (*python3.PyObject, error)

PyTableGetPyColumn returns the PyColumn given the name from the PyTable

func PyTableToColumns 

func PyTableToColumns(pyTable *python3.PyObject) (*arrow.Schema, []array.Column, error)

PyTableToColumns returns the records in the pyarrow table.

func PyTableToColumnsWithSchema 

func PyTableToColumnsWithSchema(pyTable *python3.PyObject, schema *arrow.Schema) ([]array.Column, error)

PyTableToColumns returns the columns in the pyarrow table.

func PyTableToTable 

func PyTableToTable(pyTable *python3.PyObject) (array.Table, error)

Types

This section is empty.

Source Files

View all Source files

Directories

Path Synopsis
cmd
example

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.