rknnlite

package module
v0.0.0-...-f5a5e8c Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 11, 2024 License: Apache-2.0 Imports: 7 Imported by: 0

README

go-rknnlite

go-rknnlite provides Go language bindings for the RKNN Toolkit2 C API interface. It aims to provide lite bindings in the spirit of the closed source Python lite bindings used for running AI Inference models on the Rockchip NPU via the RKNN software stack.

These bindings have only been tested on the RK3588 (specifically the Radxa Rock Pi 5B) but should work on other RK3588 based SBC's. It should also work with other models in the RK35xx series supported by the RKNN Toolkit2.

Usage

go get github.com/swdee/go-rknnlite

Dependencies

The rknn-toolkit2 must be installed on your system with C header files available in the system path, eg: /usr/include/rknn_api.h.

Refer to the official documentation on how to install this on your system as it will vary based on OS and SBC vendor.

Rock Pi 5B

My usage was on the Radxa Rock Pi 5B running the official Debian 11 OS image.

I used the prebuilt RKNN libraries built here.

wget https://github.com/radxa-pkg/rknn2/releases/download/1.6.0-2/rknpu2-rk3588_1.6.0-2_arm64.deb
apt install ./rknpu2-rk3588_1.6.0-2_arm64.deb 

Examples

See the example directory.

Pooled Runtimes

Running multiple Runtimes in a Pool allows you to take advantage of all three NPU cores. For our usage of an EfficentNet-Lite0 model, a single runtime has an inference speed of 7.9ms per image, however running a Pool of 9 runtimes brings the average inference speed down to 1.65ms per image.

See the Pool example.

Reference Material

Documentation

Overview

go-rknnlite provides Go language bindings for the RKNN Toolkit2 C API interface. It aims to provide lite bindings in the spirit of the closed source Python lite bindings used for running AI Inference models on the Rockchip NPU via the RKNN software stack.

These bindings have only been tested on the RK3588 (specifically the Radxa Rock Pi 5B) but should work on other RK3588 based SBC's. It should also work with other models in the RK35xx series supported by the RKNN Toolkit2.

See example code and usage in the examples subdirectory.

Index

Constants

View Source
const MAX_TOP_NUM = 20

Variables

This section is empty.

Functions

func GetTop

func GetTop(pfProb []float32, pfMaxProb []float32, pMaxClass []int32,
	outputCount int32, topNum int32) int

GetTop takes outputs and produces a top list of matches by probability

Types

type AttrMaxDimensions

type AttrMaxDimensions int

AttrMaxDimensions are the maximum dimensions for an attribute in a tensor

maximum field lengths of attributes in a tensor

type CoreMask

type CoreMask int

CoreMask wraps C.rknn_core_mask

rknn_core_mask values used to target which cores on the NPU the model is run on. The rk3588 has three cores, auto will pick an idle core to run the model on, whilst the others specify the specific core or combined number of cores to run. For multi-core modes the following ops have better acceleration: Conv, DepthwiseConvolution, Add, Concat, Relu, Clip, Relu6, ThresholdedRelu, Prelu, and LeakyRelu. Other type of ops will fallback to Core0 to continue running

type ErrorCodes

type ErrorCodes int

ErrorCodes

error code values returned by the C API

func (ErrorCodes) String

func (e ErrorCodes) String() string

String returns a readable description of the error code

type IONumber

type IONumber struct {
	NumberInput  uint32
	NumberOutput uint32
}

IONumber represents the C.rknn_input_output_num struct

type Input

type Input struct {
	// Index is the input index
	Index uint32
	// Buf is the gocv Mat input
	Buf unsafe.Pointer
	// Size is the number of bytes of Buf
	Size uint32
	// Passthrough defines the mode, if True the buf data is passed directly to
	// the input node of the rknn model without any conversion.  If False the
	// buf data is converted into an input consistent with the model according
	// to the following type and fmt
	PassThrough bool
	// Type is the data type of Buf. This is a required parameter if Passthrough
	// is False
	Type TensorType
	// Fmt is the data format of Buf.  This is a required parameter if Passthrough
	// is False
	Fmt TensorFormat
}

Input represents the C.rknn_input struct and defines the Input used for inference

type Output

type Output struct {
	WantFloat  uint8     // want transfer output data to float
	IsPrealloc uint8     // whether buf is pre-allocated
	Index      uint32    // the output index
	Buf        []float32 // the output buf
	Size       uint32    // the size of output buf
}

Output wraps C.rknn_output

type Pool

type Pool struct {
	// contains filtered or unexported fields
}

Pool is a simple runtime pool to open multiple of the same Model across all NPU cores

func NewPool

func NewPool(size int, modelFile string) (*Pool, error)

NewPool creates a new runtime pool

func (*Pool) Close

func (p *Pool) Close()

Close the pool and all runtimes in it

func (*Pool) Get

func (p *Pool) Get() *Runtime

Gets a runtime from the pool

func (*Pool) Return

func (p *Pool) Return(runtime *Runtime)

Return a runtime to the pool

type Probability

type Probability struct {
	LabelIndex  int32
	Probability float32
}

func GetTop5

func GetTop5(outputs []Output) []Probability

GetTop5 outputs the Top5 matches in the model, with left column as label index and right column the match probability. The results are returned in the Probability slice in descending order from top match.

type Runtime

type Runtime struct {
	// contains filtered or unexported fields
}

Runtime defines the RKNN run time instance

func NewRuntime

func NewRuntime(modelFile string, core CoreMask) (*Runtime, error)

NewRuntime returns a RKNN run time instance. Provide the full path and filename of the RKNN compiled model file to run.

func (*Runtime) Close

func (r *Runtime) Close() error

Close wraps C.rknn_destroy which unloads the RKNN model from the runtime and destroys the context releasing all C resources

func (*Runtime) GetOutputs

func (r *Runtime) GetOutputs(nOutputs uint32) ([]Output, error)

GetOutputs returns the Output results

func (*Runtime) Inference

func (r *Runtime) Inference(mats []gocv.Mat) ([]Output, error)

Inference runs the model inference on the given inputs

func (*Runtime) QueryInputTensors

func (r *Runtime) QueryInputTensors() ([]TensorAttr, error)

QueryInputTensors gets the model Input Tensor attributes

func (*Runtime) QueryModelIONumber

func (r *Runtime) QueryModelIONumber() (ioNum IONumber, err error)

QueryModelIONumber queries the number of Input and Output tensors of the model

func (*Runtime) QueryOutputTensors

func (r *Runtime) QueryOutputTensors() ([]TensorAttr, error)

QueryOutputTensors gets the model Output Tensor attributes

func (*Runtime) RunModel

func (r *Runtime) RunModel() error

RunModel wraps C.rknn_run

func (*Runtime) SDKVersion

func (r *Runtime) SDKVersion() (SDKVersion, error)

SDKVersion returns the RKNN API and Driver versions

func (*Runtime) SetInputs

func (r *Runtime) SetInputs(inputs []Input) error

setInputs wraps C.rknn_inputs_set

type SDKVersion

type SDKVersion struct {
	DriverVersion string
	APIVersion    string
}

SDKVersion represents the C.rknn_sdk_version struct

type TensorAttr

type TensorAttr struct {
	Index          uint32
	NDims          uint32
	Dims           [AttrMaxDimension]uint32
	Name           string
	NElems         uint32
	Size           uint32
	Fmt            TensorFormat
	Type           TensorType
	QntType        TensorQntType
	FL             int8
	ZP             int32
	Scale          float32
	WStride        uint32
	SizeWithStride uint32
	PassThrough    bool
	HStride        uint32
}

TensorAttr represents the C.rknn_tensor_attr structure

func (TensorAttr) String

func (a TensorAttr) String() string

String returns the TensorAttr's attributes formatted as a string

type TensorFormat

type TensorFormat int

TensorFormat wraps C.rknn_tensor_format

const (
	TensorNCHW      TensorFormat = C.RKNN_TENSOR_NCHW
	TensorNHWC      TensorFormat = C.RKNN_TENSOR_NHWC
	TensorNC1HWC2   TensorFormat = C.RKNN_TENSOR_NC1HWC2
	TensorUndefined TensorFormat = C.RKNN_TENSOR_UNDEFINED
)

func (TensorFormat) String

func (t TensorFormat) String() string

String returns a readable description of the TensorFormat

type TensorQntType

type TensorQntType int

TensorQntType wraps C.rknn_tensor_qnt_type

func (TensorQntType) String

func (t TensorQntType) String() string

String returns a readable description of the TensorQntType

type TensorType

type TensorType int

TensorType wraps C.rknn_tensor_type

func (TensorType) String

func (t TensorType) String() string

String returns a readable description of the TensorType

Directories

Path Synopsis
example
mobilenet
Example code showing how to perform inferencing using a MobileNetv1 model.
Example code showing how to perform inferencing using a MobileNetv1 model.
pool
Running multiple Runtimes in a Pool allows you to take advantage of all three NPU cores to significantly reduce average inferencing time.
Running multiple Runtimes in a Pool allows you to take advantage of all three NPU cores to significantly reduce average inferencing time.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL