npyio: github.com/sbinet/npyio Index | Examples | Files

package npyio

import "github.com/sbinet/npyio"

Package npyio provides read/write access to files following the NumPy data file format:

https://docs.scipy.org/doc/numpy/neps/npy-format.html

Supported types

npyio supports r/w of scalars, arrays, slices and gonum/mat.Dense. Supported scalars are:

- bool,
- (u)int{8,16,32,64},
- float{32,64},
- complex{64,128}

Reading

Reading from a NumPy data file can be performed like so:

f, err := os.Open("data.npy")
var m mat.Dense
err = npyio.Read(f, &m)
fmt.Printf("data = %v\n", mat.Formatted(&m, mat.Prefix("       "))))

npyio can also read data directly into slices, arrays or scalars, provided the on-disk data type and the provided one match.

Example:

var data []float64
err = npyio.Read(f, &data)

var data uint64
err = npyio.Read(f, &data)

Writing

Writing into a NumPy data file can be done like so:

f, err := os.Create("data.npy")
var m mat.Dense = ...
err = npyio.Write(f, m)

Scalars, arrays and slices are also supported:

var data []float64 = ...
err = npyio.Write(f, data)

var data int64 = 42
err = npyio.Write(f, data)

var data [42]complex128 = ...
err = npyio.Write(f, data)

Index

Examples

Package Files

npyio.go reader.go writer.go

Variables

var (

    // ErrInvalidNumPyFormat is the error returned by NewReader when
    // the underlying io.Reader is not a valid or recognized NumPy data
    // file format.
    ErrInvalidNumPyFormat = errors.New("npyio: not a valid NumPy file format")

    // ErrTypeMismatch is the error returned by Reader when the on-disk
    // data type and the user provided one do NOT match.
    ErrTypeMismatch = errors.New("npyio: types don't match")

    // ErrInvalidType is the error returned by Reader and Writer when
    // confronted with a type that is not supported or can not be
    // reliably (de)serialized.
    ErrInvalidType = errors.New("npyio: invalid or unsupported type")

    // Magic header present at the start of a NumPy data file format.
    // See http://docs.scipy.org/doc/numpy-1.10.1/neps/npy-format.html
    Magic = [6]byte{'\x93', 'N', 'U', 'M', 'P', 'Y'}
)

func Read Uses

func Read(r io.Reader, ptr interface{}) error

Read reads the data from the r NumPy data file io.Reader, into the provided pointed at value ptr. Read returns an error if the on-disk data type and the one provided don't match.

If a *mat.Dense matrix is passed to Read, the numpy-array data is loaded into the Dense matrix, honouring Fortran/C-order and dimensions/shape parameters.

Only numpy-arrays with up to 2 dimensions are supported. Only numpy-arrays with elements convertible to float64 are supported.

Code:

m := mat.NewDense(2, 3, []float64{0, 1, 2, 3, 4, 5})
fmt.Printf("-- original data --\n")
fmt.Printf("data = %v\n", mat.Formatted(m, mat.Prefix("       ")))
buf := new(bytes.Buffer)

err := npyio.Write(buf, m)
if err != nil {
    log.Fatalf("error writing data: %v\n", err)
}

// modify original data
m.Set(0, 0, 6)

var data mat.Dense
err = npyio.Read(buf, &data)
if err != nil {
    log.Fatalf("error reading data: %v\n", err)
}

fmt.Printf("-- data read back --\n")
fmt.Printf("data = %v\n", mat.Formatted(&data, mat.Prefix("       ")))

fmt.Printf("-- modified original data --\n")
fmt.Printf("data = %v\n", mat.Formatted(m, mat.Prefix("       ")))

Output:

-- original data --
data = ⎡0  1  2⎤
       ⎣3  4  5⎦
-- data read back --
data = ⎡0  1  2⎤
       ⎣3  4  5⎦
-- modified original data --
data = ⎡6  1  2⎤
       ⎣3  4  5⎦

func TypeFrom Uses

func TypeFrom(dtype string) reflect.Type

TypeFrom returns the reflect.Type corresponding to the numpy-dtype string, if any.

func Write Uses

func Write(w io.Writer, val interface{}) error

Write writes 'val' into 'w' in the NumPy data format.

- if val is a scalar, it must be of a supported type (bools, (u)ints, floats and complexes)
- if val is a slice or array, it must be a slice/array of a supported type.
  the shape (len,) will be written out.
- if val is a mat.Dense, the correct shape will be transmitted. (ie: (nrows, ncols))

The data-array will always be written out in C-order (row-major).

Code:

m := mat.NewDense(2, 3, []float64{0, 1, 2, 3, 4, 5})
fmt.Printf("-- original data --\n")
fmt.Printf("data = %v\n", mat.Formatted(m, mat.Prefix("       ")))
buf := new(bytes.Buffer)

err := npyio.Write(buf, m)
if err != nil {
    log.Fatalf("error writing data: %v\n", err)
}

// modify original data
m.Set(0, 0, 6)

var data mat.Dense
err = npyio.Read(buf, &data)
if err != nil {
    log.Fatalf("error reading data: %v\n", err)
}

fmt.Printf("-- data read back --\n")
fmt.Printf("data = %v\n", mat.Formatted(&data, mat.Prefix("       ")))

fmt.Printf("-- modified original data --\n")
fmt.Printf("data = %v\n", mat.Formatted(m, mat.Prefix("       ")))

Output:

-- original data --
data = ⎡0  1  2⎤
       ⎣3  4  5⎦
-- data read back --
data = ⎡0  1  2⎤
       ⎣3  4  5⎦
-- modified original data --
data = ⎡6  1  2⎤
       ⎣3  4  5⎦
type Header struct {
    Major byte // data file major version
    Minor byte // data file minor version
    Descr struct {
        Type    string // data type of array elements ('<i8', '<f4', ...)
        Fortran bool   // whether the array data is stored in Fortran-order (col-major)
        Shape   []int  // array shape (e.g. [2,3] a 2-rows, 3-cols array
    }
}

Header describes the data content of a NumPy data file.

func (Header) String Uses

func (h Header) String() string

type Reader Uses

type Reader struct {
    Header Header
    // contains filtered or unexported fields
}

Reader reads data from a NumPy data file.

func NewReader Uses

func NewReader(r io.Reader) (*Reader, error)

NewReader creates a new NumPy data file format reader.

func (*Reader) Read Uses

func (r *Reader) Read(ptr interface{}) error

Read reads the numpy-array data from the underlying NumPy file. Read returns an error if the on-disk data type and the provided one don't match.

See npyio.Read() for documentation.

Package npyio imports 13 packages (graph) and is imported by 3 packages. Updated 2019-03-09. Refresh now. Tools for package owners.