go-reader: github.com/spatialcurrent/go-reader/reader Index | Files

package reader

import "github.com/spatialcurrent/go-reader/reader"

Package reader provides the interfaces, embedded structs, and implementing code for normalizing the reading a stream of bytes from compressed files. This package supports the gzip and snappy compression algorithms. No compression is usually identified as "none". This package is used by the go-stream package.

- https://godoc.org/github.com/spatialcurrent/go-stream/stream

Usage

You can import reader as a package into your own Go project or use the command line interface.

import (
  "github.com/spatialcurrent/go-reader/reader"
)

r, err := reader.OpenFile("data-for-2018.sz", "snappy")
if err != nil {
  panic(err)
}
for {
  b, err := input_reader.ReadBytes([]byte("\n")[0])
  if err != nil {
    if err != io.EOF {
      fmt.Println(errors.Wrap(err, "Error reading bytes from file"))
      os.Exit(1)
    }
  }
  if len(b) > 0 {
    fmt.Println(string(b))
  }
  if err != nil && err == io.EOF {
    break
  }
}

See the github.com/go-reader/cmd/go-reader package for a command line tool for testing DFL expressions.

- https://godoc.org/github.com/spatialcurrent/go-reader/reader

Projects

go-reader is used by the go-stream project.

- https://godoc.org/github.com/spatialcurrent/go-stream/stream

Index

Package Files

BZip2File.go ByteReadCloser.go ByteReader.go Bytes.go Cache.go ErrorNotImplemented.go File.go GzipBytes.go GzipFile.go Metadata.go OpenBytes.go OpenFile.go OpenHTTPFile.go OpenResource.go OpenS3Object.go OpenStdin.go Reader.go SnappyBytes.go SnappyFile.go SplitUri.go Version.go ZipFile.go

Variables

var VERSION = "0.0.2"

func OpenHTTPFile Uses

func OpenHTTPFile(uri string, alg string, cache bool) (ByteReadCloser, *Metadata, error)

OpenHTTPFile returns a ByteReadCloser for an object for a web address alg may be "bzip2", "gzip", "snappy", or "".

- https://golang.org/pkg/compress/bzip2/
- https://golang.org/pkg/compress/gzip/
- https://godoc.org/github.com/golang/snappy

func OpenResource Uses

func OpenResource(uri string, alg string, buffer_size int, cache bool, s3_client *s3.S3) (ByteReadCloser, *Metadata, error)

func OpenS3Object Uses

func OpenS3Object(bucket string, key string, alg string, cache bool, s3_client *s3.S3) (ByteReadCloser, *Metadata, error)

OpenS3Object returns a ByteReadCloser for an object in AWS S3. alg may be "bzip2", "gzip", "snappy", or "".

- https://golang.org/pkg/compress/bzip2/
- https://golang.org/pkg/compress/gzip/
- https://godoc.org/github.com/golang/snappy

func SplitUri Uses

func SplitUri(uri string) (string, string)

SplitUri splits a uri string into a scheme and path. If no scheme is specified, then returns "" as the scheme and the full path.

type ByteReadCloser Uses

type ByteReadCloser interface {
    ByteReader
    io.Closer
    ReadAt(i int) (byte, error)
    ReadAll() ([]byte, error)
    ReadRange(start int, end int) ([]byte, error)
}

ByteReader is an interface that extends io.Reader, io.ByteReader, and adds a range function. ByteReader provides functions for reading bytes.

func Bytes Uses

func Bytes(b []byte) (ByteReadCloser, error)

Bytes returns a reader for reading the bytes from an input array, and an error if any.

func Bzip2File Uses

func Bzip2File(path string, cache bool, buffer_size int) (ByteReadCloser, error)

Bzip2File returns a reader for reading bytes from a bzip2-compressed file Wraps the "compress/gzip" package.

- https://golang.org/pkg/compress/gzip/

func File Uses

func File(path string, cache bool, buffer_size int) (ByteReadCloser, error)

File returns a ByteReader for reading bytes without any transformation from a file, and an error if any.

func GzipBytes Uses

func GzipBytes(b []byte) (ByteReadCloser, error)

GzipBytes returns a reader for reading gzip bytes from an input array. Wraps the "compress/gzip" package.

- https://golang.org/pkg/compress/gzip/

func GzipFile Uses

func GzipFile(path string, cache bool, buffer_size int) (ByteReadCloser, error)

GzipFile returns a reader for reading bytes from a gzip-compressed file Wraps the "compress/gzip" package.

- https://golang.org/pkg/compress/gzip/

func OpenBytes Uses

func OpenBytes(b []byte, alg string) (ByteReadCloser, error)

OpenBytes returns a ByteReader for a byte array with a given compression. alg may be "snappy", "gzip", or "none."

- https://golang.org/pkg/compress/gzip/
- https://godoc.org/github.com/golang/snappy

func OpenFile Uses

func OpenFile(uri string, alg string, cache bool, buffer_size int) (ByteReadCloser, error)

OpenFile returns a ByteReader for a file with a given compression. alg may be "snappy", "gzip", or "none."

- https://golang.org/pkg/compress/gzip/
- https://godoc.org/github.com/golang/snappy

func OpenStdin Uses

func OpenStdin(alg string, cache bool) (ByteReadCloser, error)

OpenStdin returns a ByteReadCloser for a file with a given compression. alg may be "bzip2", "gzip", "snappy", or "".

- https://golang.org/pkg/compress/bzip2/
- https://golang.org/pkg/compress/gzip/
- https://godoc.org/github.com/golang/snappy

func SnappyBytes Uses

func SnappyBytes(b []byte) (ByteReadCloser, error)

SnappyBytes returns a reader for an input of snappy-compressed bytes, and an error if any.

- https://godoc.org/github.com/golang/snappy

func SnappyFile Uses

func SnappyFile(path string, cache bool, buffer_size int) (ByteReadCloser, error)

SnappyFile returns a reader for a snappy-compressed file, and an error if any.

- https://godoc.org/github.com/golang/snappy

func ZipFile Uses

func ZipFile(path string, cache bool, buffer_size int) (ByteReadCloser, error)

ZipFile returns a ByteReadCloser for reading bytes from a zip-compressed file Wraps the "archive/zip" package.

- https://golang.org/pkg/archive/zip/

type ByteReader Uses

type ByteReader interface {
    io.Reader
    io.ByteReader
    ReadBytes(delim byte) ([]byte, error)
}

ByteReader is an interface that extends io.Reader and io.ByteReader. ByteReader provides functions for reading bytes.

type Cache Uses

type Cache struct {
    *Reader
    Cursor   int
    Complete *bool
    Content  *[]byte
}

func NewCache Uses

func NewCache(r *Reader) *Cache

func NewCacheWithContent Uses

func NewCacheWithContent(r *Reader, c *[]byte, i int) *Cache

func (*Cache) Read Uses

func (c *Cache) Read(p []byte) (n int, err error)

Read reads a maximum len(p) bytes from the reader and returns an error, if any.

func (*Cache) ReadAll Uses

func (c *Cache) ReadAll() ([]byte, error)

ReadAll reads all content from the underlying reader and returns the content

func (*Cache) ReadAt Uses

func (c *Cache) ReadAt(i int) (byte, error)

ReadAt returns the byte at the index given by i

func (*Cache) ReadFirst Uses

func (c *Cache) ReadFirst() (byte, error)

ReadFirst returns the first byte stating at the cursor.

func (*Cache) ReadRange Uses

func (c *Cache) ReadRange(start int, end int) ([]byte, error)

ReadRange reads a range of btes from the cache. End points to the index of the last byte to read, so [start:end+1]

type ErrorNotImplemented Uses

type ErrorNotImplemented struct {
    Function string
    Object   string
}

func (*ErrorNotImplemented) Error Uses

func (e *ErrorNotImplemented) Error() string

type Metadata Uses

type Metadata struct {
    ContentType   string
    LastModified  *time.Time
    ContentLength int64
    Header        map[string][]string
}

func NewMetadataFromHeader Uses

func NewMetadataFromHeader(header map[string][]string) *Metadata

func NewMetadataFromS3 Uses

func NewMetadataFromS3(output *s3.GetObjectOutput) *Metadata

type Reader Uses

type Reader struct {
    Reader ByteReader // the instance of ByteReader used for reading bytes
    Closer io.Closer  // Used for closing readers with footer metadata, e.g., gzip.  Not always needed, e.g., snappy
    File   *os.File   // underlying file, if any
}

Reader is a struct for normalizing reading of bytes from files with arbitrary compression and for closing underlying resources. Reader implements the ByteReader interface by wrapping around a subordinate ByteReader.

func (*Reader) Close Uses

func (r *Reader) Close() error

Close closes the Closer and the underlying *os.File if not nil.

func (*Reader) Read Uses

func (r *Reader) Read(p []byte) (n int, err error)

Read reads a maximum len(p) bytes from the reader and returns an error, if any.

func (*Reader) ReadAll Uses

func (r *Reader) ReadAll() ([]byte, error)

ReadAll is not implemented by Reader

func (*Reader) ReadAt Uses

func (r *Reader) ReadAt(i int) (byte, error)

ReadAt is not implemented by Reader

func (*Reader) ReadByte Uses

func (r *Reader) ReadByte() (byte, error)

ReadByte returns a single byte from the underlying reader.

func (*Reader) ReadBytes Uses

func (r *Reader) ReadBytes(delim byte) ([]byte, error)

Read returns all bytes up to an including the first occurrence of the delimiter "delim" and an error, if any.

func (*Reader) ReadFirst Uses

func (r *Reader) ReadFirst() (byte, error)

ReadFirst is not implemented by Reader

func (*Reader) ReadRange Uses

func (r *Reader) ReadRange(start int, end int) ([]byte, error)

ReadRange is not implemented by Reader

Package reader imports 17 packages (graph) and is imported by 1 packages. Updated 2018-10-06. Refresh now. Tools for package owners.