Documentation
¶
Overview ¶
Package reader provides the interfaces, embedded structs, and implementing code for normalizing the reading a stream of bytes from compressed files. This package supports the gzip and snappy compression algorithms. No compression is usually identified as "none". This package is used by the go-stream package.
Usage ¶
You can import reader as a package into your own Go project or use the command line interface.
import (
"github.com/spatialcurrent/go-reader/reader"
)
r, err := reader.OpenFile("data-for-2018.sz", "snappy")
if err != nil {
panic(err)
}
for {
b, err := input_reader.ReadBytes([]byte("\n")[0])
if err != nil {
if err != io.EOF {
fmt.Println(errors.Wrap(err, "Error reading bytes from file"))
os.Exit(1)
}
}
if len(b) > 0 {
fmt.Println(string(b))
}
if err != nil && err == io.EOF {
break
}
}
See the github.com/go-reader/cmd/go-reader package for a command line tool for testing DFL expressions.
Projects ¶
go-reader is used by the go-stream project.
Index ¶
- Variables
- func OpenHTTPFile(uri string, alg string, cache bool) (ByteReadCloser, *Metadata, error)
- func OpenResource(uri string, alg string, buffer_size int, cache bool, s3_client *s3.S3) (ByteReadCloser, *Metadata, error)
- func OpenS3Object(bucket string, key string, alg string, cache bool, s3_client *s3.S3) (ByteReadCloser, *Metadata, error)
- func SplitUri(uri string) (string, string)
- type ByteReadCloser
- func Bytes(b []byte) (ByteReadCloser, error)
- func Bzip2File(path string, cache bool, buffer_size int) (ByteReadCloser, error)
- func File(path string, cache bool, buffer_size int) (ByteReadCloser, error)
- func GzipBytes(b []byte) (ByteReadCloser, error)
- func GzipFile(path string, cache bool, buffer_size int) (ByteReadCloser, error)
- func OpenBytes(b []byte, alg string) (ByteReadCloser, error)
- func OpenFile(uri string, alg string, cache bool, buffer_size int) (ByteReadCloser, error)
- func OpenStdin(alg string, cache bool) (ByteReadCloser, error)
- func SnappyBytes(b []byte) (ByteReadCloser, error)
- func SnappyFile(path string, cache bool, buffer_size int) (ByteReadCloser, error)
- func ZipFile(path string, cache bool, buffer_size int) (ByteReadCloser, error)
- type ByteReader
- type Cache
- type ErrorNotImplemented
- type Metadata
- type Reader
- func (r *Reader) Close() error
- func (r *Reader) Read(p []byte) (n int, err error)
- func (r *Reader) ReadAll() ([]byte, error)
- func (r *Reader) ReadAt(i int) (byte, error)
- func (r *Reader) ReadByte() (byte, error)
- func (r *Reader) ReadBytes(delim byte) ([]byte, error)
- func (r *Reader) ReadFirst() (byte, error)
- func (r *Reader) ReadRange(start int, end int) ([]byte, error)
Constants ¶
This section is empty.
Variables ¶
var VERSION = "0.0.2"
Functions ¶
func OpenHTTPFile ¶
OpenHTTPFile returns a ByteReadCloser for an object for a web address alg may be "bzip2", "gzip", "snappy", or "".
func OpenResource ¶
Types ¶
type ByteReadCloser ¶
type ByteReadCloser interface {
ByteReader
io.Closer
ReadAt(i int) (byte, error)
ReadAll() ([]byte, error)
ReadRange(start int, end int) ([]byte, error)
}
ByteReader is an interface that extends io.Reader, io.ByteReader, and adds a range function. ByteReader provides functions for reading bytes.
func Bytes ¶
func Bytes(b []byte) (ByteReadCloser, error)
Bytes returns a reader for reading the bytes from an input array, and an error if any.
func Bzip2File ¶
func Bzip2File(path string, cache bool, buffer_size int) (ByteReadCloser, error)
Bzip2File returns a reader for reading bytes from a bzip2-compressed file Wraps the "compress/gzip" package.
func File ¶
func File(path string, cache bool, buffer_size int) (ByteReadCloser, error)
File returns a ByteReader for reading bytes without any transformation from a file, and an error if any.
func GzipBytes ¶
func GzipBytes(b []byte) (ByteReadCloser, error)
GzipBytes returns a reader for reading gzip bytes from an input array. Wraps the "compress/gzip" package.
func GzipFile ¶
func GzipFile(path string, cache bool, buffer_size int) (ByteReadCloser, error)
GzipFile returns a reader for reading bytes from a gzip-compressed file Wraps the "compress/gzip" package.
func OpenBytes ¶
func OpenBytes(b []byte, alg string) (ByteReadCloser, error)
OpenBytes returns a ByteReader for a byte array with a given compression. alg may be "snappy", "gzip", or "none."
func OpenFile ¶
OpenFile returns a ByteReader for a file with a given compression. alg may be "snappy", "gzip", or "none."
func OpenStdin ¶
func OpenStdin(alg string, cache bool) (ByteReadCloser, error)
OpenStdin returns a ByteReadCloser for a file with a given compression. alg may be "bzip2", "gzip", "snappy", or "".
func SnappyBytes ¶
func SnappyBytes(b []byte) (ByteReadCloser, error)
SnappyBytes returns a reader for an input of snappy-compressed bytes, and an error if any.
func SnappyFile ¶
func SnappyFile(path string, cache bool, buffer_size int) (ByteReadCloser, error)
SnappyFile returns a reader for a snappy-compressed file, and an error if any.
type ByteReader ¶
ByteReader is an interface that extends io.Reader and io.ByteReader. ByteReader provides functions for reading bytes.
type Cache ¶
func (*Cache) Read ¶
Read reads a maximum len(p) bytes from the reader and returns an error, if any.
func (*Cache) ReadAll ¶
ReadAll reads all content from the underlying reader and returns the content
type ErrorNotImplemented ¶
func (*ErrorNotImplemented) Error ¶
func (e *ErrorNotImplemented) Error() string
type Metadata ¶
type Metadata struct {
ContentType string
LastModified *time.Time
ContentLength int64
Header map[string][]string
}
func NewMetadataFromHeader ¶
func NewMetadataFromS3 ¶
func NewMetadataFromS3(output *s3.GetObjectOutput) *Metadata
type Reader ¶
type Reader struct {
Reader ByteReader // the instance of ByteReader used for reading bytes
Closer io.Closer // Used for closing readers with footer metadata, e.g., gzip. Not always needed, e.g., snappy
File *os.File // underlying file, if any
}
Reader is a struct for normalizing reading of bytes from files with arbitrary compression and for closing underlying resources. Reader implements the ByteReader interface by wrapping around a subordinate ByteReader.
func (*Reader) Read ¶
Read reads a maximum len(p) bytes from the reader and returns an error, if any.
func (*Reader) ReadBytes ¶
Read returns all bytes up to an including the first occurrence of the delimiter "delim" and an error, if any.
Source Files