grw

package
v0.0.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 14, 2021 License: MIT Imports: 31 Imported by: 2

Documentation

Overview

Package grw provides the interfaces, embedded structs, and implementing code for normalizing the reading/writing of a stream of bytes from archive/compressed files. This package supports the bzip2, gzip, snappy, and zip archive/compression algorithms. No compression can be identified as "none" or a blank string. This package is used by the go-stream package.

Usage

You can import reader as a package into your own Go project or use the command line interface. 

import (
  "github.com/spatialcurrent/go-reader-writer/grw"
)

r, err := grw.ReadFromFilePath("data-for-2018.sz", "snappy", false, 4096)
if err != nil {
  panic(err)
}
for {
  b, err := input_reader.ReadBytes([]byte("\n")[0])
  if err != nil {
    if err != io.EOF {
      fmt.Println(fmt.Errorf("error reading bytes from file: %w", err))
      os.Exit(1)
    }
  }
  if len(b) > 0 {
    fmt.Println(string(b))
  }
  if err != nil && err == io.EOF {
    break
  }
}

See the github.com/go-reader/cmd/go-reader package for a command line tool for testing DFL expressions.

Projects go-reader-writer is used by the railgun project and go-stream

Index

Constants

This section is empty.

Variables

View Source 
var (
	NoDict   = []byte{} // no dictionary
	NoBuffer = 0
)
View Source 
var (
	Algorithms = []string{
		alg.AlgorithmBzip2,
		alg.AlgorithmFlate,
		alg.AlgorithmGzip,
		alg.AlgorithmNone,
		alg.AlgorithmSnappy,
		alg.AlgorithmZip,
		alg.AlgorithmZlib,
	}
)
View Source 
var (
	DefaultBufferSize = 4096
)
View Source 
var (
	ErrPathMissing = errors.New("path is missing")
)

Functions

func CloseWriters added in v0.0.3 

func CloseWriters(input *CloseWritersInput) error

CloseWriters closes a map of writers. If flush is true, will flush each writer before closing it.

func ExpandOpenAndStat 

func ExpandOpenAndStat(path string) (*os.File, os.FileInfo, error)

ExpandOpenAndStat expands a path, returns *os.File and os.FileInfo for a given file path and an error, if any.

func UploadS3Object 

func UploadS3Object(input *UploadS3ObjectInput) error

UploadS3Object uploads an object to S3. alg may be "bzip2", "gzip", "snappy", "zip", or "".

func WrapReader 

func WrapReader(r io.ReadCloser, alg string, dict []byte, bufferSize int) (io.ReadCloser, error)

func WrapWriter added in v0.0.3 

func WrapWriter(w io.WriteCloser, alg string, dict []byte, bufferSize int) (io.WriteCloser, error)

WrapWriter wraps the given writer with a buffer and the given compression. alg is the algorithm. dict is the initial dictionary (if the algorithm uses one).

func WriteAllAndClose added in v0.0.3 

func WriteAllAndClose(input *WriteAllAndCloseInput) error

WriteAllAndClose writes the bytes to the resource indicated by the uri given, flushes, and closes the resource.

func WriteBuffers 

func WriteBuffers(input *WriteBuffersInput) error

WriteBuffers writes a map of buffers to the resources defined by the keys. alg is the compression algorithm. If the buffer already includes compressed data, then use "" or "none" as alg. If append is true, then append to existing files. If mkdirs is true, then parent directories are created on-demand.

func WriteFile added in v0.0.3 

func WriteFile(input *WriteFileInput) (io.WriteCloser, error)

WriteFile returns a Writer for writing to a local file

func WriteToFileSystem 

func WriteToFileSystem(input *WriteToFileSystemInput) (io.WriteCloser, error)

WriteToFileSystem returns a ByteWriteCloser for a file with a given compression. alg may be "snappy", "gzip", or "none."

Types

type CloseWritersInput added in v0.0.3 

type CloseWritersInput struct {
	Writers map[string]io.Writer
	Flush   bool
}

type Closer 

type Closer struct {
	Closers []io.Closer // underlying list of io.Closer
}

Closer is a helper struct for closing a sequential list of closers associated with a resource. This is used for flushing the footer for a given file compression alogrithm before closing the underlying file.

func (*Closer) Close 

func (c *Closer) Close() error

Close closes all the underlying io.Closer sequentially.

type ErrFunctionNotImplemented 

type ErrFunctionNotImplemented struct {
	Function string
	Object   string
}

func (*ErrFunctionNotImplemented) Error 

func (e *ErrFunctionNotImplemented) Error() string

type ErrReaderNotImplemented 

type ErrReaderNotImplemented struct {
	Algorithm string
}

func (*ErrReaderNotImplemented) Error 

func (e *ErrReaderNotImplemented) Error() string

type ErrWriterNotImplemented 

type ErrWriterNotImplemented struct {
	Algorithm string
}

func (*ErrWriterNotImplemented) Error 

func (e *ErrWriterNotImplemented) Error() string

type FunctionWriteCloser added in v0.0.3 

type FunctionWriteCloser struct {
	Writer func(p []byte) (n int, err error)
	Closer func() error
}

func (*FunctionWriteCloser) Close added in v0.0.3 

func (w *FunctionWriteCloser) Close() error

func (*FunctionWriteCloser) Write added in v0.0.3 

func (w *FunctionWriteCloser) Write(p []byte) (n int, err error)

type Metadata 

type Metadata struct {
	ContentType   string
	LastModified  *time.Time
	ContentLength int64
	Header        map[string][]string
}

func NewMetadataFromHeader 

func NewMetadataFromHeader(header map[string][]string) *Metadata

func NewMetadataFromS3 

func NewMetadataFromS3(output *s3.GetObjectOutput) *Metadata

type ReadFromResourceInput added in v0.0.3 

type ReadFromResourceInput struct {
	URI        string       // uri to read from
	Alg        string       // compression algorithm
	Dict       []byte       // compression dictionary
	BufferSize int          // input reader buffer size
	S3Client   *s3.S3       // AWS S3 Client
	SSHClient  *ssh.Client  // SSH Client
	SFTPClient *sftp.Client // SFTP Client
	Password   string       // password
	PrivateKey []byte       // private key
}

type ReadFromResourceOutput added in v0.0.3 

type ReadFromResourceOutput struct {
	Reader   io.ReadCloser
	Metadata *Metadata
}

func ReadFromResource 

func ReadFromResource(input *ReadFromResourceInput) (*ReadFromResourceOutput, error)

type Reader 

type Reader struct {
	Reader io.ReadCloser // the instance of ByteReader used for reading bytes
}

Reader is a struct for normalizing reading of bytes from files with arbitrary compression and for closing underlying resources. Reader implements the ByteReader interface by wrapping around a subordinate ByteReader.

func (*Reader) Close 

func (r *Reader) Close() error

func (*Reader) Read 

func (r *Reader) Read(p []byte) (n int, err error)

Read reads a maximum len(p) bytes from the reader and returns an error, if any.

func (*Reader) ReadAll 

func (r *Reader) ReadAll() ([]byte, error)

ReadAll is not implemented by Reader

func (*Reader) ReadAllAndClose 

func (r *Reader) ReadAllAndClose() ([]byte, error)

ReadAllAndClose reads all the bytes from the underlying reader and attempts to close the reader, even if there was an error reading.

func (*Reader) ReadAt 

func (r *Reader) ReadAt(p []byte, off int64) (n int, err error)

func (*Reader) ReadByte 

func (r *Reader) ReadByte() (byte, error)

ReadByte returns a single byte from the underlying reader.

func (*Reader) ReadBytes 

func (r *Reader) ReadBytes(delim byte) ([]byte, error)

ReadBytes returns all bytes up to an including the first occurrence of the delimiter "delim" and an error, if any.

func (*Reader) ReadFirst 

func (r *Reader) ReadFirst() (byte, error)

ReadFirst is not implemented by Reader

func (*Reader) ReadRange 

func (r *Reader) ReadRange(start int, end int) ([]byte, error)

ReadRange is not implemented by Reader

func (*Reader) ReadString 

func (r *Reader) ReadString(delim byte) (string, error)

ReadString returns a string of all the bytes to an including the first occurrence of the delimiter "delim" and an error, if any.

type UploadS3ObjectInput added in v0.0.3 

type UploadS3ObjectInput struct {
	ACL    string
	Bucket string
	Key    string
	Object io.Reader
	Client *s3.S3
}

type WriteAllAndCloseInput added in v0.0.3 

type WriteAllAndCloseInput struct {
	ACL        string // ACL for objects written to AWS S3
	BufferSize int    // buffer size
	Bytes      []byte // the content to write
	URI        string // uri to write to
	Alg        string // compression algorithm
	Dict       []byte // compression dictionary
	Append     bool   // append to output resource
	Parents    bool   // automatically create parent directories as necessary
	S3Client   *s3.S3 // AWS S3 Client
}

WriteAllAndCloseInput contains the input parameters for WriteAllAndClose.

type WriteBuffersInput 

type WriteBuffersInput struct {
	Buffers    map[string]io.Buffer
	Algorithm  string
	Dictionary []byte
	Overwrite  bool
	Append     bool
	Mkdirs     bool
	S3Client   *s3.S3
}

type WriteFileInput added in v0.0.3 

type WriteFileInput struct {
	Path       string // required field
	BufferSize int    // if zero, then no buffer is used.
	Mode       uint32 // defaults to 0600
}

WriteFileInput holds the input for the WriteFile method

type WriteToFileSystemInput added in v0.0.3 

type WriteToFileSystemInput struct {
	Alg        string // compression algorithm
	BufferSize int    // buffer size
	Dict       []byte // compression dictionary
	Flag       int    // flag for file descriptor
	Mode       uint32 // mode of the output file
	Path       string // path to write to
	Parents    bool   // automatically create parent directories as necessary
}

WriteToFileSystemInput contains the input parameters for WriteToFileSystem.

type WriteToResourceInput added in v0.0.3 

type WriteToResourceInput struct {
	ACL        string       // ACL for objects written to AWS s3
	Alg        string       // compression algorithm
	Append     bool         // append to output resource
	BufferSize int          // buffer size
	Dict       []byte       // compression dictionary
	Mode       uint32       // mode of the output file
	Parents    bool         // automatically create parent directories as necessary
	Password   string       // password
	PrivateKey []byte       // private key
	S3Client   *s3.S3       // AWS S3 Client
	SSHClient  *ssh.Client  // SSH Client
	SFTPClient *sftp.Client // SFTP Client
	URI        string       // uri to write to
}

type WriteToResourceOutput added in v0.0.3 

type WriteToResourceOutput struct {
	Writer io.WriteCloser
}

func WriteToResource 

func WriteToResource(input *WriteToResourceInput) (*WriteToResourceOutput, error)

WriteToResource returns a ByteWriteCloser and error, if any.

type Writer 

type Writer struct {
	*sync.Mutex               // inherits Lock and Unlock Functions
	Writer      io.ByteWriter // the instance of ByteWriter used for reading bytes
}

Writer is a struct for normalizing reading of bytes from files with arbitrary compression and for closing underlying resources. Writer implements the ByteWriter interface by wrapping around a subordinate ByteWriter.

func NewWriter 

func NewWriter(w io.ByteWriter) *Writer

func (*Writer) Close 

func (w *Writer) Close() error

Close recursively closes all the underlying writers.

func (*Writer) CloseSafe 

func (w *Writer) CloseSafe() error

CloseSafe closes the Closer and the underlying *os.File if not nil. CloseSafe also locks the writer for the duration of flushing using a sync.Mutex.

func (*Writer) Flush 

func (w *Writer) Flush() error

Flush recursively flushes all the underlying writers.

func (*Writer) FlushSafe 

func (w *Writer) FlushSafe() error

FlushSafe flushes any intermediate writer. FlushSafe also locks the writer for the duration of flushing using a sync.Mutex.

func (*Writer) Write 

func (w *Writer) Write(p []byte) (n int, err error)

Write writes a slice of bytes to the underlying writer and returns an error, if any.

func (*Writer) WriteByte 

func (w *Writer) WriteByte(b byte) error

WriteByte writes a single byte to the underlying writer.

func (*Writer) WriteError 

func (w *Writer) WriteError(e error) (n int, err error)

WriteError writes a an error as a string with a trailing newline to the underlying writer and returns an error, if any.

func (*Writer) WriteErrorSafe 

func (w *Writer) WriteErrorSafe(e error) (n int, err error)

WriteErrorSafe writes a an error as a string with a trailing newline to the underlying writer and returns an error, if any. WriteErrorSafe also locks the writer for the duration of writing using a sync.Mutex.

func (*Writer) WriteLine 

func (w *Writer) WriteLine(s string) (n int, err error)

WriteLine writes a string with a trailing newline to the underlying writer and returns an error, if any.

func (*Writer) WriteLineSafe 

func (w *Writer) WriteLineSafe(s string) (n int, err error)

WriteLineSafe writes a string with a trailing newline to the underlying writer and returns an error, if any. WriteLineSafe also locks the writer for the duration of writing using a sync.Mutex.

func (*Writer) WriteString 

func (w *Writer) WriteString(s string) (n int, err error)

WriteString writes a string to the underlying writer and returns an error, if any.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.