ioutil

package module
v0.0.0-...-1600d81 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 7, 2019 License: GPL-3.0 Imports: 13 Imported by: 0

README

pipeline status coverage report

Coverage Report

Ioutil

A simple library that includes the standard functionality of the original io/ioutil package, but with additional features.

Why?

This library was created to quickly add Input/Output utilities to a single library without having to both import io/ioutil and another utility library. Proposals to io/ioutils are unlikely to be accepted and added quickly, hence this library.

Documentation

Index

Constants

View Source 
const (
	// DefaultWriteTimeout is the default amount of time to wait for more write to arrive before
	// killing the goroutine that waits for writes.
	DefaultWriteTimeout = 500 * time.Millisecond

	// DefaultWriteQueueSize is the default size of the write messages that can accumulate before
	// calls to write become synchronous or timeout.
	DefaultWriteQueueSize = 100
)

Variables

This section is empty.

Functions

func FileExists 

func FileExists(path string) bool

FileExists returns true if the file exists.

func NopCloser 

func NopCloser(r io.Reader) io.ReadCloser

NopCloser returns a ReadCloser with a no-op Close method wrapping the provided Reader r.

func NopReadCloser 

func NopReadCloser(r io.Reader) io.ReadCloser

NopReadCloser returns a ReadCloser with a no-op Close method wrapping the provided Reader r.

func NopReadWriteCloser 

func NopReadWriteCloser(r io.ReadWriter) io.ReadWriteCloser

NopCloser returns a ReadWriteCloser with a no-op Close method wrapping the provided ReadWriter r.

func NopWriteCloser 

func NopWriteCloser(w io.Writer) io.WriteCloser

NopWriteCloser returns a WriteCloser with a no-op Close method wrapping the provided Writer r.

func ReadAll 

func ReadAll(r io.Reader) ([]byte, error)

ReadAll reads from r until an error or EOF and returns the data it read. A successful call returns err == nil, not err == EOF. Because ReadAll is defined to read from src until EOF, it does not treat an EOF from Read as an error to be reported.

func ReadDir 

func ReadDir(dirname string) ([]os.FileInfo, error)

ReadDir reads the directory named by dirname and returns a list of directory entries sorted by filename.

func ReadFile 

func ReadFile(filename string) ([]byte, error)

ReadFile reads the file named by filename and returns the contents. A successful call returns err == nil, not err == EOF. Because ReadFile reads the whole file, it does not treat an EOF from Read as an error to be reported.

func TempDir 

func TempDir(dir, prefix string) (name string, err error)

TempDir creates a new temporary directory in the directory dir with a name beginning with prefix and returns the path of the new directory. If dir is the empty string, TempDir uses the default directory for temporary files (see os.TempDir). Multiple programs calling TempDir simultaneously will not choose the same directory. It is the caller's responsibility to remove the directory when no longer needed.

func TempFile 

func TempFile(dir, prefix string) (f *os.File, err error)

TempFile creates a new temporary file in the directory dir with a name beginning with prefix, opens the file for reading and writing, and returns the resulting *os.File. If dir is the empty string, TempFile uses the default directory for temporary files (see os.TempDir). Multiple programs calling TempFile simultaneously will not choose the same file. The caller can use f.Name() to find the pathname of the file. It is the caller's responsibility to remove the file when no longer needed.

func WriteAllBytes 

func WriteAllBytes(writer io.Writer, b []byte) (err error)

WriteAllBytes writes all the bytes in b to the writer with some simple retry logic. The passed writer is called multiple times when the writer reports fewer bytes written than what it was given. After a number of unsuccessful attempts at writing, an error is returned.

func WriteFile 

func WriteFile(filename string, data []byte, perm os.FileMode) error

WriteFile writes data to a file named by filename. If the file does not exist, WriteFile creates it with permissions perm; otherwise WriteFile truncates it before writing.

func WriteWithRetry 

func WriteWithRetry(buffer io.Writer, p []byte, totalTimeToWait time.Duration) (int, error)

WriteWithRetry writes the contents of p to buffer. If p was unable to be completely written, the remaining parts of p are continually attempted until either all of p is written or until too much time waiting has passed, as specified by totalTimeToWait.

Types

type AsyncWriter 

type AsyncWriter struct {

	// The amount of time to wait between writes before the goroutine that is writing quits. If
	// another write arrives then goroutine is recreated. Frequently creating the write go routine
	// slow downs writes, as all new calls to Write are blocked while the goroutine is created, and once
	// created any queues writes are quickly written to the underlying writer.
	Timeout time.Duration

	// ErrorHandler is called whenever the underlying WriteCloser returns an error. If ErrorHandler returns
	// an error, then all pending writes are dropped and future writes are ignored.
	ErrorHandler AsyncWriterErrorHandler
	// contains filtered or unexported fields
}

AsyncWriter defines some state that maintains asynchronous writing.

Although AsyncWriter implements the Write interface, it does not actual obey the behavior. See the write function for AsyncWriter for more information.

func NewAsyncWriter 

func NewAsyncWriter(writer io.WriteCloser, queueSize int) *AsyncWriter

func (*AsyncWriter) Close 

func (asyncWriter *AsyncWriter) Close() error

func (*AsyncWriter) Wait 

func (asyncWriter *AsyncWriter) Wait()

func (*AsyncWriter) Write 

func (asyncWriter *AsyncWriter) Write(p []byte) (n int, err error)

Write queues up p into a buffer that will eventually be written to the underlying writer. Write not return the number of bytes written because the write happens later. An error might return if p could not be enqueued.

type AsyncWriterErrorHandler 

type AsyncWriterErrorHandler func(p []byte, writer io.WriteCloser, err error) error
var DefaultAsyncWriterErrorHandler AsyncWriterErrorHandler

type BasicHashBuildingReader 

type BasicHashBuildingReader struct {
	// contains filtered or unexported fields
}

BasicHashBuildingReader is a file reader that builds a hash while the file is read. It avoid need for either reading the file twice; one to build the hash and again to use the file data for something else.

func (*BasicHashBuildingReader) AppendSha1 

func (reader *BasicHashBuildingReader) AppendSha1(b bool)

AppendSha1 sets whether the hash should be included the read operation once the entire stream is read in.

func (*BasicHashBuildingReader) Close 

func (reader *BasicHashBuildingReader) Close() error

Close the underlying reader

func (*BasicHashBuildingReader) Hash 

func (reader *BasicHashBuildingReader) Hash() []byte

Hash returns the SHA1 checksum.

func (*BasicHashBuildingReader) Read 

func (reader *BasicHashBuildingReader) Read(b []byte) (l int, err error)

Read a portion of a file into the provided byte buffer, building a hash in the process. If AppendHash was BasicHashBuildingReader to true, then the hash is included in the read

type HashBuildingReader 

type HashBuildingReader interface {
	io.ReadCloser

	// Get the hash for all data that as thus far been read
	Hash() []byte

	// If set true, when the entire underlying stream has been read, the SHA1 checksum
	// is read as part of the stream.
	AppendSha1(b bool)
}

HashBuildingReader is similar to the io.ReadCloser interface except that implementations are expected to support the inclusion of the hash upon finishing the read of the underlying stream as well as a way to get the hash that is built while reading the stream.

Building the hash while reading makes it easy to build hashes of streams where otherwise the data would be read twice, once for constructing the hash and a second time to store or send the data.

When reading a file from disk using a HashBuilding reader avoids having to read the file twice when do not need the hash of the file until the end of the read, such as when we are sending a file to an API that allows the hash to be sent after the data.

func NewBasicHashBuildingReader 

func NewBasicHashBuildingReader(hash hash.Hash, reader io.ReadCloser) HashBuildingReader

NewBasicHashBuildingReader creates a HashBuildingReader that builds an hash while reading from a reader.

func NewRandomDataGenerator 

func NewRandomDataGenerator(length int64) HashBuildingReader

NewRandomDataGenerator creates a new RandomDataGenerator that implements the io.ReaderCloser interface on random data.

func NewSha1BuildingReader 

func NewSha1BuildingReader(reader io.ReadCloser) HashBuildingReader

NewSha1BuildingReader creates a new HashBuildingReader that builds a sha1 hash while reading a reader.

type HashGeneratingBuffer 

type HashGeneratingBuffer interface {
	io.Reader
	io.Writer
	io.Closer

	GetHash() []byte
	WriteHashOnClose(bool)
}

type HashWriteError 

type HashWriteError error

type RandomDataGenerator 

type RandomDataGenerator struct {
	// contains filtered or unexported fields
}

RandomDataGenerator implements the io.ReaderCloser interface that generates random data of specified size. Useful in testing situations where the content of the data does not matter.

func (*RandomDataGenerator) Close 

func (buff *RandomDataGenerator) Close() (err error)

Close the random data reader. Effectively does nothing.

func (*RandomDataGenerator) Read 

func (buff *RandomDataGenerator) Read(b []byte) (n int, err error)

Read random data into buffer b, returning the number of bytes written to b. The returned error only returns io.EOF when enough data has been read as specified in NewRandomDataGenerator has been read.

type ReTryTimeout 

type ReTryTimeout error

type ReTryWriter 

type ReTryWriter interface {
	io.Writer

	// WriteWithTimeout writes the contents of p, retrying up to d amount of time before giving up with an error
	// of type ReTryTimeout.
	WriteWithTimeout(p []byte, d time.Duration)
}

ReTryWriter is a writer that includes re-try logic when writing with the intent to provide more reliable writes in situations were the underlying storage medium or execution context is prone to intermittent failures.

type Sha1BuildingByteBuffer 

type Sha1BuildingByteBuffer struct {
	// contains filtered or unexported fields
}

A buffer that when continually builds a hash hash when written to. Using this buffer avoids the needs to read a file twice. Once for the hash hash and again for when

func NewSha1BuildingByteBuffer 

func NewSha1BuildingByteBuffer() *Sha1BuildingByteBuffer

Create a buffer that when written to builds a hash of the data.

func (*Sha1BuildingByteBuffer) Close 

func (buffer *Sha1BuildingByteBuffer) Close() error

Close closes the byte buffer, making it impossible to write more to. If WriteOnClose was called with a true value then the hash will be written to the buffer before closing.

func (*Sha1BuildingByteBuffer) GetHash 

func (buffer *Sha1BuildingByteBuffer) GetHash() []byte

GetHash returns the has of the data thus far written.

func (*Sha1BuildingByteBuffer) Grow 

func (buffer *Sha1BuildingByteBuffer) Grow(n int)

Grow grows the buffer's capacity, if necessary, to guarantee space for another n bytes. After Grow(n), at least n bytes can be written to the buffer without another allocation. If n is negative, Grow will panic. If the buffer can't grow it will panic with ErrTooLarge.

func (*Sha1BuildingByteBuffer) Read 

func (buffer *Sha1BuildingByteBuffer) Read(p []byte) (n int, err error)

Read reads the next len(p) bytes from the buffer or until the buffer is drained. The return value n is the number of bytes read. If the buffer has no data to return, err is io.EOF (unless len(p) is zero); otherwise it is nil. If Closed has been called on buffer, an error is returned.

func (*Sha1BuildingByteBuffer) ReadByte 

func (buffer *Sha1BuildingByteBuffer) ReadByte() (byte, error)

ReadByte reads and returns the next byte from the buffer. If no byte is available, it returns error io.EOF. If Closed has been called on buffer, an error is returned.

func (*Sha1BuildingByteBuffer) ReadFrom 

func (buffer *Sha1BuildingByteBuffer) ReadFrom(r io.Reader) (n int64, err error)

ReadFrom reads data from r until EOF and appends it to the buffer, growing the buffer as needed. The return value n is the number of bytes read. Any error except io.EOF encountered during the read is also returned. If the buffer becomes too large, ReadFrom will panic with ErrTooLarge. If Closed has been called on buffer, an error is returned.

func (*Sha1BuildingByteBuffer) String 

func (buffer *Sha1BuildingByteBuffer) String() string

String returns a string representation of the buffer.

func (*Sha1BuildingByteBuffer) UnreadByte 

func (buffer *Sha1BuildingByteBuffer) UnreadByte() (err error)

UnreadByte unreads the last byte returned by the most recent successful read operation that read at least one byte. If a write has happened since the last read, if the last read returned an error, or if the read read zero bytes, UnreadByte returns an error. If Closed has been called on buffer, an error is returned.

func (*Sha1BuildingByteBuffer) Write 

func (buffer *Sha1BuildingByteBuffer) Write(p []byte) (n int, err error)

Write appends the contents of p to the buffer, growing the buffer as needed. The return value n is the length of p; err is always nil. If the buffer becomes too large, Write will panic with ErrTooLarge. If the hash cannot be computed, HashWriteError is returned. If Closed has been called on buffer, an error is returned.

func (*Sha1BuildingByteBuffer) WriteByte 

func (buffer *Sha1BuildingByteBuffer) WriteByte(c byte) error

WriteByte appends the byte c to the buffer, growing the buffer as needed. The returned error is always nil, but is included to match bufio.Writer's WriteByte. If the buffer becomes too large, WriteByte will panic with ErrTooLarge. If Closed has been called on buffer, an error is returned.

func (*Sha1BuildingByteBuffer) WriteHashOnClose 

func (buffer *Sha1BuildingByteBuffer) WriteHashOnClose(write bool)

Write the hash hash to the stream just before closing the buffer.

func (*Sha1BuildingByteBuffer) WriteRune 

func (buffer *Sha1BuildingByteBuffer) WriteRune(r rune) (n int, err error)

WriteRune appends the UTF-8 encoding of Unicode code point r to the buffer, returning its length and an error, which is always nil but is included to match bufio.Writer's WriteRune. The buffer is grown as needed; if it becomes too large, WriteRune will panic with ErrTooLarge. If the hash cannot be computed, HashWriteError is returned. If Closed has been called on buffer, an error is returned.

func (*Sha1BuildingByteBuffer) WriteString 

func (buffer *Sha1BuildingByteBuffer) WriteString(s string) (n int, err error)

WriteString appends the contents of s to the buffer, growing the buffer as needed. The return value n is the length of s; err is always nil. If the buffer becomes too large, WriteString will panic with ErrTooLarge. If the hash cannot be computed, HashWriteError is returned. If Closed has been called on buffer, an error is returned.

type Sha1BuildingFileBuffer 

type Sha1BuildingFileBuffer struct {
	// contains filtered or unexported fields
}

Sha1BuildingFileBuffer maintains the state of a buffer used to read or write a file while also building a sha1 hash.

func NewSha1BuildingFileBuffer 

func NewSha1BuildingFileBuffer(filePath string, flag int, perm os.FileMode) (*Sha1BuildingFileBuffer, error)

NewSha1BuildingFileBuffer creates a new buffer for a file that computes a hash hash while reading the file or writing the file.

func (*Sha1BuildingFileBuffer) Close 

func (buffer *Sha1BuildingFileBuffer) Close() error

func (*Sha1BuildingFileBuffer) GetHash 

func (buffer *Sha1BuildingFileBuffer) GetHash() []byte

func (*Sha1BuildingFileBuffer) Read 

func (buffer *Sha1BuildingFileBuffer) Read(p []byte) (n int, err error)

Read operates identically to io.Read except that it uses what is read in the building of a sha1 hash.

func (*Sha1BuildingFileBuffer) ReadSha1OnEOF 

func (buffer *Sha1BuildingFileBuffer) ReadSha1OnEOF(b bool)

func (*Sha1BuildingFileBuffer) Write 

func (buffer *Sha1BuildingFileBuffer) Write(b []byte) (n int, err error)

func (*Sha1BuildingFileBuffer) WriteHashOnClose 

func (buffer *Sha1BuildingFileBuffer) WriteHashOnClose(b bool)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.