taskcluster-worker: github.com/taskcluster/taskcluster-worker/runtime/ioext Index | Files

package ioext

import "github.com/taskcluster/taskcluster-worker/runtime/ioext"

Package ioext contains interfaces and implementations for when the default io types are not sufficient.


Package Files

asyncpipe.go blockedpipe.go copy.go copyclose.go copyflush.go doc.go fileutils.go ioext.go nopconn.go readatmost.go readutils.go tellreader.go watchpipe.go


var ErrFileTooBig = errors.New("File is larger than max size given")

ErrFileTooBig is used the indicate that a file is too big.

var ErrMaxSizeExceeded = errors.New("MaxSize was exceeded before EOF was reached")

ErrMaxSizeExceeded signals that EOF wasn't reached instead max size was was read and, hence, we stopped reading.

var ErrPipeFull = errors.New("The internal pipe buffer have reached its capacity")

ErrPipeFull is returned from AsyncPipeWriter.Write if the pipes capacity has been reached.

func BoundedReadAll Uses

func BoundedReadAll(r io.Reader, maxBytes int) ([]byte, error)

BoundedReadAll will read up to maxBytes from r, and returns ErrFileTooBig if the file is larger.

func BoundedReadFile Uses

func BoundedReadFile(filename string, maxBytes int) ([]byte, error)

BoundedReadFile will read up to maxBytes from filename, and returns ErrFileTooBig if the file is larger, and *os.PathError if file doesn't exist.

func Copy Uses

func Copy(dst io.Writer, src io.Reader) (written int64, werr, rerr error)

Copy from dst to src returning bytes written, write error werr and read error rerr. This is similar to io.Copy except users can distinguish between read and write errors.

func CopyAndClose Uses

func CopyAndClose(w io.WriteCloser, r io.Reader) (int64, error)

CopyAndClose will copy from r to w and close w, returning the number of bytes copied and the first error, if any, this always closes regardless of error.

func CopyAndFlush Uses

func CopyAndFlush(dst WriteFlusher, src io.Reader, interval time.Duration) (int64, error)

CopyAndFlush will copy src to dst flushing at given interval.

func IsFileLessThan Uses

func IsFileLessThan(filePath string, maxSize int64) bool

IsFileLessThan returns true if filePath is a file less than maxSize

func IsPlainFile Uses

func IsPlainFile(filePath string) bool

IsPlainFile returns an true if filePath is a plain file, not a directory, symlink, device, etc.

func IsPlainFileInfo Uses

func IsPlainFileInfo(info os.FileInfo) bool

IsPlainFileInfo returns true, if info is for a plain file, not a dictionary, symlink, device, etc.

func NopConn Uses

func NopConn(conn io.ReadWriteCloser) net.Conn

NopConn wraps conn so that it provides a trivial implementation of net.Conn. This is only useful for testing, deadlines are ignored and address methods will return nil.

func ReadAtMost Uses

func ReadAtMost(r io.Reader, maxSize int64) ([]byte, error)

ReadAtMost will read at-most maxSize bytes from r and return an error if we didn't reach EOF. Returns ErrMaxSizeExceeded if r contains more than maxSize bytes. If maxSize is -1 ReadAtMost will read everything.

This utility is useful when reading HTTP requests, in particular if reading from untrusted sources. If using io.ReadAll it's easy to run the server out of memory, a maxSize of 2MiB is usually sane and prevents such attacks.

func WatchPipe Uses

func WatchPipe(pipe io.ReadWriteCloser, onClose func(error)) io.ReadWriteCloser

WatchPipe will wrap an io.ReadWriteCloser such that onClose is called when the first read or write error happens, or after close() is called.

The onClose callback will be called with the error from read, write or close. To allow for locking for example to remove the pipe from a list when closed, the onClose callback is always called on in separate go-routine.

func WriteNopCloser Uses

func WriteNopCloser(w io.Writer) io.WriteCloser

WriteNopCloser wraps an io.Writer and creates a io.WriteCloser where Close is a noop.

type AsyncPipeReader Uses

type AsyncPipeReader struct {

AsyncPipeReader is a reading side of an AsyncPipe, similar to io.PipeReader.

func AsyncPipe Uses

func AsyncPipe(capacity int, tell chan<- int) (*AsyncPipeReader, *AsyncPipeWriter)

AsyncPipe is similar to io.Pipe() except that writes isn't blocking, instead data will be added to an internal buffer that can grow up to specified capacity. Additionally, you may supply a channel tell that will be told whenever N bytes have been read, so that more bytes can be requested to be written.

This pipe kind is useful when implementing congestion control.

func (*AsyncPipeReader) Close Uses

func (r *AsyncPipeReader) Close() error

Close the pipe reader

func (*AsyncPipeReader) Read Uses

func (r *AsyncPipeReader) Read(p []byte) (int, error)

type AsyncPipeWriter Uses

type AsyncPipeWriter struct {
    // contains filtered or unexported fields

AsyncPipeWriter is a writing side of a BlockedPipe, similar to io.PipeWriter.

func (*AsyncPipeWriter) Close Uses

func (w *AsyncPipeWriter) Close() error

Close will close the stream

func (*AsyncPipeWriter) Write Uses

func (w *AsyncPipeWriter) Write(p []byte) (int, error)

type PipeReader Uses

type PipeReader struct {
    // contains filtered or unexported fields

PipeReader is a reading side of a BlockedPipe, similar to io.PipeReader.

func BlockedPipe Uses

func BlockedPipe() (*PipeReader, *PipeWriter)

BlockedPipe is similar to io.Pipe() except the pipe is blocked until PipeReader.Unblock(n) is called to unblock n bytes. This is useful when implementing congestion control.

Note: PipeReader.Unblock(-1) will permanently unblock the pipe, PipeReader.Close() will break the pipe, either one must be called unless the pipe is read EOF. Otherwise, the pipe will remain blocked and you may leak a go routine.

func (*PipeReader) Close Uses

func (r *PipeReader) Close() error

Close mirrors io.PipeReader.Close()

func (*PipeReader) CloseWithError Uses

func (r *PipeReader) CloseWithError(err error) error

CloseWithError mirrors io.PipeReader.CloseWithError()

func (*PipeReader) Read Uses

func (r *PipeReader) Read(data []byte) (int, error)

Read mirrors io.PipeReader.Read() except it won't read more bytes than have been unblocked by calling r.Unblock().

func (*PipeReader) Unblock Uses

func (r *PipeReader) Unblock(n int64)

Unblock allows n bytes to traverse through the pipe. Typically, this pipe is used when implementing congestion control and r.Unblock(n) is then called when the remote side have acknowledged n bytes. This way the network isn't congested with lots of outstanding bytes.

As a special case n = -1 permanently unblocks the pipe.

Note: That with this reader it is important to call r.Close() when cleaning up or r.Unblock(-1) as w.Close() won't propagate unless enough bytes are unblocked, and this could otherwise leave the pipe permanently blocked, which easily leaves you leaking go routines.

type PipeWriter Uses

type PipeWriter struct {
    // contains filtered or unexported fields

PipeWriter is a writing side of a BlockedPipe, similar to io.PipeWriter.

func (*PipeWriter) Close Uses

func (w *PipeWriter) Close() error

Close mirrors io.PipeWriter.Close()

func (*PipeWriter) CloseWithError Uses

func (w *PipeWriter) CloseWithError(err error) error

CloseWithError mirrors io.CloseWithError.CloseWithError()

func (*PipeWriter) Write Uses

func (w *PipeWriter) Write(data []byte) (int, error)

type ReadSeekCloser Uses

type ReadSeekCloser interface {

ReadSeekCloser implements io.Reader, io.Seeker, and io.Closer. It is trivially implemented by os.File.

func NopCloser Uses

func NopCloser(r io.ReadSeeker) ReadSeekCloser

NopCloser wraps a io.ReadSeeker as ReadSeekCloser with Close being a noop. This is useful for compliance with interface, if you don't care about closing.

type TellReader Uses

type TellReader struct {
    // contains filtered or unexported fields

TellReader is an io.Reader wrapper that can tell how much has been read.

This is useful for monitoring download progress.

func (*TellReader) Read Uses

func (r *TellReader) Read(p []byte) (n int, err error)

func (*TellReader) Tell Uses

func (r *TellReader) Tell() int64

Tell the number bytes read so far

type WriteFlusher Uses

type WriteFlusher interface {

WriteFlusher is a combination of io.Writer and http.Flusher, basically a stream that can be flushed.

func NopFlusher Uses

func NopFlusher(w io.Writer) WriteFlusher

NopFlusher returns a WriteFlusher implementation that wraps w, with Flush being a no-op.

Package ioext imports 10 packages (graph) and is imported by 22 packages. Updated 2018-06-21. Refresh now. Tools for package owners.