Documentation ¶
Overview ¶
Package ioutil implements some I/O utility functions which are not covered by the standard library.
Package ioutil implements some I/O utility functions which are not covered by the standard library.
Index ¶
- Constants
- Variables
- func AppendFile(dst string, src string, osync bool) error
- func Copy(dst io.Writer, src io.Reader) (written int64, err error)
- func CopyAligned(w io.Writer, r io.Reader, alignedBuf []byte, totalSize int64, file *os.File) (int64, error)
- func DiscardReader(r io.Reader)
- func HardLimitReader(r io.Reader, n int64) io.Reader
- func NewDeadlineWriter(w io.WriteCloser, timeout time.Duration) io.WriteCloser
- func NewSkipReader(r io.Reader, n int64) io.Reader
- func NopCloser(w io.Writer) io.WriteCloser
- func ReadFile(name string) ([]byte, error)
- func ReadFileWithFileInfo(name string) ([]byte, fs.FileInfo, error)
- func SafeClose[T any](c chan<- T)
- func SameFile(fi1, fi2 os.FileInfo) bool
- func WaitPipe() (*PipeReader, *PipeWriter)
- func WithDeadline[V any](ctx context.Context, timeout time.Duration, ...) (result V, err error)
- type DeadlineWorker
- type DeadlineWriter
- type HardLimitedReader
- type LimitWriter
- type PipeReader
- type PipeWriter
- type SkipReader
- type WriteOnCloser
Constants ¶
const ( SmallBlock = 32 * humanize.KiByte // Default r/w block size for smaller objects. LargeBlock = 1 * humanize.MiByte // Default r/w block size for normal objects. )
Block sizes constant.
const DirectioAlignSize = 4096
DirectioAlignSize - DirectIO alignment needs to be 4K. Defined here as directio.AlignSize is defined as 0 in MacOS causing divide by 0 error.
Variables ¶
var ( ODirectPoolLarge = sync.Pool{ New: func() interface{} { b := disk.AlignedBlock(LargeBlock) return &b }, } ODirectPoolSmall = sync.Pool{ New: func() interface{} { b := disk.AlignedBlock(SmallBlock) return &b }, } )
aligned sync.Pool's
var ( // OpenFileDirectIO allows overriding default function. OpenFileDirectIO = disk.OpenFileDirectIO // OsOpen allows overriding default function. OsOpen = os.Open // OsOpenFile allows overriding default function. OsOpenFile = os.OpenFile )
var Discard io.Writer = discard{}
Discard is just like io.Discard without the io.ReaderFrom compatible implementation which is buggy on NUMA systems, we have to use a simpler io.Writer implementation alone avoids also unnecessary buffer copies, and as such incurred latencies.
var ErrOverread = errors.New("input provided more bytes than specified")
ErrOverread is returned to the reader when the hard limit of HardLimitReader is exceeded.
Functions ¶
func AppendFile ¶
AppendFile - appends the file "src" to the file "dst"
func CopyAligned ¶
func CopyAligned(w io.Writer, r io.Reader, alignedBuf []byte, totalSize int64, file *os.File) (int64, error)
CopyAligned - copies from reader to writer using the aligned input buffer, it is expected that input buffer is page aligned to 4K page boundaries. Without passing aligned buffer may cause this function to return error.
This code is similar in spirit to io.Copy but it is only to be used with DIRECT I/O based file descriptor and it is expected that input writer *os.File not a generic io.Writer. Make sure to have the file opened for writes with syscall.O_DIRECT flag.
func HardLimitReader ¶
HardLimitReader returns a Reader that reads from r but returns an error if the source provides more data than allowed. This means the source *will* be overread unless EOF is returned prior. The underlying implementation is a *HardLimitedReader. This will ensure that at most n bytes are returned and EOF is reached.
func NewDeadlineWriter ¶
func NewDeadlineWriter(w io.WriteCloser, timeout time.Duration) io.WriteCloser
NewDeadlineWriter wraps a writer to make it respect given deadline value per Write(). If there is a blocking write, the returned Writer will return whenever the timer hits (the return values are n=0 and err=context.DeadlineExceeded.)
func NewSkipReader ¶
NewSkipReader - creates a SkipReader
func NopCloser ¶
func NopCloser(w io.Writer) io.WriteCloser
NopCloser returns a WriteCloser with a no-op Close method wrapping the provided Writer w.
func ReadFile ¶
ReadFile reads the named file and returns the contents. A successful call returns err == nil, not err == EOF. Because ReadFile reads the whole file, it does not treat an EOF from Read as an error to be reported.
passes NOATIME flag for reads on Unix systems to avoid atime updates.
func ReadFileWithFileInfo ¶
ReadFileWithFileInfo reads the named file and returns the contents. A successful call returns err == nil, not err == EOF. Because ReadFile reads the whole file, it does not treat an EOF from Read as an error to be reported.
func WaitPipe ¶
func WaitPipe() (*PipeReader, *PipeWriter)
WaitPipe implements wait-group backend io.Pipe to provide synchronization between read() end with write() end.
func WithDeadline ¶
func WithDeadline[V any](ctx context.Context, timeout time.Duration, work func(ctx context.Context) (result V, err error)) (result V, err error)
WithDeadline will execute a function with a deadline and return a value of a given type. If the deadline/context passes before the function finishes executing, the zero value and the context error is returned.
Types ¶
type DeadlineWorker ¶
type DeadlineWorker struct {
// contains filtered or unexported fields
}
DeadlineWorker implements the deadline/timeout resiliency pattern.
func NewDeadlineWorker ¶
func NewDeadlineWorker(timeout time.Duration) *DeadlineWorker
NewDeadlineWorker constructs a new DeadlineWorker with the given timeout. To return values, use the WithDeadline helper instead.
func (*DeadlineWorker) Run ¶
func (d *DeadlineWorker) Run(work func() error) error
Run runs the given function, passing it a stopper channel. If the deadline passes before the function finishes executing, Run returns context.DeadlineExceeded to the caller. channel so that the work function can attempt to exit gracefully. Multiple calls to Run will run independently of each other.
type DeadlineWriter ¶
type DeadlineWriter struct { io.WriteCloser // contains filtered or unexported fields }
DeadlineWriter deadline writer with timeout
func (*DeadlineWriter) Close ¶
func (w *DeadlineWriter) Close() error
Close closer interface to close the underlying closer
type HardLimitedReader ¶
A HardLimitedReader reads from R but limits the amount of data returned to just N bytes. Each call to Read updates N to reflect the new amount remaining. Read returns EOF when N <= 0 or when the underlying R returns EOF.
type LimitWriter ¶
LimitWriter implements io.WriteCloser.
This is implemented such that we want to restrict an enscapsulated writer upto a certain length and skip a certain number of bytes.
func LimitedWriter ¶
func LimitedWriter(w io.Writer, skipBytes int64, limit int64) *LimitWriter
LimitedWriter takes an io.Writer and returns an ioutil.LimitWriter.
func (*LimitWriter) Close ¶
func (w *LimitWriter) Close() error
Close closes the LimitWriter. It behaves like io.Closer.
type PipeReader ¶
type PipeReader struct { *io.PipeReader // contains filtered or unexported fields }
PipeReader is similar to io.PipeReader with wait group
func (*PipeReader) CloseWithError ¶
func (r *PipeReader) CloseWithError(err error) error
CloseWithError close with supplied error the reader end
type PipeWriter ¶
type PipeWriter struct { *io.PipeWriter // contains filtered or unexported fields }
PipeWriter is similar to io.PipeWriter with wait group
func (*PipeWriter) CloseWithError ¶
func (w *PipeWriter) CloseWithError(err error) error
CloseWithError close with supplied error the writer end.
type SkipReader ¶
SkipReader skips a given number of bytes and then returns all remaining data.
type WriteOnCloser ¶
WriteOnCloser implements io.WriteCloser and always executes at least one write operation if it is closed.
This can be useful within the context of HTTP. At least one write operation must happen to send the HTTP headers to the peer.
func WriteOnClose ¶
func WriteOnClose(w io.Writer) *WriteOnCloser
WriteOnClose takes an io.Writer and returns an ioutil.WriteOnCloser.
func (*WriteOnCloser) Close ¶
func (w *WriteOnCloser) Close() error
Close closes the WriteOnCloser. It behaves like io.Closer.
func (*WriteOnCloser) HasWritten ¶
func (w *WriteOnCloser) HasWritten() bool
HasWritten returns true if at least one write operation was performed.