README
¶
zstdpool
Zstdpool provides flexible pool implementations for the Encoder and Decoder types in github.com/klauspost/compress/zstd which do not leak goroutines.
Why not use sync.Pool?
zstd.Encoder leaks goroutines if it is garbage collected without Close()
being called first. So we can't safely put an unclosed encoder in a sync.Pool.
But encoders cannot be reused after being closed, so we can't put a closed
encoder in a sync.Pool either.
zstd.Decoders can be reused after being closed, so you can close them before
placing them in a sync.Pool, but doing so frees resources that we would want
to keep until the decoder is no longer used.
These problems might be possible to work around with finalizers, but it is difficult to confirm is working as expected, and could silently break with internal changes in github.com/klauspost/compress/zstd.
Status
This code is not yet well tested, and the API may change at any time.
Usage
Encoding
var encPool = zstdpool.NewEncoderPool()
func compressStream(in io.Reader, out io.Writer) error {
enc, err := encPool.Get(out)
if err != nil {
return err
}
defer encPool.Put(enc)
_, err = enc.ReadFrom(in)
return err
}
Decoding
var decPool = zstdpool.NewDecoderPool()
func decompressStream(in io.Reader, out io.Writer) error {
dec, err := decPool.Get(in)
if err != nil {
return err
}
defer decPool.Put(dec)
_, err = dec.WriteTo(out)
return err
}
Contributions
Contributions are always welcome.
License
This code is released under the Apache 2.0 license.
Documentation
¶
Overview ¶
Package zstdpool provides pools for github.com/klauspost/compress/zstd's Encoder and Decoder types, which do not leak goroutines as naive usage of sync.Pool might.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type DecoderPool ¶
type DecoderPool struct {
// contains filtered or unexported fields
}
DecoderPool implements a non-leaky pool of zstd.Decoders, since sync.Pool can leak goroutines when used with zstd.Decoder.
func NewDecoderPool ¶
func NewDecoderPool(opts ...zstd.DOption) DecoderPool
NewDecoderPool returns a DecoderPool that pools *zstd.Decoders created with the specified zstd.DOptions.
func (*DecoderPool) Get ¶
Get returns a new (or reset) *zstd.Decoder for reading from r from the pool. The *zstd.Decoder should not be Close()'ed before being returned to the pool with Put().
Note that the decoder.IOReadCloser() should not be Close()'ed before the *zstd.Decoder is returned to the pool. Consider using GetReadCloser instead of Get if you want to use the decoder.IOReadCloser().
func (*DecoderPool) GetReadCloser ¶
func (p *DecoderPool) GetReadCloser(r io.Reader) (*DecoderReadCloser, error)
GetReadCloser returns an io.ReadCloser that decompresses zstandard data from r and returns the underlying *zstd.Decoder to this DecoderPool on Close().
func (*DecoderPool) Put ¶
func (p *DecoderPool) Put(d *zstd.Decoder)
Put adds an unused *zstd.Decoder to the pool. You should only add decoders to the pool that were returned by the pool's Get function, or were created with same zstd.DOptions as the pool.
func (*DecoderPool) Resize ¶
func (p *DecoderPool) Resize(ts TargetSize) (old, new int, err error)
Resize takes a TargetSize function ts, which it asks for a target size to reduce the pool size to. It returns the original size of the pool, the new size of the pool, and an error if something went wrong.
type DecoderReadCloser ¶
type DecoderReadCloser struct {
// contains filtered or unexported fields
}
DecoderReadCloser implements io.Readcloser by wrapping a *zstd.Decoder and returning it to a DecoderPool when Close() is called.
func (*DecoderReadCloser) Close ¶
func (c *DecoderReadCloser) Close()
Close does not close the underlying *zstd.Decoder, but instead returns it to the DecoderPool.
type EncoderPool ¶
type EncoderPool struct {
// contains filtered or unexported fields
}
EncoderPool implements a non-leaky pool of zstd.Encoders, since sync.Pool can leak goroutines when used with zstd.Encoder.
func NewEncoderPool ¶
func NewEncoderPool(opts ...zstd.EOption) EncoderPool
NewEncoderPool returns an EncoderPool that pools *zstd.Encoders created with the specified zstd.EOptions.
func (*EncoderPool) Get ¶
Get returns a new (or reset) *zstd.Encoder for writing to w from the pool. The *zstd.Encoder should be Close()'ed before being returned to the pool with Put().
func (*EncoderPool) Put ¶
func (p *EncoderPool) Put(e *zstd.Encoder)
Put adds an unused *zstd.Encoder to the pool. You should only add encoders to the pool that were returned by the pool's Get function, or were created with same zstd.EOptions as the pool.
func (*EncoderPool) Resize ¶
func (p *EncoderPool) Resize(ts TargetSize) (old, new int, err error)
Resize takes a TargetSize function ts, which it asks for a target size to reduce the pool size to. It returns the original size of the pool, the new size of the pool, and an error if something went wrong.
type TargetSize ¶
TargetSize functions take the current size of a pool, and return the target size (which must not be larger than the current size), and should not be negative.
Source Files