lz4

package module
v4.1.21 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 8, 2024 License: BSD-3-Clause Imports: 10 Imported by: 309

README

lz4 : LZ4 compression in pure Go

Go Reference CI Go Report Card GitHub tag (latest SemVer)

Overview

This package provides a streaming interface to LZ4 data streams as well as low level compress and uncompress functions for LZ4 data blocks. The implementation is based on the reference C one.

Install

Assuming you have the go toolchain installed:

go get github.com/pierrec/lz4/v4

There is a command line interface tool to compress and decompress LZ4 files.

go install github.com/pierrec/lz4/v4/cmd/lz4c@latest

Usage

Usage of lz4c:
  -version
        print the program version

Subcommands:
Compress the given files or from stdin to stdout.
compress [arguments] [<file name> ...]
  -bc
        enable block checksum
  -l int
        compression level (0=fastest)
  -sc
        disable stream checksum
  -size string
        block max size [64K,256K,1M,4M] (default "4M")

Uncompress the given files or from stdin to stdout.
uncompress [arguments] [<file name> ...]

Example

// Compress and uncompress an input string.
s := "hello world"
r := strings.NewReader(s)

// The pipe will uncompress the data from the writer.
pr, pw := io.Pipe()
zw := lz4.NewWriter(pw)
zr := lz4.NewReader(pr)

go func() {
	// Compress the input string.
	_, _ = io.Copy(zw, r)
	_ = zw.Close() // Make sure the writer is closed
	_ = pw.Close() // Terminate the pipe
}()

_, _ = io.Copy(os.Stdout, zr)

// Output:
// hello world

Contributing

Contributions are very welcome for bug fixing, performance improvements...!

  • Open an issue with a proper description
  • Send a pull request with appropriate test case(s)

Contributors

Thanks to all contributors so far!

Special thanks to @Zariel for his asm implementation of the decoder.

Special thanks to @greatroar for his work on the asm implementations of the decoder for amd64 and arm64.

Special thanks to @klauspost for his work on optimizing the code.

Documentation

Overview

Package lz4 implements reading and writing lz4 compressed data.

The package supports both the LZ4 stream format, as specified in http://fastcompression.blogspot.fr/2013/04/lz4-streaming-format-final.html, and the LZ4 block format, defined at http://fastcompression.blogspot.fr/2011/05/lz4-explained.html.

See https://github.com/lz4/lz4 for the reference C implementation.

Example 
// Compress and uncompress an input string.
s := "hello world"
r := strings.NewReader(s)

// The pipe will uncompress the data from the writer.
pr, pw := io.Pipe()
zw := lz4.NewWriter(pw)
zr := lz4.NewReader(pr)

go func() {
	// Compress the input string.
	_, _ = io.Copy(zw, r)
	_ = zw.Close() // Make sure the writer is closed
	_ = pw.Close() // Terminate the pipe
}()

_, _ = io.Copy(os.Stdout, zr)

Output:

hello world

Index

Examples

Constants

View Source 
const (
	// ErrInvalidSourceShortBuffer is returned by UncompressBlock or CompressBLock when a compressed
	// block is corrupted or the destination buffer is not large enough for the uncompressed data.
	ErrInvalidSourceShortBuffer = lz4errors.ErrInvalidSourceShortBuffer
	// ErrInvalidFrame is returned when reading an invalid LZ4 archive.
	ErrInvalidFrame = lz4errors.ErrInvalidFrame
	// ErrInternalUnhandledState is an internal error.
	ErrInternalUnhandledState = lz4errors.ErrInternalUnhandledState
	// ErrInvalidHeaderChecksum is returned when reading a frame.
	ErrInvalidHeaderChecksum = lz4errors.ErrInvalidHeaderChecksum
	// ErrInvalidBlockChecksum is returned when reading a frame.
	ErrInvalidBlockChecksum = lz4errors.ErrInvalidBlockChecksum
	// ErrInvalidFrameChecksum is returned when reading a frame.
	ErrInvalidFrameChecksum = lz4errors.ErrInvalidFrameChecksum
	// ErrOptionInvalidCompressionLevel is returned when the supplied compression level is invalid.
	ErrOptionInvalidCompressionLevel = lz4errors.ErrOptionInvalidCompressionLevel
	// ErrOptionClosedOrError is returned when an option is applied to a closed or in error object.
	ErrOptionClosedOrError = lz4errors.ErrOptionClosedOrError
	// ErrOptionInvalidBlockSize is returned when
	ErrOptionInvalidBlockSize = lz4errors.ErrOptionInvalidBlockSize
	// ErrOptionNotApplicable is returned when trying to apply an option to an object not supporting it.
	ErrOptionNotApplicable = lz4errors.ErrOptionNotApplicable
	// ErrWriterNotClosed is returned when attempting to reset an unclosed writer.
	ErrWriterNotClosed = lz4errors.ErrWriterNotClosed
)

Variables

View Source 
var (
	DefaultBlockSizeOption = BlockSizeOption(Block4Mb)
	DefaultChecksumOption  = ChecksumOption(true)
	DefaultConcurrency     = ConcurrencyOption(1)
)

Default options.

Functions

func CompressBlock 

func CompressBlock(src, dst []byte, _ []int) (int, error)

CompressBlock is equivalent to Compressor.CompressBlock. The final argument is ignored and should be set to nil.

This function is deprecated. Use a Compressor instead.

Example 
s := "hello world"
data := []byte(strings.Repeat(s, 100))
buf := make([]byte, lz4.CompressBlockBound(len(data)))

var c lz4.Compressor
n, err := c.CompressBlock(data, buf)
if err != nil {
	fmt.Println(err)
}
if n >= len(data) {
	fmt.Printf("`%s` is not compressible", s)
}
buf = buf[:n] // compressed data

// Allocate a very large buffer for decompression.
out := make([]byte, 10*len(data))
n, err = lz4.UncompressBlock(buf, out)
if err != nil {
	fmt.Println(err)
}
out = out[:n] // uncompressed data

fmt.Println(string(out[:len(s)]))

Output:

hello world

func CompressBlockBound 

func CompressBlockBound(n int) int

CompressBlockBound returns the maximum size of a given buffer of size n, when not compressible.

func CompressBlockHC 

func CompressBlockHC(src, dst []byte, depth CompressionLevel, _, _ []int) (int, error)

CompressBlockHC is equivalent to CompressorHC.CompressBlock. The final two arguments are ignored and should be set to nil.

This function is deprecated. Use a CompressorHC instead.

func UncompressBlock 

func UncompressBlock(src, dst []byte) (int, error)

UncompressBlock uncompresses the source buffer into the destination one, and returns the uncompressed size.

The destination buffer must be sized appropriately.

An error is returned if the source data is invalid or the destination buffer is too small.

func UncompressBlockWithDict added in v4.1.5 

func UncompressBlockWithDict(src, dst, dict []byte) (int, error)

UncompressBlockWithDict uncompresses the source buffer into the destination one using a dictionary, and returns the uncompressed size.

The destination buffer must be sized appropriately.

An error is returned if the source data is invalid or the destination buffer is too small.

func ValidFrameHeader added in v4.1.10 

func ValidFrameHeader(in []byte) (bool, error)

ValidFrameHeader returns a bool indicating if the given bytes slice matches a LZ4 header.

Types

type BlockSize 

type BlockSize uint32

BlockSizeIndex defines the size of the blocks to be compressed. 

const (
	Block64Kb BlockSize = 1 << (16 + iota*2)
	Block256Kb
	Block1Mb
	Block4Mb
)

func (BlockSize) String 

func (i BlockSize) String() string

type CompressingReader added in v4.1.20 

type CompressingReader struct {
	// contains filtered or unexported fields
}

func NewCompressingReader added in v4.1.20 

func NewCompressingReader(src io.ReadCloser) *CompressingReader

NewCompressingReader creates a reader which reads compressed data from raw stream. This makes it a logical opposite of a normal lz4.Reader. We require an io.ReadCloser as an underlying source for compatibility with Go's http.Request.

func (*CompressingReader) Apply added in v4.1.20 

func (zrd *CompressingReader) Apply(options ...Option) (err error)

Apply applies useful options to the lz4 encoder.

func (*CompressingReader) Close added in v4.1.20 

func (zrd *CompressingReader) Close() error

Close simply invokes the underlying stream Close method. This method is provided for the benefit of Go http client/server, which relies on Close for goroutine termination.

func (*CompressingReader) Read added in v4.1.20 

func (zrd *CompressingReader) Read(p []byte) (n int, err error)

Read allows reading of lz4 compressed data

func (*CompressingReader) Reset added in v4.1.20 

func (zrd *CompressingReader) Reset(src io.ReadCloser)

Reset makes the stream usable again; mostly handy to reuse lz4 encoder instances.

func (*CompressingReader) Source added in v4.1.20 

func (zrd *CompressingReader) Source() io.ReadCloser

Source exposes the underlying source stream for introspection and control.

type CompressionLevel 

type CompressionLevel uint32

CompressionLevel defines the level of compression to use. The higher the better, but slower, compression. 

const (
	Fast   CompressionLevel = 0
	Level1 CompressionLevel = 1 << (8 + iota)
	Level2
	Level3
	Level4
	Level5
	Level6
	Level7
	Level8
	Level9
)

func (CompressionLevel) String 

func (i CompressionLevel) String() string

type Compressor added in v4.0.3 

type Compressor struct {
	// contains filtered or unexported fields
}

A Compressor compresses data into the LZ4 block format. It uses a fast compression algorithm.

A Compressor is not safe for concurrent use by multiple goroutines.

Use a Writer to compress into the LZ4 stream format.

func (*Compressor) CompressBlock added in v4.0.3 

func (c *Compressor) CompressBlock(src, dst []byte) (int, error)

CompressBlock compresses the source buffer src into the destination dst.

If compression is successful, the first return value is the size of the compressed data, which is always >0.

If dst has length at least CompressBlockBound(len(src)), compression always succeeds. Otherwise, the first return value is zero. The error return is non-nil if the compressed data does not fit in dst, but it might fit in a larger buffer that is still smaller than CompressBlockBound(len(src)). The return value (0, nil) means the data is likely incompressible and a buffer of length CompressBlockBound(len(src)) should be passed in.

type CompressorHC added in v4.0.3 

type CompressorHC struct {
	// Level is the maximum search depth for compression.
	// Values <= 0 mean no maximum.
	Level CompressionLevel
	// contains filtered or unexported fields
}

A CompressorHC compresses data into the LZ4 block format. Its compression ratio is potentially better than that of a Compressor, but it is also slower and requires more memory.

A Compressor is not safe for concurrent use by multiple goroutines.

Use a Writer to compress into the LZ4 stream format.

func (*CompressorHC) CompressBlock added in v4.0.3 

func (c *CompressorHC) CompressBlock(src, dst []byte) (int, error)

CompressBlock compresses the source buffer src into the destination dst.

If compression is successful, the first return value is the size of the compressed data, which is always >0.

If dst has length at least CompressBlockBound(len(src)), compression always succeeds. Otherwise, the first return value is zero. The error return is non-nil if the compressed data does not fit in dst, but it might fit in a larger buffer that is still smaller than CompressBlockBound(len(src)). The return value (0, nil) means the data is likely incompressible and a buffer of length CompressBlockBound(len(src)) should be passed in.

type Option 

type Option func(applier) error

Option defines the parameters to setup an LZ4 Writer or Reader.

func BlockChecksumOption 

func BlockChecksumOption(flag bool) Option

BlockChecksumOption enables or disables block checksum (default=false).

func BlockSizeOption 

func BlockSizeOption(size BlockSize) Option

BlockSizeOption defines the maximum size of compressed blocks (default=Block4Mb).

func ChecksumOption 

func ChecksumOption(flag bool) Option

ChecksumOption enables/disables all blocks or content checksum (default=true).

func CompressionLevelOption 

func CompressionLevelOption(level CompressionLevel) Option

CompressionLevelOption defines the compression level (default=Fast).

func ConcurrencyOption 

func ConcurrencyOption(n int) Option

ConcurrencyOption sets the number of go routines used for compression. If n <= 0, then the output of runtime.GOMAXPROCS(0) is used.

func LegacyOption added in v4.1.0 

func LegacyOption(legacy bool) Option

LegacyOption provides support for writing LZ4 frames in the legacy format.

See https://github.com/lz4/lz4/blob/dev/doc/lz4_Frame_format.md#legacy-frame.

NB. compressed Linux kernel images use a tweaked LZ4 legacy format where the compressed stream is followed by the original (uncompressed) size of the kernel (https://events.static.linuxfound.org/sites/events/files/lcjpcojp13_klee.pdf). This is also supported as a special case.

func OnBlockDoneOption 

func OnBlockDoneOption(handler func(size int)) Option

OnBlockDoneOption is triggered when a block has been processed. For a Writer, it is when is has been compressed, for a Reader, it is when it has been uncompressed.

func SizeOption 

func SizeOption(size uint64) Option

SizeOption sets the size of the original uncompressed data (default=0). It is useful to know the size of the whole uncompressed data stream.

func (Option) String 

func (o Option) String() string

String returns a string representation of the option with its parameter(s).

type Reader 

type Reader struct {
	// contains filtered or unexported fields
}

Reader allows reading an LZ4 stream.

func NewReader 

func NewReader(r io.Reader) *Reader

NewReader returns a new LZ4 frame decoder.

func (*Reader) Apply 

func (r *Reader) Apply(options ...Option) (err error)

func (*Reader) Read 

func (r *Reader) Read(buf []byte) (n int, err error)

func (*Reader) Reset 

func (r *Reader) Reset(reader io.Reader)

Reset clears the state of the Reader r such that it is equivalent to its initial state from NewReader, but instead reading from reader. No access to reader is performed.

func (*Reader) Size 

func (r *Reader) Size() int

Size returns the size of the underlying uncompressed data, if set in the stream.

func (*Reader) WriteTo 

func (r *Reader) WriteTo(w io.Writer) (n int64, err error)

WriteTo efficiently uncompresses the data from the Reader underlying source to w.

type Writer 

type Writer struct {
	// contains filtered or unexported fields
}

Writer allows writing an LZ4 stream.

func NewWriter 

func NewWriter(w io.Writer) *Writer

NewWriter returns a new LZ4 frame encoder.

func (*Writer) Apply 

func (w *Writer) Apply(options ...Option) (err error)

func (*Writer) Close 

func (w *Writer) Close() error

Close closes the Writer, flushing any unwritten data to the underlying writer without closing it.

func (*Writer) Flush added in v4.1.15 

func (w *Writer) Flush() (err error)

Flush any buffered data to the underlying writer immediately.

func (*Writer) ReadFrom 

func (w *Writer) ReadFrom(r io.Reader) (n int64, err error)

ReadFrom efficiently reads from r and compressed into the Writer destination.

func (*Writer) Reset 

func (w *Writer) Reset(writer io.Writer)

Reset clears the state of the Writer w such that it is equivalent to its initial state from NewWriter, but instead writing to writer. Reset keeps the previous options unless overwritten by the supplied ones. No access to writer is performed.

w.Close must be called before Reset or pending data may be dropped.

func (*Writer) Write 

func (w *Writer) Write(buf []byte) (n int, err error)

Source Files

View all Source files

Directories

Path Synopsis
internal
lz4block
Package lz4block provides LZ4 BlockSize types and pools of buffers.
 Package lz4block provides LZ4 BlockSize types and pools of buffers.
lz4errors
lz4stream
Package lz4stream provides the types that support reading and writing LZ4 data streams.
 Package lz4stream provides the types that support reading and writing LZ4 data streams.
xxh32
Package xxh32 implements the very fast XXH hashing algorithm (32 bits version).
 Package xxh32 implements the very fast XXH hashing algorithm (32 bits version).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.