compress: github.com/klauspost/compress/fse Index | Examples | Files

package fse

import "github.com/klauspost/compress/fse"

Package fse provides Finite State Entropy encoding and decoding.

Finite State Entropy encoding provides a fast near-optimal symbol encoding/decoding for byte blocks as implemented in zstd.

See https://github.com/klauspost/compress/tree/master/fse for more information.

Index

Examples

Package Files

bitreader.go bitwriter.go bytereader.go compress.go decompress.go fse.go

Variables

var (
    // ErrIncompressible is returned when input is judged to be too hard to compress.
    ErrIncompressible = errors.New("input is not compressible")

    // ErrUseRLE is returned from the compressor when the input is a single byte value repeated.
    ErrUseRLE = errors.New("input is single value repeated")
)

func Compress Uses

func Compress(in []byte, s *Scratch) ([]byte, error)

Compress the input bytes. Input must be < 2GB. Provide a Scratch buffer to avoid memory allocations. Note that the output is also kept in the scratch buffer. If input is too hard to compress, ErrIncompressible is returned. If input is a single byte value repeated ErrUseRLE is returned.

Code:

// Read data
data, err := ioutil.ReadFile("../testdata/e.txt")
if err != nil {
    panic(err)
}

// Create re-usable scratch buffer.
var s Scratch
b, err := Compress(data, &s)
if err != nil {
    panic(err)
}
fmt.Printf("Compress: %d -> %d bytes (%.2f:1)\n", len(data), len(b), float64(len(data))/float64(len(b)))

Output:

Compress: 100003 -> 41564 bytes (2.41:1)

func Decompress Uses

func Decompress(b []byte, s *Scratch) ([]byte, error)

Decompress a block of data. You can provide a scratch buffer to avoid allocations. If nil is provided a temporary one will be allocated. It is possible, but by no way guaranteed that corrupt data will return an error. It is up to the caller to verify integrity of the returned data. Use a predefined Scrach to set maximum acceptable output size.

Code:

// Read data
data, err := ioutil.ReadFile("../testdata/e.txt")
if err != nil {
    panic(err)
}

// Create re-usable scratch buffer.
var s Scratch
b, err := Compress(data, &s)
if err != nil {
    panic(err)
}

// Since we use the output of compression, it cannot be used as output for decompression.
s.Out = make([]byte, 0, len(data))
d, err := Decompress(b, &s)
if err != nil {
    panic(err)
}
fmt.Printf("Input matches: %t\n", bytes.Equal(d, data))

Output:

Input matches: true

type Scratch Uses

type Scratch struct {

    // Out is output buffer.
    // If the scratch is re-used before the caller is done processing the output,
    // set this field to nil.
    // Otherwise the output buffer will be re-used for next Compression/Decompression step
    // and allocation will be avoided.
    Out []byte

    // DecompressLimit limits the maximum decoded size acceptable.
    // If > 0 decompression will stop when approximately this many bytes
    // has been decoded.
    // If 0, maximum size will be 2GB.
    DecompressLimit int

    // MaxSymbolValue will override the maximum symbol value of the next block.
    MaxSymbolValue uint8

    // TableLog will attempt to override the tablelog for the next block.
    TableLog uint8
    // contains filtered or unexported fields
}

Scratch provides temporary storage for compression and decompression.

func (*Scratch) Histogram Uses

func (s *Scratch) Histogram() []uint32

Histogram allows to populate the histogram and skip that step in the compression, It otherwise allows to inspect the histogram when compression is done. To indicate that you have populated the histogram call HistogramFinished with the value of the highest populated symbol, as well as the number of entries in the most populated entry. These are accepted at face value. The returned slice will always be length 256.

func (*Scratch) HistogramFinished Uses

func (s *Scratch) HistogramFinished(maxSymbol uint8, maxCount int)

HistogramFinished can be called to indicate that the histogram has been populated. maxSymbol is the index of the highest set symbol of the next data segment. maxCount is the number of entries in the most populated entry. These are accepted at face value.

Package fse imports 5 packages (graph) and is imported by 5 packages. Updated 2020-06-06. Refresh now. Tools for package owners.