README
¶
fpc
fpc is a Go implementation of Burtscher and Ratanaworabhan's 'FPC' algorithm for compressing a stream of floating point data.
Why?
The FPC algorithm can losslessly encode and decode huge amounts of floating-point data very quickly. It scales well to gigabyte-per-second streams. Compression ratios are better than just about any generic compressor like gzip or bzip, and compression and decompression throughput are much better (like, 8x to 300x faster) than other algorithms. For more on this, the paper introducing FPC is really readable - I highly recommend it!
Usage
fpc provides a Writer and a Reader, following the pattern set by
the Go standard library's compression packages. The Writer wraps an
io.Writer that you want to write compressed data into, and the Reader
wraps an io.Reader that you want to read compressed data out of.
Since FPC encodes streams of float64s, they impose some additional
expectations on callers: when calling Reader.Read(p []byte) or
Writer.Write(p []byte), the length of p must be a multiple of 8,
to match the expectation that the bytes represent a stream of 8-byte
float64s.
In addition, utility methods are provided: Reader has a
ReadFloats(fs []float64) (int, error) method which will read bytes
from its underlying source, parse them as float64s, put them in fs,
and return the number of float64s it placed in fs. When it reaches the
end of the compressed stream, it will return 0, io.EOF.
Similarly, Writer has a WriteFloat(f float64) error method which
writes a single float64 to the compressed stream.
Performance
In benchmarks on a fairly vanilla laptop, reading or writing from an
in-memory stream, fpc is able to encode at about 1.2 gigabytes per
second, and it can decode at about 0.9 gigabytes per
second. Benchmarks can be run on your own hardware with go test -bench "Read|Write" ..
Documentation
¶
Index ¶
Constants ¶
const ( DefaultCompression = 10 // The reference implementation uses 255 for a max compression, but that // hardly seems realistic: merely 32 will require 68 gigabytes of working // memory to compute hashes. Beyond 35 we start hitting panics. MaxCompression = 32 )
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type DataError ¶
type DataError string
A DataError is returned when the FPC data is found to be syntactically invalid.
type Reader ¶
type Reader struct {
// contains filtered or unexported fields
}
A Reader provides io.Reader-style access to a stream of FPC compressed data.
func NewReader ¶
NewReader creates a new Reader which reads and decompresses FPC data from the given io.Reader.
func (*Reader) Read ¶
Read reads from up to (len(buf) / 8) IEEE 754 64-bit floating point values into buf. It is an error to provide a buf whose length is not a multiple of 8, because that would prevent encoding of the read float64s.
If more values might be available, Read will return len(buf), nil. If no more values are available, Read will return with err==io.EOF
func (*Reader) ReadFloat ¶
ReadFloat will read data from the underlying io.Reader until it has read enough data to provide a float64, decodes that data, and returns the decoded float64. If an error is encountered while reading, it returns 0 and that error. If no more values are available, ReadFloat will return with err==io.EOF.
type Writer ¶
type Writer struct {
// contains filtered or unexported fields
}
A Writer is an io.WriteCloser which FPC-compresses data it receives and writes it to an underlying writer, w. Writes to a Writer are
func NewWriter ¶
NewWriter makes a new Writer which writes compressed data to w using the default compression level.
func NewWriterLevel ¶
NewWriterLevel makes a new Writer which writes compressed data to w using a provided compression level. Higher compression levels will result in more compressed data, but require exponentially more memory. The space required is O(2^level) bytes. NewWriterLevel returns an error if an invalid compression level is provided.
func (*Writer) Close ¶
Close will flush the Writer and make any subsequent writes return errors. It does not close the underlying io.Writer which w is delegating to.
func (*Writer) Flush ¶
Flush will make sure all internally-buffered values are written to w. FPC's format specifies that data get written in blocks; calling Flush will write the current data to a block, even if it results in a partial block.
Flush does not flush the underlying io.Writer which w is delegating to.
func (*Writer) Write ¶
Write interprets b as a stream of byte-encoded, 64-bit IEEE 754 floating point values. The length of b must be a multiple of 8 in order to match this expectation.
func (*Writer) WriteFloat ¶
WriteFloat writes a single float64 value to the encoded stream.
Source Files
Directories