Documentation
¶
Overview ¶
Package chunk implements streaming block splitters. Splitters read data from a reader and provide byte slices (chunks) The size and contents of these slices depend on the splitting method used.
Index ¶
Constants ¶
const ( // DefaultBlockSize is the chunk size that splitters produce (or aim to). DefaultBlockSize int64 = 1024 * 256 // No leaf block should contain more than 1MiB of payload data ( wrapping overhead aside ) // This effectively mandates the maximum chunk size // See discussion at https://github.com/ipfs/boxo/chunker/pull/21#discussion_r369124879 for background ChunkSizeLimit int = 1048576 )
Variables ¶
var ( ErrRabinMin = errors.New("rabin min must be greater than 16") ErrSize = errors.New("chunker size must be greater than 0") ErrSizeMax = fmt.Errorf("chunker parameters may not exceed the maximum chunk size of %d", ChunkSizeLimit) )
var IpfsRabinPoly = chunker.Pol(17437180132763653)
IpfsRabinPoly is the irreducible polynomial of degree 53 used by for Rabin.
Functions ¶
Types ¶
type Rabin ¶
type Rabin struct {
// contains filtered or unexported fields
}
Rabin implements the Splitter interface and splits content with Rabin fingerprints.
func NewRabinMinMax ¶
NewRabinMinMax returns a new Rabin splitter which uses the given min, average and max block sizes.
type Splitter ¶
A Splitter reads bytes from a Reader and creates "chunks" (byte slices) that can be used to build DAG nodes.
func DefaultSplitter ¶
DefaultSplitter returns a SizeSplitter with the DefaultBlockSize.
func FromString ¶
FromString returns a Splitter depending on the given string: it supports "default" (""), "size-{size}", "rabin", "rabin-{blocksize}", "rabin-{min}-{avg}-{max}" and "buzhash".
type SplitterGen ¶
SplitterGen is a splitter generator, given a reader.
func SizeSplitterGen ¶
func SizeSplitterGen(size int64) SplitterGen
SizeSplitterGen returns a SplitterGen function which will create a splitter with the given size when called.
Source Files
Directories