README
¶
go-btfs-chunker
go-btfs-chunker implements data Splitters for go-btfs.
go-btfs-chunker provides the Splitter interface. BTFS splitters read data from a reader an create "chunks". These chunks are used to build the BTFS DAGs (Merkle Tree) and are the base unit to obtain the sums that BTFS uses to address content.
The package provides a SizeSplitter which creates chunks of equal size and it is used by default in most cases, and a rabin fingerprint chunker. This chunker will attempt to split data in a way that the resulting blocks are the same when the data has repetitive patterns, thus optimizing the resulting DAGs.
Table of Contents
Install
go-btfs-chunker works like a regular Go module:
> go get github.com/bittorrent/go-btfs-chunker
Usage
import "github.com/bittorrent/go-btfs-chunker"
Contribute
PRs accepted.
Small note: If editing the README, please conform to the standard-readme specification.
License
MIT © TRON-US.
Documentation
¶
Overview ¶
Package chunk implements streaming block splitters. Splitters read data from a reader and provide byte slices (chunks) The size and contents of these slices depend on the splitting method used.
Index ¶
- Constants
- Variables
- func Chan(s Splitter) (<-chan []byte, <-chan error)
- func IsReedSolomon(chunker string) bool
- func NewReedSolomonSplitter(r io.Reader, numData, numParity, size uint64) (*reedSolomonSplitter, error)
- type Buzhash
- type MetaSplitter
- type MultiSplitter
- type Rabin
- type RsMetaMap
- type Splitter
- type SplitterGen
Constants ¶
const ( PrefixForDefault = "default" PrefixForSize = "size-" PrefixForRabin = "rabin" PrefixForReedSolomon = "reed-solomon" // DefaultBlockSize is the chunk size that splitters produce (or aim to). DefaultBlockSize int64 = 1024 * 256 // No leaf block should contain more than 1MiB of payload data ( wrapping overhead aside ) // This effectively mandates the maximum chunk size // See discussion at https://github.com/ipfs/go-ipfs-chunker/pull/21#discussion_r369124879 for background ChunkSizeLimit int = 1048576 )
const ( DefaultReedSolomonDataShards = 10 DefaultReedSolomonParityShards = 20 DefaultReedSolomonShardSize = DefaultBlockSize )
Variables ¶
var ( ErrRabinMin = errors.New("rabin min must be greater than 16") ErrSize = errors.New("chunker size must be greater than 0") ErrSizeMax = fmt.Errorf("chunker parameters may not exceed the maximum chunk size of %d", ChunkSizeLimit) )
var IpfsRabinPoly = chunker.Pol(17437180132763653)
IpfsRabinPoly is the irreducible polynomial of degree 53 used by for Rabin.
Functions ¶
func Chan ¶
Chan returns a channel that receives each of the chunks produced by a splitter, along with another one for errors.
func IsReedSolomon ¶
Types ¶
type Buzhash ¶
type Buzhash struct {
// contains filtered or unexported fields
}
func NewBuzhash ¶
type MetaSplitter ¶
type MetaSplitter struct {
// contains filtered or unexported fields
}
func (*MetaSplitter) ChunkSize ¶
func (ms *MetaSplitter) ChunkSize() uint64
ChunkSize returns the chunk size of this Splitter.
func (*MetaSplitter) MetaData ¶
func (ms *MetaSplitter) MetaData() interface{}
MetaData returns metadata object from this chunker (none).
func (*MetaSplitter) NextBytes ¶
func (ms *MetaSplitter) NextBytes() ([]byte, error)
NextBytes produces a new chunk.
func (*MetaSplitter) Reader ¶
func (ms *MetaSplitter) Reader() io.Reader
Reader returns the io.Reader associated to this Splitter.
func (*MetaSplitter) SetIsDir ¶
func (rss *MetaSplitter) SetIsDir(v bool)
type MultiSplitter ¶
A MultiSplitter encapsulates multiple splitters useful for concurrent reading of chunks and also specialized dag building schemas. Each MultiSplitter also provides Splitter-compatible interface to read sequentially (the Splitter-default way).
type Rabin ¶
type Rabin struct {
// contains filtered or unexported fields
}
Rabin implements the Splitter interface and splits content with Rabin fingerprints.
func NewRabinMinMax ¶
NewRabinMinMax returns a new Rabin splitter which uses the given min, average and max block sizes.
func (*Rabin) MetaData ¶
func (r *Rabin) MetaData() interface{}
MetaData returns metadata object from this chunker (none).
type RsMetaMap ¶
func GetRsMetaMapFromString ¶
type Splitter ¶
type Splitter interface {
Reader() io.Reader
NextBytes() ([]byte, error)
ChunkSize() uint64
MetaData() interface{}
SetIsDir(bool)
}
A Splitter reads bytes from a Reader and creates "chunks" (byte slices) that can be used to build DAG nodes.
func DefaultSplitter ¶
DefaultSplitter returns a SizeSplitter with the DefaultBlockSize.
func FromString ¶
FromString returns a Splitter depending on the given string: it supports "default" (""), "size-{size}", "rabin", "rabin-{blocksize}", "rabin-{min}-{avg}-{max}", "reed-solomon", "reed-solomon-{#data}-{#parity}-{size}" and "buzhash".
type SplitterGen ¶
SplitterGen is a splitter generator, given a reader.
func MetaSplitterGen ¶
func MetaSplitterGen(size int64) SplitterGen
func SizeSplitterGen ¶
func SizeSplitterGen(size int64) SplitterGen
SizeSplitterGen returns a SplitterGen function which will create a splitter with the given size when called.
Source Files
Directories