chunk

package module
v0.0.6 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 25, 2023 License: MIT Imports: 10 Imported by: 122

README

go-ipfs-chunker

go-ipfs-chunker implements data Splitters for go-ipfs.

standard-readme compliant GoDoc Build Status

❗ This repo is no longer maintained.

👉 We highly recommend switching to the maintained version at https://github.com/ipfs/boxo/tree/main/chunker. 🏎️ Good news! There is tooling and documentation to expedite a switch in your repo.

⚠️ If you continue using this repo, please note that security fixes will not be provided (unless someone steps in to maintain it).

📚 Learn more, including how to take the maintainership mantle or ask questions, here.

Summary

go-ipfs-chunker provides the Splitter interface. IPFS splitters read data from a reader an create "chunks". These chunks are used to build the ipfs DAGs (Merkle Tree) and are the base unit to obtain the sums that ipfs uses to address content.

The package provides a SizeSplitter which creates chunks of equal size and it is used by default in most cases, and a rabin fingerprint chunker. This chunker will attempt to split data in a way that the resulting blocks are the same when the data has repetitive patterns, thus optimizing the resulting DAGs.

Table of Contents

Install

go-ipfs-chunker works like a regular Go module:

> go get github.com/ipfs/go-ipfs-chunker

Usage

import "github.com/ipfs/go-ipfs-chunker"

Check the GoDoc documentation

License

MIT © Protocol Labs, Inc.

Documentation

Overview

Package chunk implements streaming block splitters. Splitters read data from a reader and provide byte slices (chunks) The size and contents of these slices depend on the splitting method used.

Index

Constants

View Source 
const (
	// DefaultBlockSize is the chunk size that splitters produce (or aim to).
	//
	// Deprecated: use github.com/ipfs/boxo/chunker.DefaultBlockSize
	DefaultBlockSize int64 = 1024 * 256

	// No leaf block should contain more than 1MiB of payload data ( wrapping overhead aside )
	// This effectively mandates the maximum chunk size
	// See discussion at https://github.com/ipfs/go-ipfs-chunker/pull/21#discussion_r369124879 for background
	//
	// Deprecated: use github.com/ipfs/boxo/chunker.ChunkSizeLimit
	ChunkSizeLimit int = 1048576
)

Variables

View Source 
var (
	// Deprecated: use github.com/ipfs/boxo/chunker.ErrRabinMin
	ErrRabinMin = errors.New("rabin min must be greater than 16")
	// Deprecated: use github.com/ipfs/boxo/chunker.ErrSize
	ErrSize = errors.New("chunker size must be greater than 0")
	// Deprecated: use github.com/ipfs/boxo/chunker.ErrSizeMax
	ErrSizeMax = fmt.Errorf("chunker parameters may not exceed the maximum chunk size of %d", ChunkSizeLimit)
)
View Source 
var IpfsRabinPoly = chunker.Pol(17437180132763653)

IpfsRabinPoly is the irreducible polynomial of degree 53 used by for Rabin.

Deprecated: use github.com/ipfs/boxo/chunker.IpfsRabinPoly

Functions

func Chan deprecated 

func Chan(s Splitter) (<-chan []byte, <-chan error)

Chan returns a channel that receives each of the chunks produced by a splitter, along with another one for errors.

Deprecated: use github.com/ipfs/boxo/chunker.Chan

Types

type Buzhash deprecated added in v0.0.3 

type Buzhash struct {
	// contains filtered or unexported fields
}

Deprecated: use github.com/ipfs/boxo/chunker.Buzhash

func NewBuzhash deprecated added in v0.0.3 

func NewBuzhash(r io.Reader) *Buzhash

Deprecated: use github.com/ipfs/boxo/chunker.NewBuzhash

func (*Buzhash) NextBytes added in v0.0.3 

func (b *Buzhash) NextBytes() ([]byte, error)

func (*Buzhash) Reader added in v0.0.3 

func (b *Buzhash) Reader() io.Reader

type Rabin deprecated 

type Rabin struct {
	// contains filtered or unexported fields
}

Rabin implements the Splitter interface and splits content with Rabin fingerprints.

Deprecated: use github.com/ipfs/boxo/chunker.Rabin

func NewRabin deprecated 

func NewRabin(r io.Reader, avgBlkSize uint64) *Rabin

NewRabin creates a new Rabin splitter with the given average block size.

Deprecated: use github.com/ipfs/boxo/chunker.NewRabin

func NewRabinMinMax deprecated 

func NewRabinMinMax(r io.Reader, min, avg, max uint64) *Rabin

NewRabinMinMax returns a new Rabin splitter which uses the given min, average and max block sizes.

Deprecated: use github.com/ipfs/boxo/chunker.NewRabinMinMax

func (*Rabin) NextBytes 

func (r *Rabin) NextBytes() ([]byte, error)

NextBytes reads the next bytes from the reader and returns a slice.

func (*Rabin) Reader 

func (r *Rabin) Reader() io.Reader

Reader returns the io.Reader associated to this Splitter.

type Splitter deprecated 

type Splitter interface {
	Reader() io.Reader
	NextBytes() ([]byte, error)
}

A Splitter reads bytes from a Reader and creates "chunks" (byte slices) that can be used to build DAG nodes.

Deprecated: use github.com/ipfs/boxo/chunker.Splitter

func DefaultSplitter deprecated 

func DefaultSplitter(r io.Reader) Splitter

DefaultSplitter returns a SizeSplitter with the DefaultBlockSize.

Deprecated: use github.com/ipfs/boxo/chunker.DefaultSplitter

func FromString deprecated 

func FromString(r io.Reader, chunker string) (Splitter, error)

FromString returns a Splitter depending on the given string: it supports "default" (""), "size-{size}", "rabin", "rabin-{blocksize}", "rabin-{min}-{avg}-{max}" and "buzhash".

Deprecated: use github.com/ipfs/boxo/chunker.FromString

func NewSizeSplitter deprecated 

func NewSizeSplitter(r io.Reader, size int64) Splitter

NewSizeSplitter returns a new size-based Splitter with the given block size.

Deprecated: use github.com/ipfs/boxo/chunker.NewSizeSplitter

type SplitterGen deprecated 

type SplitterGen func(r io.Reader) Splitter

SplitterGen is a splitter generator, given a reader.

Deprecated: use github.com/ipfs/boxo/chunker.SplitterGen

func SizeSplitterGen deprecated 

func SizeSplitterGen(size int64) SplitterGen

SizeSplitterGen returns a SplitterGen function which will create a splitter with the given size when called.

Deprecated: use github.com/ipfs/boxo/chunker.SizeSplitterGen

Source Files

View all Source files

Directories

Path Synopsis
This file generates bytehash LUT
 This file generates bytehash LUT

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.