car

package

v0.0.0-...-e23051b Latest Latest Go to latest Published: Jun 3, 2023 License: Apache-2.0, MIT Imports: 20 Imported by: 0

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

github.com/sevenrats/boxo

Links

Open Source Insights

README ¶

go-car (go!)

Work with car (Content addressed ARchive) files!

This is a Golang implementation of the CAR specifications, both CARv1 and CARv2.

As a format, there are two major module versions:

go-car/v2 is geared towards reading and writing CARv2 files, and also supports consuming CARv1 files and using CAR files as an IPFS blockstore.
go-car, in the root directory, only supports reading and writing CARv1 files.

Most users should use v2, especially for new software, since the v2 API transparently supports both CAR formats.

Unstable

This package will be refactored and thinned out soon. You can find an implementation that supports indexes and go-ipld-prime there at ipld/go-car.

Usage / Installation

This repository provides a car binary that can be used for creating, extracting, and working with car files.

To install the latest version of car, run:

go install github.com/boxo/ipld/car/cmd/car@latest

More information about this binary is available in cmd/car

Features

CARv2 features:

Generate index from an existing CARv1 file
Wrap CARv1 files into a CARv2 with automatic index generation.
Random-access to blocks in a CAR file given their CID via Read-Only blockstore API, with transparent support for both CARv1 and CARv2
Write CARv2 files via Read-Write blockstore API, with support for appending blocks to an existing CARv2 file, and resumption from a partially written CARv2 files.
Individual access to inner CARv1 data payload and index of a CARv2 file via the Reader API.

API Documentation

See docs on pkg.go.dev.

Examples

Here is a shortlist of other examples from the documentation

Contribute

PRs are welcome!

When editing the Readme, please conform to the standard-readme specification.

License

Apache-2.0/MIT © Protocol Labs

Documentation ¶

Index ¶

func DefaultWalkFunc(nd format.Node) ([]*format.Link, error)
func HeaderSize(h *CarHeader) (uint64, error)
func WriteCar(ctx context.Context, ds format.NodeGetter, roots []cid.Cid, w io.Writer, ...) error
func WriteCarWithWalker(ctx context.Context, ds format.NodeGetter, roots []cid.Cid, w io.Writer, ...) error
func WriteHeader(h *CarHeader, w io.Writer) error
type Block
type CarHeader
- func LoadCar(ctx context.Context, s Store, r io.Reader) (*CarHeader, error)
- func ReadHeader(br *bufio.Reader) (*CarHeader, error)
type CarReader
- func NewCarReader(r io.Reader) (*CarReader, error)
- func (cr *CarReader) Next() (blocks.Block, error)
type Dag
type OnCarHeaderFunc
type OnNewCarBlockFunc
type Option
- func MaxTraversalLinks(MaxTraversalLinks uint64) Option
- func TraverseLinksOnlyOnce() Option
type ReadStore
type SelectiveCar
- func NewSelectiveCar(ctx context.Context, store ReadStore, dags []Dag, opts ...Option) SelectiveCar
- func (sc SelectiveCar) Prepare(userOnNewCarBlocks ...OnNewCarBlockFunc) (SelectiveCarPrepared, error)
- func (sc SelectiveCar) Write(w io.Writer, userOnNewCarBlocks ...OnNewCarBlockFunc) error
type SelectiveCarPrepared
- func (sc SelectiveCarPrepared) Cids() []cid.Cid
- func (sc SelectiveCarPrepared) Dump(ctx context.Context, w io.Writer) error
- func (sc SelectiveCarPrepared) Header() CarHeader
- func (sc SelectiveCarPrepared) Size() uint64
type Store
type WalkFunc

Constants ¶

This section is empty.

Variables ¶

This section is empty.

Functions ¶

func DefaultWalkFunc ¶

func DefaultWalkFunc(nd format.Node) ([]*format.Link, error)

func HeaderSize ¶

func HeaderSize(h *CarHeader) (uint64, error)

func WriteCar ¶

func WriteCar(ctx context.Context, ds format.NodeGetter, roots []cid.Cid, w io.Writer, options ...merkledag.WalkOption) error

func WriteCarWithWalker ¶

func WriteCarWithWalker(ctx context.Context, ds format.NodeGetter, roots []cid.Cid, w io.Writer, walk WalkFunc, options ...merkledag.WalkOption) error

func WriteHeader ¶

func WriteHeader(h *CarHeader, w io.Writer) error

Types ¶

type Block ¶

type Block struct {
	BlockCID cid.Cid
	Data     []byte
	Offset   uint64
	Size     uint64
}

Block is all information and metadata about a block that is part of a car file

type CarHeader ¶

type CarHeader struct {
	Roots   []cid.Cid
	Version uint64
}

func LoadCar ¶

func LoadCar(ctx context.Context, s Store, r io.Reader) (*CarHeader, error)

func ReadHeader ¶

func ReadHeader(br *bufio.Reader) (*CarHeader, error)

type CarReader ¶

type CarReader struct {
	Header *CarHeader
	// contains filtered or unexported fields
}

func NewCarReader ¶

func NewCarReader(r io.Reader) (*CarReader, error)

func (*CarReader) Next ¶

func (cr *CarReader) Next() (blocks.Block, error)

type Dag ¶

type Dag struct {
	Root     cid.Cid
	Selector ipld.Node
}

Dag is a root/selector combo to put into a car

type OnCarHeaderFunc ¶

type OnCarHeaderFunc func(CarHeader) error

OnCarHeaderFunc is called during traversal when the header is created

type OnNewCarBlockFunc ¶

type OnNewCarBlockFunc func(Block) error

OnNewCarBlockFunc is called during traveral when a new unique block is encountered

type Option ¶

type Option func(*options)

Option describes an option which affects behavior when interacting with the interface.

func MaxTraversalLinks ¶

func MaxTraversalLinks(MaxTraversalLinks uint64) Option

MaxTraversalLinks changes the allowed number of links a selector traversal can execute before failing.

Note that setting this option may cause an error to be returned from selector execution when building a SelectiveCar.

func TraverseLinksOnlyOnce ¶

func TraverseLinksOnlyOnce() Option

TraverseLinksOnlyOnce prevents the traversal engine from repeatedly visiting the same links more than once.

This can be an efficient strategy for an exhaustive selector where it's known that repeat visits won't impact the completeness of execution. However it should be used with caution with most other selectors as repeat visits of links for different reasons during selector execution can be valid and necessary to perform full traversal.

type ReadStore ¶

type ReadStore interface {
	Get(context.Context, cid.Cid) (blocks.Block, error)
}

type SelectiveCar ¶

type SelectiveCar struct {
	// contains filtered or unexported fields
}

SelectiveCar is a car file based on root + selector combos instead of just a single root and complete dag walk

func NewSelectiveCar ¶

func NewSelectiveCar(ctx context.Context, store ReadStore, dags []Dag, opts ...Option) SelectiveCar

NewSelectiveCar creates a new SelectiveCar for the given car file based a block store and set of root+selector pairs

func (SelectiveCar) Prepare ¶

func (sc SelectiveCar) Prepare(userOnNewCarBlocks ...OnNewCarBlockFunc) (SelectiveCarPrepared, error)

Prepare traverse a car file and collects data on what is about to be written, but does not actually write the file

func (SelectiveCar) Write ¶

func (sc SelectiveCar) Write(w io.Writer, userOnNewCarBlocks ...OnNewCarBlockFunc) error

type SelectiveCarPrepared ¶

type SelectiveCarPrepared struct {
	SelectiveCar
	// contains filtered or unexported fields
}

SelectiveCarPrepared is a SelectiveCar that has already been traversed, such that it can be written quicker with Dump. It also contains metadata already collection about the Car file like size and number of blocks that go into it

func (SelectiveCarPrepared) Cids ¶

func (sc SelectiveCarPrepared) Cids() []cid.Cid

Cids returns the list of unique block cids that will be written to the car file

func (SelectiveCarPrepared) Dump ¶

func (sc SelectiveCarPrepared) Dump(ctx context.Context, w io.Writer) error

Dump writes the car file as quickly as possible based on information already collected

func (sc SelectiveCarPrepared) Header() CarHeader

Header returns the header for the car file that will be written

func (SelectiveCarPrepared) Size ¶

func (sc SelectiveCarPrepared) Size() uint64

Size returns the total size in bytes of the car file that will be written

type Store ¶

type Store interface {
	Put(context.Context, blocks.Block) error
}

type WalkFunc ¶

type WalkFunc func(format.Node) ([]*format.Link, error)

Source Files ¶

View all Source files

Directories ¶

Path	Synopsis
util
v2 Package car allows inspecting and reading CAR files, described at https://ipld.io/specs/transport/car/.	Package car allows inspecting and reading CAR files, described at https://ipld.io/specs/transport/car/.
blockstore Package blockstore implements the IPFS blockstore interface backed by a CAR file.	Package blockstore implements the IPFS blockstore interface backed by a CAR file.
index package index provides indexing functionality for CARv1 data payload represented as a mapping of CID to offset.	package index provides indexing functionality for CARv1 data payload represented as a mapping of CID to offset.
internal/carv1 Forked from CARv1 to avoid dependency to ipld-prime 0.9.0 due to outstanding upgrades in filecoin.	Forked from CARv1 to avoid dependency to ipld-prime 0.9.0 due to outstanding upgrades in filecoin.
internal/carv1/util
internal/errsort
internal/io
internal/loader
internal/store
storage Package storage provides a CAR abstraction for the github.com/ipld/go-ipld-prime/storage interfaces in the form of a StorageCar.	Package storage provides a CAR abstraction for the github.com/ipld/go-ipld-prime/storage interfaces in the form of a StorageCar.

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL