graphsync

go-graphsync

An implementation of the graphsync protocol in go!

Table of Contents

• Background
• Install
• Usage
• Architecture
• Contribute
• License

Background

GraphSync is a protocol for synchronizing IPLD graphs among peers. It allows a host to make a single request to a remote peer for all of the results of traversing an IPLD selector on the remote peer's local IPLD graph.

go-graphsync provides an implementation of the Graphsync protocol in go.

Go-IPLD-Prime

go-graphsync relies on go-ipld-prime to traverse IPLD Selectors in an IPLD graph. go-ipld-prime implements the IPLD specification in go and is an alternative to older implementations such as go-ipld-format and go-ipld-cbor. In order to use go-graphsync, some understanding and use of go-ipld-prime concepts is necessary.

If your existing library (i.e. go-ipfs or go-filecoin) uses these other older libraries, you can largely use go-graphsync without switching to go-ipld-prime across your codebase, but it will require some translations

Install

go-graphsync requires Go >= 1.11 and can be installed using Go modules

Usage

Initializing a GraphSync Exchange

import (
  graphsync "github.com/maurycy/go-graphsync/impl"
  gsnet "github.com/maurycy/go-graphsync/network"
  gsbridge "github.com/maurycy/go-graphsync/ipldbridge"
  ipld "github.com/ipld/go-ipld-prime"
)

var ctx context.Context
var host libp2p.Host
var loader ipld.Loader
var storer ipld.Storer

network := gsnet.NewFromLibp2pHost(host)
ipldBridge := gsbridge.NewIPLDBridge()
exchange := graphsync.New(ctx, network, ipldBridge, loader, storer)

Parameter Notes:

1. context is just the parent context for all of GraphSync
2. network is a network abstraction provided to Graphsync on top of libp2p. This allows graphsync to be tested without the actual network
3. ipldBridge is an IPLD abstraction provided to Graphsync on top of go-ipld-prime. This makes the graphsync library testable in isolation
4. loader is used to load blocks from content ids from the local block store. It's used when RESPONDING to requests from other clients. It should conform to the IPLD loader interface: https://github.com/ipld/go-ipld-prime/blob/master/linking.go
5. storer is used to store incoming blocks to the local block store. It's used when REQUESTING a graphsync query, to store blocks locally once they are validated as part of the correct response. It should conform to the IPLD storer interface: https://github.com/ipld/go-ipld-prime/blob/master/linking.go

Using GraphSync With An IPFS BlockStore

GraphSync provides two convenience functions in the storeutil package for integrating with BlockStore's from IPFS.

import (
  graphsync "github.com/maurycy/go-graphsync/impl"
  gsnet "github.com/maurycy/go-graphsync/network"
  gsbridge "github.com/maurycy/go-graphsync/ipldbridge"
  gsbridge "github.com/maurycy/go-graphsync/storeutil"
  ipld "github.com/ipld/go-ipld-prime"
  blockstore "github.com/ipfs/go-ipfs-blockstore"
)

var ctx context.Context
var host libp2p.Host
var bs blockstore.Blockstore
var storer ipld.Storer

network := gsnet.NewFromLibp2pHost(host)
ipldBridge := gsbridge.NewIPLDBridge()
loader := storeutil.LoaderForBlockstore(bs)
storer := storeutil.StorerForBlockstore(bs)

exchange := graphsync.New(ctx, network, ipldBridge, loader, storer)

Write A Loader An IPFS BlockStore

If you are using a traditional go-ipfs-blockstore, your link loading function looks like this:

type BlockStore interface {
	Get(lnk cid.Cid) (blocks.Block, error)
}

or, more generally:

type Cid2BlockFn func (lnk cid.Cid) (blocks.Block, error)

in go-ipld-prime, the signature for a link loader is as follows:

type Loader func(lnk Link, lnkCtx LinkContext) (io.Reader, error)

go-ipld-prime intentionally keeps its interfaces as abstract as possible to limit dependencies on other ipfs/filecoin specific packages. An IPLD Link is an abstraction for a CID, and IPLD expects io.Reader's rather than an actual block. IPLD provides a cidLink package for working with Links that use CIDs as the underlying data, and it's safe to assume that's the type in use if your code deals only with CIDs. A conversion would look something like this:

import (
   ipld "github.com/ipld/go-ipld-prime"
   cidLink "github.com/ipld/go-ipld-prime/linking/cid"
)

func LoaderFromCid2BlockFn(cid2BlockFn Cid2BlockFn) ipld.Loader {
	return func(lnk ipld.Link, lnkCtx ipld.LinkContext) (io.Reader, error) {
		asCidLink, ok := lnk.(cidlink.Link)
		if !ok {
			return nil, fmt.Errorf("Unsupported Link Type")
		}
		block, err := cid2BlockFn(asCidLink.Cid)
		if err != nil {
			return nil, err
		}
		return bytes.NewReader(block.RawData()), nil
	}
}

Write A Storer From An IPFS BlockStore

If you are using a traditional go-ipfs-blockstore, your storage function looks like this:

type BlockStore interface {
	Put(blocks.Block) error
}

or, more generally:

type BlockStoreFn func (blocks.Block) (error)

in go-ipld-prime, the signature for a link storer is a bit different:

type StoreCommitter func(Link) error
type Storer func(lnkCtx LinkContext) (io.Writer, StoreCommitter, error)

go-ipld-prime stores in two parts to support streaming -- the storer is called and returns an IO.Writer and a function to commit changes when finished. Here's how you can write a storer from a traditional block storing signature.

import (
	blocks "github.com/ipfs/go-block-format"
  ipld "github.com/ipld/go-ipld-prime"
  cidLink "github.com/ipld/go-ipld-prime/linking/cid"
)

func StorerFromBlockStoreFn(blockStoreFn BlockStoreFn) ipld.Storer {
	return func(lnkCtx ipld.LinkContext) (io.Writer, ipld.StoreCommitter, error) {
		var buffer bytes.Buffer
		committer := func(lnk ipld.Link) error {
			asCidLink, ok := lnk.(cidlink.Link)
			if !ok {
				return fmt.Errorf("Unsupported Link Type")
			}
			block := blocks.NewBlockWithCid(buffer.Bytes(), asCidLink.Cid)
			return blockStoreFn(block)
		}
		return &buffer, committer, nil
	}
}

Calling Graphsync

var exchange graphsync.GraphSync
var ctx context.Context
var p peer.ID
var selector ipld.Node
var rootLink ipld.Link

var responseProgress <-chan graphsync.ResponseProgress
var errors <-chan error

responseProgress, errors = exchange.Request(ctx context.Context, p peer.ID, root ipld.Link, selector ipld.Node)

Paramater Notes:

1. ctx is the context for this request. To cancel an in progress request, cancel the context.
2. p is the peer you will send this request to
3. link is an IPLD Link, i.e. a CID (cidLink.Link{Cid})
4. selector is an IPLD selector node. Recommend using selector builders from go-ipld-prime to construct these

Response Type

type ResponseProgress struct {
  Node      ipld.Node // a node which matched the graphsync query
  Path      ipld.Path // the path of that node relative to the traversal start
	LastBlock struct {  // LastBlock stores the Path and Link of the last block edge we had to load.
		ipld.Path
		ipld.Link
	}
}

The above provides both immediate and relevant metadata for matching nodes in a traversal, and is very similar to the information provided by a local IPLD selector traversal in go-ipld-prime

Contribute

PRs are welcome!

Before doing anything heavy, checkout the Graphsync Architecture

Small note: If editing the Readme, please conform to the standard-readme specification.

License

This library is dual-licensed under Apache 2.0 and MIT terms.

Copyright 2019. Protocol Labs, Inc.