linking

type BlockReadOpener ¶

type BlockReadOpener func(LinkContext, datamodel.Link) (io.Reader, error)

BlockReadOpener defines the shape of a function used to open a reader for a block of data.

In a content-addressed system, the Link parameter should be only determiner of what block body is returned.

The LinkContext may be zero, or may be used to carry extra information: it may be used to carry info which hints at different storage pools; it may be used to carry authentication data; etc. (Any such behaviors are something that a BlockReadOpener implementation will needs to document at a higher detail level than this interface specifies. In this interface, we can only note that it is possible to pass such information opaquely via the LinkContext or by attachments to the general-purpose Context it contains.) The LinkContext should not have effect on the block body returned, however; at most should only affect data availability (e.g. whether any block body is returned, versus an error).

Reads are cancellable by cancelling the LinkContext.Context.

Other parts of the IPLD library suite (such as the traversal package, and all its functions) will typically take a Context as a parameter or piece of config from the caller, and will pass that down through the LinkContext, meaning this can be used to carry information as well as cancellation control all the way through the system.

BlockReadOpener is typically not used directly, but is instead composed in a LinkSystem and used via the methods of LinkSystem. LinkSystem methods will helpfully handle the entire process of opening block readers, verifying the hash of the data stream, and applying a Decoder to build Nodes -- all as one step.

BlockReadOpener implementations are not required to validate that the contents which will be streamed out of the reader actually match and hash in the Link parameter before returning. (This is something that the LinkSystem composition will handle if you're using it.)

BlockReadOpener can also be created out of storage.ReadableStorage and attached to a LinkSystem via the LinkSystem.SetReadStorage method.

Users of a BlockReadOpener function should also check the io.Reader for matching the io.Closer interface, and use the Close function as appropriate if present.

type BlockWriteCommitter ¶

type BlockWriteCommitter func(datamodel.Link) error

BlockWriteCommitter defines the shape of a function which, together with BlockWriteOpener, handles the writing and "committing" of a write to a content-addressable storage system.

BlockWriteCommitter is a function which is will be called at the end of a write process. It should flush any buffers and close the io.Writer which was made available earlier from the BlockWriteOpener call that also returned this BlockWriteCommitter.

BlockWriteCommitter takes a Link parameter. This Link is expected to be a reasonable hash of the content, so that the BlockWriteCommitter can use this to commit the data to storage in a content-addressable fashion. See the documentation of BlockWriteOpener for more description of this and an example of how this is likely to be reduced to practice.

type BlockWriteOpener ¶

type BlockWriteOpener func(LinkContext) (io.Writer, BlockWriteCommitter, error)

BlockWriteOpener defines the shape of a function used to open a writer into which data can be streamed, and which will eventually be "commited". Committing is done using the BlockWriteCommitter returned by using the BlockWriteOpener, and finishes the write along with requiring stating the Link which should identify this data for future reading.

The LinkContext may be zero, or may be used to carry extra information: it may be used to carry info which hints at different storage pools; it may be used to carry authentication data; etc.

Writes are cancellable by cancelling the LinkContext.Context.

Other parts of the IPLD library suite (such as the traversal package, and all its functions) will typically take a Context as a parameter or piece of config from the caller, and will pass that down through the LinkContext, meaning this can be used to carry information as well as cancellation control all the way through the system.

BlockWriteOpener is typically not used directly, but is instead composed in a LinkSystem and used via the methods of LinkSystem. LinkSystem methods will helpfully handle the entire process of traversing a Node tree, encoding this data, hashing it, streaming it to the writer, and committing it -- all as one step.

BlockWriteOpener implementations are expected to start writing their content immediately, and later, the returned BlockWriteCommitter should also be able to expect that the Link which it is given is a reasonable hash of the content. (To give an example of how this might be efficiently implemented: One might imagine that if implementing a disk storage mechanism, the io.Writer returned from a BlockWriteOpener will be writing a new tempfile, and when the BlockWriteCommiter is called, it will flush the writes and then use a rename operation to place the tempfile in a permanent path based the Link.)

BlockWriteOpener can also be created out of storage.WritableStorage and attached to a LinkSystem via the LinkSystem.SetWriteStorage method.

type ErrHashMismatch ¶

type ErrHashMismatch struct {
	Actual   datamodel.Link
	Expected datamodel.Link
}

ErrHashMismatch is the error returned when loading data and verifying its hash and finding that the loaded data doesn't re-hash to the expected value. It is typically seen returned by functions like LinkSystem.Load or LinkSystem.Fill.

func (e ErrHashMismatch) Error() string

type ErrLinkingSetup ¶

type ErrLinkingSetup struct {
	Detail string // Perhaps an enum here as well, which states which internal function was to blame?
	Cause  error
}

ErrLinkingSetup is returned by methods on LinkSystem when some part of the system is not set up correctly, or when one of the components refuses to handle a Link or LinkPrototype given. (It is not yielded for errors from the storage nor codec systems once they've started; those errors rise without interference.)

func (e ErrLinkingSetup) Error() string

func (e ErrLinkingSetup) Unwrap() error

type LinkContext ¶

type LinkContext struct {
	// Ctx is the familiar golang Context pattern.
	// Use this for cancellation, or attaching additional info
	// (for example, perhaps to pass auth tokens through to the storage functions).
	Ctx context.Context

	// Path where the link was encountered.  May be zero.
	//
	// Functions in the traversal package will set this automatically.
	LinkPath datamodel.Path

	// When traversing data or encoding: the Node containing the link --
	// it may have additional type info, etc, that can be accessed.
	// When building / decoding: not present.
	//
	// Functions in the traversal package will set this automatically.
	LinkNode datamodel.Node

	// When building data or decoding: the NodeAssembler that will be receiving the link --
	// it may have additional type info, etc, that can be accessed.
	// When traversing / encoding: not present.
	//
	// Functions in the traversal package will set this automatically.
	LinkNodeAssembler datamodel.NodeAssembler

	// Parent of the LinkNode.  May be zero.
	//
	// Functions in the traversal package will set this automatically.
	ParentNode datamodel.Node
}

LinkContext is a structure carrying ancilary information that may be used while loading or storing data -- see its usage in BlockReadOpener, BlockWriteOpener, and in the methods on LinkSystem which handle loading and storing data.

A zero value for LinkContext is generally acceptable in any functions that use it. In this case, any operations that need a context.Context will quietly use Context.Background (thus being uncancellable) and simply have no additional information to work with.

type LinkSystem ¶

type LinkSystem struct {
	EncoderChooser     func(datamodel.LinkPrototype) (codec.Encoder, error)
	DecoderChooser     func(datamodel.Link) (codec.Decoder, error)
	HasherChooser      func(datamodel.LinkPrototype) (hash.Hash, error)
	StorageWriteOpener BlockWriteOpener
	StorageReadOpener  BlockReadOpener
	TrustedStorage     bool
	NodeReifier        NodeReifier
	KnownReifiers      map[string]NodeReifier
}

LinkSystem is a struct that composes all the individual functions needed to load and store content addressed data using IPLD -- encoding functions, hashing functions, and storage connections -- and then offers the operations a user wants -- Store and Load -- as methods.

Typically, the functions which are fields of LinkSystem are not used directly by users (except to set them, when creating the LinkSystem), and it's the higher level operations such as Store and Load that user code then calls.

The most typical way to get a LinkSystem is from the linking/cid package, which has a factory function called DefaultLinkSystem. The LinkSystem returned by that function will be based on CIDs, and use the multicodec registry and multihash registry to select encodings and hashing mechanisms. The BlockWriteOpener and BlockReadOpener must still be provided by the user; otherwise, only the ComputeLink method will work.

Some implementations of BlockWriteOpener and BlockReadOpener may be found in the storage package. Applications are also free to write their own. Custom wrapping of BlockWriteOpener and BlockReadOpener are also common, and may be reasonable if one wants to build application features that are block-aware.

func (lsys *LinkSystem) ComputeLink(lp datamodel.LinkPrototype, n datamodel.Node) (datamodel.Link, error)

func (lsys *LinkSystem) Fill(lnkCtx LinkContext, lnk datamodel.Link, na datamodel.NodeAssembler) error

func (lsys *LinkSystem) Load(lnkCtx LinkContext, lnk datamodel.Link, np datamodel.NodePrototype) (datamodel.Node, error)

// Let's say we want to load this link (it's the same one we created in ExampleLinkSystem_Store).
cid, _ := cid.Decode("bafyrgqhai26anf3i7pips7q22coa4sz2fr4gk4q4sqdtymvvjyginfzaqewveaeqdh524nsktaq43j65v22xxrybrtertmcfxufdam3da3hbk")
lnk := cidlink.Link{Cid: cid}

// Let's get a LinkSystem.  We're going to be working with CID links,
//  so let's get the default LinkSystem that's ready to work with those.
// (This is the same as we did in ExampleLinkSystem_Store.)
lsys := cidlink.DefaultLinkSystem()

// We need somewhere to go looking for any of the data we might want to load!
//  We'll use an in-memory store for this.  (It's a package scoped variable.)
//   (This particular memory store was filled with the data we'll load earlier, during ExampleLinkSystem_Store.)
//  You can use any kind of storage system here;
//   or if you need even more control, you could also write a function that conforms to the linking.BlockReadOpener interface.
lsys.SetReadStorage(&store)

// We'll need to decide what in-memory implementation of datamodel.Node we want to use.
//  Here, we'll use the "basicnode" implementation.  This is a good getting-started choice.
//   But you could also use other implementations, or even a code-generated type with special features!
np := basicnode.Prototype.Any

// Before we use the LinkService, NOTE:
//  There's a side-effecting import at the top of the file.  It's for the dag-cbor codec.
//  See the comments in ExampleLinkSystem_Store for more discussion of this and why it's important.

// Apply the LinkSystem, and ask it to load our link!
n, err := lsys.Load(
	linking.LinkContext{}, // The zero value is fine.  Configure it it you want cancellability or other features.
	lnk,                   // The Link we want to load!
	np,                    // The NodePrototype says what kind of Node we want as a result.
)
if err != nil {
	panic(err)
}

// Tada!  We have the data as node that we can traverse and use as desired.
fmt.Printf("we loaded a %s with %d entries\n", n.Kind(), n.Length())

Output:

we loaded a map with 1 entries

func (lsys *LinkSystem) LoadPlusRaw(lnkCtx LinkContext, lnk datamodel.Link, np datamodel.NodePrototype) (datamodel.Node, []byte, error)

func (lsys *LinkSystem) LoadRaw(lnkCtx LinkContext, lnk datamodel.Link) ([]byte, error)

func (lsys *LinkSystem) MustComputeLink(lp datamodel.LinkPrototype, n datamodel.Node) datamodel.Link

func (lsys *LinkSystem) MustFill(lnkCtx LinkContext, lnk datamodel.Link, na datamodel.NodeAssembler)

func (lsys *LinkSystem) MustLoad(lnkCtx LinkContext, lnk datamodel.Link, np datamodel.NodePrototype) datamodel.Node

func (lsys *LinkSystem) MustStore(lnkCtx LinkContext, lp datamodel.LinkPrototype, n datamodel.Node) datamodel.Link

func (lsys *LinkSystem) SetReadStorage(store storage.ReadableStorage)

func (lsys *LinkSystem) SetWriteStorage(store storage.WritableStorage)

func (lsys *LinkSystem) Store(lnkCtx LinkContext, lp datamodel.LinkPrototype, n datamodel.Node) (datamodel.Link, error)

// Creating a Link is done by choosing a concrete link implementation (typically, CID),
//  getting a LinkSystem that knows how to work with that, and then using the LinkSystem methods.

// Let's get a LinkSystem.  We're going to be working with CID links,
//  so let's get the default LinkSystem that's ready to work with those.
lsys := cidlink.DefaultLinkSystem()

// We want to store the serialized data somewhere.
//  We'll use an in-memory store for this.  (It's a package scoped variable.)
//  You can use any kind of storage system here;
//   or if you need even more control, you could also write a function that conforms to the linking.BlockWriteOpener interface.
lsys.SetWriteStorage(&store)

// To create any links, first we need a LinkPrototype.
// This gathers together any parameters that might be needed when making a link.
// (For CIDs, the version, the codec, and the multihash type are all parameters we'll need.)
// Often, you can probably make this a constant for your whole application.
lp := cidlink.LinkPrototype{Prefix: cid.Prefix{
	Version:  1,    // Usually '1'.
	Codec:    0x71, // 0x71 means "dag-cbor" -- See the multicodecs table: https://github.com/multiformats/multicodec/
	MhType:   0x13, // 0x20 means "sha2-512" -- See the multicodecs table: https://github.com/multiformats/multicodec/
	MhLength: 64,   // sha2-512 hash has a 64-byte sum.
}}

// And we need some data to link to!  Here's a quick piece of example data:
n := fluent.MustBuildMap(basicnode.Prototype.Map, 1, func(na fluent.MapAssembler) {
	na.AssembleEntry("hello").AssignString("world")
})

// Before we use the LinkService, NOTE:
//  There's a side-effecting import at the top of the file.  It's for the dag-cbor codec.
//  The CID LinkSystem defaults use a global registry called the multicodec table;
//  and the multicodec table is populated in part by the dag-cbor package when it's first imported.
// You'll need that side-effecting import, too, to copy this example.
//  It can happen anywhere in your program; once, in any package, is enough.
//  If you don't have this import, the codec will not be registered in the multicodec registry,
//  and when you use the LinkSystem we got from the cidlink package, it will return an error of type ErrLinkingSetup.
// If you initialize a custom LinkSystem, you can control this more directly;
//  these registry systems are only here as defaults.

// Now: time to apply the LinkSystem, and do the actual store operation!
lnk, err := lsys.Store(
	linking.LinkContext{}, // The zero value is fine.  Configure it it you want cancellability or other features.
	lp,                    // The LinkPrototype says what codec and hashing to use.
	n,                     // And here's our data.
)
if err != nil {
	panic(err)
}

// That's it!  We got a link.
fmt.Printf("link: %s\n", lnk)
fmt.Printf("concrete type: `%T`\n", lnk)

// Remember: the serialized data was also stored to the 'store' variable as a side-effect.
//  (We set this up back when we customized the LinkSystem.)
//  We'll pick this data back up again in the example for loading.

Output:

link: bafyrgqhai26anf3i7pips7q22coa4sz2fr4gk4q4sqdtymvvjyginfzaqewveaeqdh524nsktaq43j65v22xxrybrtertmcfxufdam3da3hbk
concrete type: `cidlink.Link`

type NodeReifier ¶

type NodeReifier func(LinkContext, datamodel.Node, *LinkSystem) (datamodel.Node, error)

NodeReifier defines the shape of a function that given a node with no schema or a basic schema, constructs Advanced Data Layout node

The LinkSystem itself is passed to the NodeReifier along with a link context because Node interface methods on an ADL may actually traverse links to other pieces of context addressed data that need to be loaded with the Link system

A NodeReifier return one of three things: - original node, no error = no reification occurred, just use original node - reified node, no error = the simple node was converted to an ADL - nil, error = the simple node should have been converted to an ADL but something went wrong when we tried to do so