frisbii

package module
v0.5.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 5, 2024 License: Apache-2.0, MIT Imports: 36 Imported by: 1

README

Frisbii

An experimental minimal IPLD data provider for the IPFS network

Frisbii is currently a simple HTTP server that conforms minimally to the IPFS Trustless Gateway specification, serving IPLD data in CAR format to HTTP clients.

Content routing, when enabled, is delegated to the Interpletary Network Indexer (IPNI) service. On startup, Frisbii can announce its content to IPNI, allowing it to be discovered by clients that query the indexer for that content.

Frisbii does not directly interact with the IPFS DHT, which limits discovery potential for clients which primarily use the DHT. However, with increasing reliance on IPNI for content discovery, this will change over time. In particular, Lassie, a universal IPFS and Filecoin retrieval client, relies on IPNI for content discovery.

Installation

To install the latest version of Frisbii, run:

go install github.com/ipld/frisbii/cmd/frisbii@latest

Alternatively, binaries are available for download on the releases page.

Usage

frisbii --announce=roots --car=/path/to/file.car

Note that for announcements to be successful, Frisbii must be able to determine its public address. You may need to supply a --listen argument with a public address, or use --public-addr to override the address that Frisbii determines for itself.

Full argument list:

  • --car - path to one or more CAR files to serve, this can be a plain path, a glob path to match multiple files, and --car can be supplied multiple times.
  • --announce - announce the given roots to IPNI on startup. Can be roots or none. Defaults to none.
  • --listen - hostname and port to listen on. Defaults to :3747.
  • --public-addr - multiaddr or URL of this server as seen by the indexer and other peers if it is different to the listen address. Defaults address of the server once started (typically the value of --listen).
  • --log-file - path to file to append HTTP request and error logs to. See Log format for details of the log format. Defaults to stdout.
  • --max-response-duration - maximum duration to spend responding to a request. Defaults to 5m.
  • --max-response-bytes - maximum size of a response from IPNI. Defaults to 100MiB.
  • --compression-level - compression level to use for HTTP response data where the client accepts it. 0-9, 0 is no compression, 9 is maximum compression. Defaults to 0 (none).
  • --verbose - enable verbose logging. Defaults to false. Same as using GOLOG_LOG_LEVEL=debug as an environment variable. GOLOG_LOG_LEVEL can be used for more fine-grained control of log output.
  • --help - show help.
CAR files
  • go-car can be used to author, manipulate and inspect CAR files.
  • Kubo's dag export command can be used to export a CAR file from an IPFS node, although this is limited to complete DAGs.
  • Lassie can be used to export a CAR file from the network.

Both CARv1 and CARv2 formats are usable by Frisbii. However, on startup, Frisbii will need to generate an index in memory for a CARv1. So for faster start-up times it is recommended that you start Frisbii with CARv2 files (using go-car this can be done with car index input.car > output.car).

Using --anounce=roots will announce the roots of all CARs loaded by Frisbii to the indexer. Other blocks are not announced, and will not be discoverable by clients that query the indexer for that content, however they are served by Frisbii when requested directly or as part of a DAG whose root has been advertised.

Library usage

See https://pkg.go.dev/github.com/ipld/frisbii for full documentation.

NewFrisbiiServer() can be used to create a new server given a LinkSystem as a source of IPLD data.

Log format

Frisbii logs HTTP requests and errors to a log file that is roughly equivalent to a standard nginx or Apache log format; that is, a space-separated list of elements, where the elements that may contain spaces are quoted. The format of each line can be specified as:

%s %s %s "%s" %d %d %d %s "%s" "%s"

Where the elements are:

  1. RFC 3339 timestamp
  2. Remote address
  3. Method
  4. Path
  5. Response status code
  6. Response duration (in milliseconds)
  7. Response size
  8. Compression ratio (or - if no compression)
  9. User agent
  10. Error (or "" if no error)

Further Development

The goal of Frisbii is to be maximally minimal according to user need. It is not intended to be a full IPFS node, but rather a simple server that can be used to serve IPLD data. However, the limitations of HTTP as the only transport, the restrictions within the current minimal Trustless Gateway implementation, and reliance on IPNI for content announcement present some challenges for data provision to a wide audience.

Some areas for possible further development (perhaps by you!) include:

  • Support for additional transports, including Graphsync over a minimal DataTransfer implementation to mirror the default transport functionality of Filecoin storage providers, and Bitswap for the widest possible compatibility with existing IPFS clients. Unfortunately both transports introduce considerable complexity.
  • Support for P2P message sending for IPNI, so announcements can be made to multiple indexers via pubsub.
  • DHT support.
  • Support for alternative data storage backends.
  • More advanced announcement options, including all CIDs, or a user-supplied list of CIDs.

License

Apache-2.0/MIT © Protocol Labs

Documentation

Index

Constants

View Source 
const ContextID = "frisbii"

Variables

This section is empty.

Functions

func NewHttpIpfsHandlerFunc added in v0.3.0 

func NewHttpIpfsHandlerFunc(
	ctx context.Context,
	lsys linking.LinkSystem,
	opts ...HttpOption,
) http.HandlerFunc

func StreamCar 

func StreamCar(
	ctx context.Context,
	requestLsys linking.LinkSystem,
	out io.Writer,
	request trustlessutils.Request,
) error

StreamCar streams a DAG in CARv1 format to the given writer, using the given selector.

Types

type ErrorLogger 

type ErrorLogger interface {
	LogError(status int, err error)
}

type FrisbiiServer 

type FrisbiiServer struct {
	// contains filtered or unexported fields
}

FrisbiiServer is the main server for the frisbii application, it starts an HTTP server to serve data according to the Trustless Gateway spec and it also provides a mechanism to announce the server to the indexer service.

func NewFrisbiiServer 

func NewFrisbiiServer(
	ctx context.Context,
	lsys linking.LinkSystem,
	address string,
	httpOptions ...HttpOption,
) (*FrisbiiServer, error)

func (*FrisbiiServer) Addr 

func (fs *FrisbiiServer) Addr() net.Addr

func (*FrisbiiServer) Announce 

func (fs *FrisbiiServer) Announce() error

func (*FrisbiiServer) Serve 

func (fs *FrisbiiServer) Serve() error

func (*FrisbiiServer) SetIndexerProvider 

func (fs *FrisbiiServer) SetIndexerProvider(handlerPath string, indexerProvider IndexerProvider) error

type HttpIpfs 

type HttpIpfs struct {
	// contains filtered or unexported fields
}

HttpIpfs is an http.Handler that serves IPLD data via HTTP according to the Trustless Gateway specification.

func NewHttpIpfs 

func NewHttpIpfs(
	ctx context.Context,
	lsys linking.LinkSystem,
	opts ...HttpOption,
) *HttpIpfs

NewHttpIpfs returns an http.Handler that serves IPLD data via HTTP according to the Trustless Gateway specification.

func (*HttpIpfs) ServeHTTP 

func (hi *HttpIpfs) ServeHTTP(res http.ResponseWriter, req *http.Request)

type HttpOption added in v0.1.0 

type HttpOption func(*httpOptions)

func WithCompressionLevel added in v0.3.0 

func WithCompressionLevel(l int) HttpOption

WithCompressionLevel sets the compression level for the gzip compression applied to the response. This allows for a trade-off between CPU and bandwidth. By default, the compression level is set to gzip.NoCompression; which means compression will be disabled.

Other recommended choices are gzip.BestSpeed (1), gzip.BestCompression (9), and gzip.DefaultCompression (typically 6).

func WithLogHandler added in v0.4.0 

func WithLogHandler(h LogHandler) HttpOption

WithLogHandler sets a handler function that will be used to log requests. By default, requests are not logged. This is an alternative to WithLogWriter that allows for more control over the logging.

func WithLogWriter added in v0.4.0 

func WithLogWriter(w io.Writer) HttpOption

WithLogWriter sets the writer that will be used to log requests. By default, requests are not logged.

The log format for requests (including errors) is roughly equivalent to a standard nginx or Apache log format; that is, a space-separated list of elements, where the elements that may contain spaces are quoted. The format of each line can be specified as: 

%s %s %s "%s" %d %d %d %s "%s" "%s"

Where the elements are:

1. RFC 3339 timestamp 2. Remote address 3. Method 4. Path 5. Response status code 6. Response duration (in milliseconds) 7. Response size 8. Compression ratio (or `-` if no compression) 9. User agent 10. Error (or `""` if no error)

func WithMaxResponseBytes added in v0.1.0 

func WithMaxResponseBytes(b int64) HttpOption

WithMaxResponseBytes sets the maximum number of bytes that will be streamed before the connection is closed. This allows a server to limit the amount of data a client can request; and also restricts the ability to serve very large DAGs.

A value of 0 will disable the limitation. This is the default.

func WithMaxResponseDuration added in v0.1.0 

func WithMaxResponseDuration(d time.Duration) HttpOption

WithMaxResponseDuration sets the maximum duration for a response to be streamed before the connection is closed. This allows a server to limit the amount of time a client can hold a connection open; and also restricts the ability to serve very large DAGs.

A value of 0 will disable the limitation. This is the default.

type IndexerProvider 

type IndexerProvider interface {
	GetPublisherHttpFunc() (http.HandlerFunc, error)
	NotifyPut(ctx context.Context, provider *peer.AddrInfo, contextID []byte, md metadata.Metadata) (cid.Cid, error)
}

type LogHandler added in v0.4.0 

type LogHandler func(
	time time.Time,
	remoteAddr string,
	method string,
	url url.URL,
	status int,
	duration time.Duration,
	bytes int,
	compressionRatio string,
	userAgent string,
	msg string,
)

type LogMiddleware 

type LogMiddleware struct {
	// contains filtered or unexported fields
}

LogMiddlware is a middleware that logs requests to the given io.Writer. it wraps requests in a LoggingResponseWriter that can be used to log standardised messages to the writer.

func NewLogMiddleware 

func NewLogMiddleware(next http.Handler, httpOptions ...HttpOption) *LogMiddleware

NewLogMiddleware creates a new LogMiddleware to insert into an HTTP call chain.

The WithLogWriter option can be used to set the writer to log to.

The WithLogHandler option can be used to set a custom log handler.

func (*LogMiddleware) ServeHTTP 

func (lm *LogMiddleware) ServeHTTP(res http.ResponseWriter, req *http.Request)

type LoggingResponseWriter 

type LoggingResponseWriter struct {
	http.ResponseWriter
	// contains filtered or unexported fields
}

func NewLoggingResponseWriter 

func NewLoggingResponseWriter(
	w http.ResponseWriter,
	req *http.Request,
	logWriter io.Writer,
	logHandler LogHandler,
) *LoggingResponseWriter

NewLoggingResponseWriter creates a new LoggingResponseWriter that is used on a per-request basis to log information about the request.

func (*LoggingResponseWriter) CompressionRatio added in v0.3.0 

func (w *LoggingResponseWriter) CompressionRatio() string

func (*LoggingResponseWriter) Hijack 

func (w *LoggingResponseWriter) Hijack() (net.Conn, *bufio.ReadWriter, error)

func (*LoggingResponseWriter) Log 

func (w *LoggingResponseWriter) Log(
	status int,
	start time.Time,
	bytes int,
	CompressionRatio string,
	msg string,
)

func (*LoggingResponseWriter) LogError 

func (w *LoggingResponseWriter) LogError(status int, err error)

func (*LoggingResponseWriter) Write 

func (w *LoggingResponseWriter) Write(b []byte) (int, error)

func (*LoggingResponseWriter) WriteHeader 

func (w *LoggingResponseWriter) WriteHeader(status int)

func (*LoggingResponseWriter) WroteBytes added in v0.4.0 

func (w *LoggingResponseWriter) WroteBytes(n int)

WroteBytes can be called by the base writer, on each Write call, to indicate how many bytes were written in to the response. If LoggingResponseWriter is wrapping a compression writer, this should be the number of bytes written to the compression writer, which will be different than the number of bytes written to this writer; hence we can calculate the compression ratio.

type MultiReadableStorage 

type MultiReadableStorage struct {
	// contains filtered or unexported fields
}

MultiReadableStorage manages a list of storage.StreamingReadableStorage stores, providing a unified LinkSystem interface to them.

func NewMultiReadableStorage 

func NewMultiReadableStorage() *MultiReadableStorage

func (*MultiReadableStorage) AddStore 

func (m *MultiReadableStorage) AddStore(store storage.StreamingReadableStorage, roots []cid.Cid)

func (*MultiReadableStorage) Get 

func (m *MultiReadableStorage) Get(ctx context.Context, key string) ([]byte, error)

func (*MultiReadableStorage) GetStream 

func (m *MultiReadableStorage) GetStream(ctx context.Context, key string) (io.ReadCloser, error)

func (*MultiReadableStorage) Has 

func (m *MultiReadableStorage) Has(ctx context.Context, key string) (bool, error)

func (*MultiReadableStorage) RootsLister 

func (m *MultiReadableStorage) RootsLister() provider.MultihashLister

Source Files

View all Source files

Directories

Path Synopsis
cmd
frisbii
internal
util

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.