nopfs

package module
v0.0.12 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 7, 2023 License: Apache-2.0 Imports: 23 Imported by: 4

README

NOpfs!

NOPfs helps IPFS to say No!

ipfs-lite

NOPfs is an implementation of IPIP-383 which add supports for content blocking to the go-ipfs stack and particularly to Kubo.

Content-blocking in Kubo

  1. Grab a plugin release from the releases section matching your Kubo version and install the plugin file in ~/.ipfs/plugins.
  2. Write a custom denylist file (see syntax below) or simply download one of the supported denylists from Denyli.st and place them in ~/.config/ipfs/denylists/ (ensure .deny extension).
  3. Start Kubo (ipfs daemon). The plugin should be loaded automatically and existing denylists tracked for updates from that point (no restarts required).

Denylist syntax

Denylist files must have the .deny extension. The content consists of an optional header and a body made of blocking rules as follows:

version: 1
name: IPFSorp blocking list
description: A collection of bad things we have found in the universe
author: abuse-ipfscorp@example.com
hints:
  gateway_status: 410
  double_hash_fn: sha256
  double_hash_enc: hex
---
# Blocking by CID - blocks wrapped multihash.
# Does not block subpaths.
/ipfs/bafybeihvvulpp4evxj7x7armbqcyg6uezzuig6jp3lktpbovlqfkuqeuoq

# Block all subpaths
/ipfs/QmdWFA9FL52hx3j9EJZPQP1ZUH8Ygi5tLCX2cRDs6knSf8/*

# Block some subpaths (equivalent rules)
/ipfs/Qmah2YDTfrox4watLCr3YgKyBwvjq8FJZEFdWY6WtJ3Xt2/test*
/ipfs/QmTuvSQbEDR3sarFAN9kAeXBpiBCyYYNxdxciazBba11eC/test/*

# Block some subpaths with exceptions
/ipfs/QmUboz9UsQBDeS6Tug1U8jgoFkgYxyYood9NDyVURAY9pK/blocked*
+/ipfs/QmUboz9UsQBDeS6Tug1U8jgoFkgYxyYood9NDyVURAY9pK/blockednot
+/ipfs/QmUboz9UsQBDeS6Tug1U8jgoFkgYxyYood9NDyVURAY9pK/blocked/not
+/ipfs/QmUboz9UsQBDeS6Tug1U8jgoFkgYxyYood9NDyVURAY9pK/blocked/exceptions*

# Block IPNS domain name
/ipns/domain.example

# Block IPNS domain name and path
/ipns/domain2.example/path

# Block IPNS key - blocks wrapped multihash.
/ipns/k51qzi5uqu5dhmzyv3zac033i7rl9hkgczxyl81lwoukda2htteop7d3x0y1mf

# Legacy CID double-hash block
# sha256(bafybeiefwqslmf6zyyrxodaxx4vwqircuxpza5ri45ws3y5a62ypxti42e/)
# blocks only this CID
//d9d295bde21f422d471a90f2a37ec53049fdf3e5fa3ee2e8f20e10003da429e7

# Legacy Path double-hash block
# Blocks bafybeiefwqslmf6zyyrxodaxx4vwqircuxpza5ri45ws3y5a62ypxti42e/path
# but not any other paths.
//3f8b9febd851873b3774b937cce126910699ceac56e72e64b866f8e258d09572

# Double hash CID block
# base58btc-sha256-multihash(QmVTF1yEejXd9iMgoRTFDxBv7HAz9kuZcQNBzHrceuK9HR)
# Blocks bafybeidjwik6im54nrpfg7osdvmx7zojl5oaxqel5cmsz46iuelwf5acja
# and QmVTF1yEejXd9iMgoRTFDxBv7HAz9kuZcQNBzHrceuK9HR etc. by multihash
//QmX9dhRcQcKUw3Ws8485T5a9dtjrSCQaUAHnG4iK9i4ceM

# Double hash Path block using blake3 hashing
# base58btc-blake3-multihash(gW7Nhu4HrfDtphEivm3Z9NNE7gpdh5Tga8g6JNZc1S8E47/path)
# Blocks /ipfs/bafyb4ieqht3b2rssdmc7sjv2cy2gfdilxkfh7623nvndziyqnawkmo266a/path
# /ipfs/bafyb4ieqht3b2rssdmc7sjv2cy2gfdilxkfh7623nvndziyqnawkmo266a/path
# /ipfs/f01701e20903cf61d46521b05f926ba1634628d0bba8a7ffb5b6d5a3ca310682ca63b5ef0/path etc...
# But not /path2
//QmbK7LDv5NNBvYQzNfm2eED17SNLt1yNMapcUhSuNLgkqz

You can create double-hashes by hand with the following command:

printf "QmecDgNqCRirkc3Cjz9eoRBNwXGckJ9WvTdmY16HP88768/my/path" \
  | ipfs add --raw-leaves --only-hash --quiet \
  | ipfs cid format -f '%M' -b base58btc

where:

  • QmecDgNqCRirkc3Cjz9eoRBNwXGckJ9WvTdmY16HP88768 must always be a CidV0. If you have a CIDv1 you need to convert it to CIDv0 first. i.e ipfs cid format -v0 bafybeihrw75yfhdx5qsqgesdnxejtjybscwuclpusvxkuttep6h7pkgmze
  • /my/path is optional depending on whether you want to block a specific path. No wildcards supported here!
  • The command above should give QmSju6XPmYLG611rmK7rEeCMFVuL6EHpqyvmEU6oGx3GR8. Use it as //QmSju6XPmYLG611rmK7rEeCMFVuL6EHpqyvmEU6oGx3GR8 on the denylist.

Kubo plugin

NOpfs Kubo plugin pre-built binary releases are available in the releases section.

Simply grab the binary for your system and drop it in the ~/.ipfs/plugins folder.

From that point, starting Kubo should load the plugin and automatically work with denylists (files with extension .deny) found in /etc/ipfs/denylists and $XDG_CONFIG_HOME/ipfs/denylists (usually ~/.config/ipfs/denylists). The plugin will log some lines as the ipfs daemon starts:

$ ipfs daemon --offline
Initializing daemon...
Kubo version: 0.21.0-rc1
Repo version: 14
System version: amd64/linux
Golang version: go1.19.10
2023-06-13T21:26:56.951+0200	INFO	nopfs	nopfs-kubo-plugin/plugin.go:59	Loading Nopfs plugin: content blocking
2023-06-13T21:26:56.952+0200	INFO	nopfs	nopfs@v0.0.7/denylist.go:165	Processing /home/user/.config/ipfs/denylists/badbits.deny: badbits (2023-03-27) by @Protocol Labs

The plugin can be manually built and installed for different versions of Kubo with:

git checkout nopfs-kubo-plugin/v<kubo-version>
make plugin
make install-plugin

Project layout

The NOpfs contains three separate Go-modules (versioned separately):

  • The main module (github.com/ipfs-shipyard/nopfs) provides the implementation of a Blocker that works with IPIP-383 denylists (can parse, track and answer whether CIDs/paths are blocked)
  • The ipfs submodule (github.com/ipfs-shipyard/nopfs/ipfs) provides blocking-wrappers for types in the Boxo/stack (Resolver, BlockService etc.). It's versioning tracks Boxo tags. i.e. v0.10.0 is compatible with boxo@v0.10.0.
  • The nopfs-kubo-plugin submodule (github.com/ipfs-shipyard/nopfs/nopfs-kubo-plugin) contains only the code of the Kubo plugin, which injects blocking-wrappers into Kubo. It is tagged tracking Kubo releases.

This allows using the Blocker separately, or relying on blocking-wrappers separately in a way that it is easy to identify and select dependency-aligned versions with your project, without specifying more dependencies that needed.

Project status

  • Support for blocking CIDs
  • Support for blocking IPFS Paths
  • Support for paths with wildcards (prefix paths)
  • Support for blocking legacy badbits anchors
  • Support for blocking double-hashed CIDs, IPFS paths, IPNS paths.
  • Support for blocking prefix and non-prefix sub-path
  • Support for denylist headers
  • Support for denylist rule hints
  • Support for allow rules (undo or create exceptions to earlier rules)
  • Live processing of appended rules to denylists
  • Content-blocking-enabled IPFS BlockService implementation
  • Content-blocking-enabled IPFS NameSystem implementation
  • Content-blocking-enabled IPFS Path resolver implementation
  • Kubo plugin
  • Automatic, comprehensive testing of all rule types and edge cases
  • Work with a stable release of Kubo
  • Prebuilt plugin binaries

Documentation

Overview

Package nopfs implements content blocking for the IPFS stack.

nopfs provides an implementation of the compact denylist format (IPIP-383), with methods to check whether IPFS paths and CIDs are blocked.

In order to seamlessly be inserted into the IPFS stack, content-blocking wrappers for several components are provided (BlockService, NameSystem, Resolver...). A Kubo plugin (see kubo/) can be used to give Kubo content-blocking superpowers.

Index

Constants

This section is empty.

Variables

View Source 
var ErrHeaderNotFound = errors.New("header not found")

ErrHeaderNotFound is returned when no header can be Decoded.

View Source 
var SafeCids = map[cid.Cid]string{
	cid.MustParse("QmUNLLsPACCz1vLxQVkXqqLX5R1X345qqfHbsf67hvA3Nn"):                "empty unixfs directory",
	cid.MustParse("bafyaabakaieac"):                                                "empty unixfs directory inlined",
	cid.MustParse("bafkreihdwdcefgh4dqkjv67uzcmw7ojee6xedzdetojuzjevtenxquvyku"):   "empty block",
	cid.MustParse("bafkqaaa"):                                                      "empty block inlined",
	cid.MustParse("QmbFMke1KXqnYyBBWxB74N4c5SBnJMVAiMNRcGu6x1AwQH"):                "empty block dag-pb",
	cid.MustParse("bafyreigbtj4x7ip5legnfznufuopl4sg4knzc2cof6duas4b3q2fy6swua"):   "empty block dag-cbor",
	cid.MustParse("baguqeeraiqjw7i2vwntyuekgvulpp2det2kpwt6cd7tx5ayqybqpmhfk76fa"): "empty block dag-json",
}

SafeCids is a map of known, innoffensive CIDs that correspond to empty-blocks or empty-directories. Blocking these can break applications so they are ignored (with a warning), when they appear on a denylist.

Functions

func GetDenylistFiles 

func GetDenylistFiles() ([]string, error)

GetDenylistFiles returns a list of ".deny" files found in $XDG_CONFIG_HOME/ipfs/denylists and /etc/ipfs/denylists. The files are sortered by their names in their respective directories.

func GetDenylistFilesInDir added in v0.0.10 

func GetDenylistFilesInDir(dirpath string) ([]string, error)

GetDenylistFilesInDir returns a list of ".deny" files found in the given directory. The files are sortered by their names. It returns an empty list and no error if the directory does not exist.

Types

type BlockedPath 

type BlockedPath struct {
	Path   string
	Prefix bool
}

BlockedPath represents the path part of a blocking rule.

func NewBlockedPath 

func NewBlockedPath(rawPath string) (BlockedPath, error)

NewBlockedPath takes a raw path, unscapes and sanitizes it, detecting and handling wildcards. It may also represent an "allowed" path on "allowed" rules.

func (BlockedPath) Matches 

func (bpath BlockedPath) Matches(path string) bool

Matches returns whether the given path matched the blocked (or allowed) path.

type Blocker 

type Blocker struct {
	Denylists map[string]*Denylist
}

A Blocker binds together multiple Denylists and can decide whether a path or a CID is blocked.

func NewBlocker 

func NewBlocker(files []string) (*Blocker, error)

NewBlocker creates a Blocker using the given denylist file paths. For default denylist locations, you can use GetDenylistFiles(). TODO: Options.

func (*Blocker) Close 

func (blocker *Blocker) Close() error

Close stops all denylists from being processed and watched for updates.

func (*Blocker) IsCidBlocked 

func (blocker *Blocker) IsCidBlocked(c cid.Cid) StatusResponse

IsCidBlocked returns blocking status for a CID. A CID is blocked when a Denylist reports it as blocked. A CID is not blocked when no denylist reports it as blocked or it is explicitally allowed. Lookup stops as soon as a defined "blocked" or "allowed" status is found.

Lookup for "allowed" or "blocked" status happens in order of the denylist, thus the denylist position during Blocker creation affects which one has preference.

Note that StatusResponse.Path will be unset. See Denylist.IsCidBlocked() for more info.

func (*Blocker) IsPathBlocked 

func (blocker *Blocker) IsPathBlocked(p path.Path) StatusResponse

IsPathBlocked returns blocking status for an IPFS Path. A Path is blocked when a Denylist reports it as blocked. A Path is not blocked when no denylist reports it as blocked or it is explicitally allowed. Lookup stops as soon as a defined "blocked" or "allowed" status is found.

Lookup for "allowed" or "blocked" status happens in order of the denylist, thus the denylist position during Blocker creation affects which one has preference.

Note that StatusResponse.Cid will be unset. See Denylist.IsPathBlocked() for more info.

type BlocksDB 

type BlocksDB struct {
	// contains filtered or unexported fields
}

BlocksDB is a key-value store of Entries. Keying may vary depending on whether we are indexing IPNS names, CIDs etc.

func (*BlocksDB) Load 

func (b *BlocksDB) Load(key string) (Entries, bool)

Load returns the Entries for a key.

func (*BlocksDB) Store 

func (b *BlocksDB) Store(key string, entry Entry)

Store stores a new entry with the given key. If there are existing Entries, the new Entry will be appended to them.

type Denylist 

type Denylist struct {
	Header   DenylistHeader
	Filename string

	Entries Entries

	IPFSBlocksDB       *BlocksDB
	IPNSBlocksDB       *BlocksDB
	DoubleHashBlocksDB map[uint64]*BlocksDB // mhCode -> blocks using that code
	PathBlocksDB       *BlocksDB
	PathPrefixBlocks   Entries
	// contains filtered or unexported fields
}

A Denylist represents a denylist file and its rules. It can parse and follow a denylist file, and can answer questions about blocked or allowed items in this denylist.

func NewDenylist 

func NewDenylist(filepath string, follow bool) (*Denylist, error)

NewDenylist opens a denylist file and processes it (parses all its entries).

If follow is false, the file handle is closed.

If follow is true, the denylist file will be followed upon return. Any appended rules will be processed live-updated in the denylist. Denylist.Close() should be used when the Denylist or the following is no longer needed.

func NewDenylistReader 

func NewDenylistReader(r io.ReadSeekCloser) (*Denylist, error)

NewDenylistReader processes a denylist from the given reader (parses all its entries).

func (*Denylist) Close 

func (dl *Denylist) Close() error

Close closes the Denylist file handle and stops watching write events on it.

func (*Denylist) IsCidBlocked 

func (dl *Denylist) IsCidBlocked(c cid.Cid) StatusResponse

IsCidBlocked provides Blocking Status for a given CID. This is done by extracting the multihash and checking if it is blocked by any rule.

func (*Denylist) IsIPFSPathBlocked 

func (dl *Denylist) IsIPFSPathBlocked(cidStr, subpath string) StatusResponse

IsIPFSPathBlocked returns Blocking Status for a given IPFS CID and its subpath. The cidStr is NOT an "/ipns/cid" path, but just the cid.

func (*Denylist) IsIPLDPathBlocked 

func (dl *Denylist) IsIPLDPathBlocked(cidStr, subpath string) StatusResponse

IsIPLDPathBlocked returns Blocking Status for a given IPLD CID and its subpath. The cidStr is NOT an "/ipld/cid" path, but just the cid.

func (*Denylist) IsIPNSPathBlocked 

func (dl *Denylist) IsIPNSPathBlocked(name, subpath string) StatusResponse

IsIPNSPathBlocked returns Blocking Status for a given IPNS name and its subpath. The name is NOT an "/ipns/name" path, but just the name.

func (*Denylist) IsPathBlocked 

func (dl *Denylist) IsPathBlocked(p path.Path) StatusResponse

IsPathBlocked provides Blocking Status for a given path. This is done by interpreting the full path and checking for blocked Path, IPFS, IPNS or double-hashed items matching it.

Matching is more efficient if:

  • Paths in the form of /ipfs/Qm/... (sha2-256-multihash) are used rather than CIDv1.

  • A single double-hashing pattern is used.

  • A small number of path-only match rules using prefixes are used.

func (*Denylist) IsSubpathBlocked 

func (dl *Denylist) IsSubpathBlocked(subpath string) StatusResponse

IsSubpathBlocked returns Blocking Status for the given subpath.

type DenylistHeader 

type DenylistHeader struct {
	Version     int
	Name        string
	Description string
	Author      string
	Hints       map[string]string
	// contains filtered or unexported fields
}

DenylistHeader represents the header of a Denylist file.

func (*DenylistHeader) Decode 

func (h *DenylistHeader) Decode(r io.Reader) error

Decode decodes a DenlistHeader from a reader. Per the specification, the maximum size of a header is 1KiB. If no header is found, ErrHeaderNotFound is returned.

func (DenylistHeader) String 

func (h DenylistHeader) String() string

String provides a short string summary of the Header.

type Entries 

type Entries []Entry

Entries is a slice of Entry.

func (Entries) CheckPathStatus 

func (entries Entries) CheckPathStatus(p string) (Status, Entry)

CheckPathStatus returns whether the given path has a match in one of the Entries.

type Entry 

type Entry struct {
	Line      uint64
	AllowRule bool
	Hints     map[string]string
	RawValue  string
	Multihash multihash.Multihash // set for ipfs-paths mostly.
	Path      BlockedPath
}

Entry represents a rule (or a line) in a denylist file.

func (Entry) Clone added in v0.0.12 

func (e Entry) Clone() Entry

func (Entry) String 

func (e Entry) String() string

String provides a single-line representation of the Entry.

type HTTPSubscriber added in v0.0.12 

type HTTPSubscriber struct {
	// contains filtered or unexported fields
}

HTTPSubscriber represents a type that subscribes to a remote URL and appends data to a local file.

func NewHTTPSubscriber added in v0.0.12 

func NewHTTPSubscriber(remoteURL, localFile string, interval time.Duration) (*HTTPSubscriber, error)

NewHTTPSubscriber creates a new Subscriber instance with the given parameters.

func (*HTTPSubscriber) Stop added in v0.0.12 

func (s *HTTPSubscriber) Stop()

Stop stops the subscription process.

type Status 

type Status int

Status represent represents whether an item is blocked, allowed or simply not found in a Denylist. 

const (
	StatusNotFound Status = iota
	StatusBlocked
	StatusAllowed
	StatusErrored
)

Status values

func (Status) String 

func (st Status) String() string

type StatusError 

type StatusError struct {
	Response StatusResponse
}

StatusError implements the error interface and can be used to provide information about a blocked-status in the form of an error.

func (*StatusError) Error 

func (err *StatusError) Error() string

type StatusResponse 

type StatusResponse struct {
	Cid      cid.Cid
	Path     path.Path
	Status   Status
	Filename string
	Entry    Entry
	Error    error
}

StatusResponse provides full information for a content-block lookup, including the Filename and the Entry, when an associated rule is found.

func (StatusResponse) String added in v0.0.11 

func (r StatusResponse) String() string

String provides a string with the details of a StatusResponse.

func (StatusResponse) ToError 

func (r StatusResponse) ToError() *StatusError

ToError returns nil if the Status of the StatusResponse is Allowed or Not Found. When the status is Blocked or Errored, it returns a StatusError.

Source Files

View all Source files

Directories

Path Synopsis
cmd
httpsubs
nopfs
ipfs module
Package tester provides an implementation-agnostic way to test a Blocker.
 Package tester provides an implementation-agnostic way to test a Blocker.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.