store

package
v1.0.1-0...-d0026f9 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 21, 2016 License: BSD-2-Clause Imports: 13 Imported by: 0

README

The store Package

The store package offers a storage interface and middlewares sufficient to run a public tracker based on it.

Architecture

The store consists of three parts:

  • A set of interfaces, tests based on these interfaces and the store logic, unifying these interfaces into the store
  • Drivers, implementing the store interfaces and
  • Middleware that depends on the store

The store interfaces are IPStore, PeerStore and StringStore. During runtime, each of them will be implemented by a driver. Even though all different drivers for one interface provide the same functionality, their behaviour can be very different. For example: The memory implementation keeps all state in-memory - this is very fast, but not persistent, it loses its state on every restart. A database-backed driver on the other hand could provide persistence, at the cost of performance.

The pluggable design of Chihaya allows for the different interfaces to use different drivers. For example: A typical use case of the StringStore is to provide blacklists or whitelists for infohashes/client IDs/.... You'd typically want these lists to be persistent, so you'd choose a driver that provides persistence. The PeerStore on the other hand rarely needs to be persistent, as all peer state will be restored after one announce interval. You'd therefore typically choose a very performant but non-persistent driver for the PeerStore.

Testing

The main store package also contains a set of tests and benchmarks for drivers. Both use the store interfaces and can work with any driver that implements these interfaces. The tests verify that the driver behaves as specified by the interface and its documentation. The benchmarks can be used to compare performance of a wide range of operations on the interfaces.

This makes it very easy to implement a new driver: All functions that are part of the store interfaces can be tested easily with the tests that come with the store package. Generally the memory implementation can be used as a guideline for implementing new drivers.

Both benchmarks and tests require a clean state to work correctly. All of the test and benchmark functions therefore take a *DriverConfig as a parameter, this should be used to configure the driver in a way that it provides a clean state for every test or benchmark. For example: Imagine a file-based driver that achieves persistence by storing its state in a file. It must then be possible to provide the location of this file in the 'DriverConfig, so that every different benchmark gets to work with a new file.

Most benchmarks come in two flavors: The "normal" version and the "1K" version. A normal benchmark uses the same value over and over again to benchmark one operation. A 1K benchmark uses a different value from a set of 1000 values for every iteration, this can show caching effects, if the driver uses them. The 1K benchmarks require a little more computation to select the values and thus typically yield slightly lower results even for a "perfect" cache, i.e. the memory implementation.

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var ErrResourceDoesNotExist = errors.New("resource does not exist")

ErrResourceDoesNotExist is the error returned by all delete methods in the store if the requested resource does not exist.

Functions

func RegisterIPStoreDriver 

func RegisterIPStoreDriver(name string, driver IPStoreDriver)

RegisterIPStoreDriver makes a driver available by the provided name.

If this function is called twice with the same name or if the driver is nil, it panics.

func RegisterPeerStoreDriver 

func RegisterPeerStoreDriver(name string, driver PeerStoreDriver)

RegisterPeerStoreDriver makes a driver available by the provided name.

If this function is called twice with the same name or if the driver is nil, it panics.

func RegisterStringStoreDriver 

func RegisterStringStoreDriver(name string, driver StringStoreDriver)

RegisterStringStoreDriver makes a driver available by the provided name.

If this function is called twice with the same name or if the driver is nil, it panics.

Types

type Config 

type Config struct {
	Addr           string        `yaml:"addr"`
	RequestTimeout time.Duration `yaml:"request_timeout"`
	ReadTimeout    time.Duration `yaml:"read_timeout"`
	WriteTimeout   time.Duration `yaml:"write_timeout"`
	GCAfter        time.Duration `yaml:"gc_after"`
	PeerStore      DriverConfig  `yaml:"peer_store"`
	IPStore        DriverConfig  `yaml:"ip_store"`
	StringStore    DriverConfig  `yaml:"string_store"`
}

Config represents the configuration for the store.

type DriverConfig 

type DriverConfig struct {
	Name   string      `yaml:"name"`
	Config interface{} `yaml:"config"`
}

DriverConfig represents the configuration for a store driver.

type IPStore 

type IPStore interface {
	// AddIP adds a single IP address to the IPStore.
	AddIP(ip net.IP) error

	// AddNetwork adds a range of IP addresses, denoted by a network in CIDR
	// notation, to the IPStore.
	AddNetwork(network string) error

	// HasIP returns whether the given IP address is contained in the IPStore
	// or belongs to any of the stored networks.
	HasIP(ip net.IP) (bool, error)

	// HasAnyIP returns whether any of the given IP addresses are contained
	// in the IPStore or belongs to any of the stored networks.
	HasAnyIP(ips []net.IP) (bool, error)

	// HassAllIPs returns whether all of the given IP addresses are
	// contained in the IPStore or belongs to any of the stored networks.
	HasAllIPs(ips []net.IP) (bool, error)

	// RemoveIP removes a single IP address from the IPStore.
	//
	// This wil not remove the given address from any networks it belongs to
	// that are stored in the IPStore.
	//
	// Returns ErrResourceDoesNotExist if the given IP address is not
	// contained in the store.
	RemoveIP(ip net.IP) error

	// RemoveNetwork removes a range of IP addresses that was previously
	// added through AddNetwork.
	//
	// The given network must not, as a string, match the previously added
	// network, but rather denote the same network, e.g. if the network
	// 192.168.22.255/24 was added, removing the network 192.168.22.123/24
	// will succeed.
	//
	// Returns ErrResourceDoesNotExist if the given network is not
	// contained in the store.
	RemoveNetwork(network string) error

	// Stopper provides the Stop method that stops the IPStore.
	// Stop should shut down the IPStore in a separate goroutine and send
	// an error to the channel if the shutdown failed. If the shutdown
	// was successful, the channel is to be closed.
	stopper.Stopper
}

IPStore represents an interface for manipulating IPs and IP ranges.

func OpenIPStore 

func OpenIPStore(cfg *DriverConfig) (IPStore, error)

OpenIPStore returns an IPStore specified by a configuration.

type IPStoreBenchmarker 

type IPStoreBenchmarker interface {
	AddV4(*testing.B, *DriverConfig)
	AddV6(*testing.B, *DriverConfig)
	LookupV4(*testing.B, *DriverConfig)
	LookupV6(*testing.B, *DriverConfig)
	AddRemoveV4(*testing.B, *DriverConfig)
	AddRemoveV6(*testing.B, *DriverConfig)
	LookupNonExistV4(*testing.B, *DriverConfig)
	LookupNonExistV6(*testing.B, *DriverConfig)
	RemoveNonExistV4(*testing.B, *DriverConfig)
	RemoveNonExistV6(*testing.B, *DriverConfig)

	AddV4Network(*testing.B, *DriverConfig)
	AddV6Network(*testing.B, *DriverConfig)
	LookupV4Network(*testing.B, *DriverConfig)
	LookupV6Network(*testing.B, *DriverConfig)
	AddRemoveV4Network(*testing.B, *DriverConfig)
	AddRemoveV6Network(*testing.B, *DriverConfig)
	RemoveNonExistV4Network(*testing.B, *DriverConfig)
	RemoveNonExistV6Network(*testing.B, *DriverConfig)

	Add1KV4(*testing.B, *DriverConfig)
	Add1KV6(*testing.B, *DriverConfig)
	Lookup1KV4(*testing.B, *DriverConfig)
	Lookup1KV6(*testing.B, *DriverConfig)
	AddRemove1KV4(*testing.B, *DriverConfig)
	AddRemove1KV6(*testing.B, *DriverConfig)
	LookupNonExist1KV4(*testing.B, *DriverConfig)
	LookupNonExist1KV6(*testing.B, *DriverConfig)
	RemoveNonExist1KV4(*testing.B, *DriverConfig)
	RemoveNonExist1KV6(*testing.B, *DriverConfig)

	Add1KV4Network(*testing.B, *DriverConfig)
	Add1KV6Network(*testing.B, *DriverConfig)
	Lookup1KV4Network(*testing.B, *DriverConfig)
	Lookup1KV6Network(*testing.B, *DriverConfig)
	AddRemove1KV4Network(*testing.B, *DriverConfig)
	AddRemove1KV6Network(*testing.B, *DriverConfig)
	RemoveNonExist1KV4Network(*testing.B, *DriverConfig)
	RemoveNonExist1KV6Network(*testing.B, *DriverConfig)
}

IPStoreBenchmarker is a collection of benchmarks for IPStore drivers. Every benchmark expects a new, clean storage. Every benchmark should be called with a DriverConfig that ensures this.

func PrepareIPStoreBenchmarker 

func PrepareIPStoreBenchmarker(driver IPStoreDriver) IPStoreBenchmarker

PrepareIPStoreBenchmarker prepares a reusable suite for StringStore driver benchmarks.

type IPStoreDriver 

type IPStoreDriver interface {
	New(*DriverConfig) (IPStore, error)
}

IPStoreDriver represents an interface for creating a handle to the storage of IPs.

type IPStoreTester 

type IPStoreTester interface {
	TestIPStore(*testing.T, *DriverConfig)
	TestHasAllHasAny(*testing.T, *DriverConfig)
	TestNetworks(*testing.T, *DriverConfig)
	TestHasAllHasAnyNetworks(*testing.T, *DriverConfig)
}

IPStoreTester is a collection of tests for an IPStore driver. Every benchmark expects a new, clean storage. Every benchmark should be called with a DriverConfig that ensures this.

func PrepareIPStoreTester 

func PrepareIPStoreTester(driver IPStoreDriver) IPStoreTester

PrepareIPStoreTester prepares a reusable suite for IPStore driver tests.

type PeerStore 

type PeerStore interface {
	// PutSeeder adds a seeder for the infoHash to the PeerStore.
	PutSeeder(infoHash chihaya.InfoHash, p chihaya.Peer) error
	// DeleteSeeder removes a seeder for the infoHash from the PeerStore.
	//
	// Returns ErrResourceDoesNotExist if the infoHash or peer does not
	// exist.
	DeleteSeeder(infoHash chihaya.InfoHash, p chihaya.Peer) error

	// PutLeecher adds a leecher for the infoHash to the PeerStore.
	PutLeecher(infoHash chihaya.InfoHash, p chihaya.Peer) error
	// DeleteLeecher removes a leecher for the infoHash from the PeerStore.
	//
	// Returns ErrResourceDoesNotExist if the infoHash or peer does not
	// exist.
	DeleteLeecher(infoHash chihaya.InfoHash, p chihaya.Peer) error

	// GraduateLeecher promotes a peer from a leecher to a seeder for the
	// infoHash within the PeerStore.
	//
	// If the given Peer is not a leecher, it will still be added to the
	// list of seeders and no error will be returned.
	GraduateLeecher(infoHash chihaya.InfoHash, p chihaya.Peer) error

	// AnnouncePeers returns a list of both IPv4, and IPv6 peers for an
	// announce.
	//
	// If seeder is true then the peers returned will only be leechers, the
	// ammount of leechers returned will be the smaller value of numWant or
	// the available leechers.
	// If it is false then seeders will be returned up until numWant or the
	// available seeders, whichever is smaller. If the available seeders is
	// less than numWant then peers are returned until numWant or they run out.
	AnnouncePeers(infoHash chihaya.InfoHash, seeder bool, numWant int, peer4, peer6 chihaya.Peer) (peers, peers6 []chihaya.Peer, err error)
	// CollectGarbage deletes peers from the peerStore which are older than the
	// cutoff time.
	CollectGarbage(cutoff time.Time) error

	// GetSeeders gets all the seeders for a particular infoHash.
	GetSeeders(infoHash chihaya.InfoHash) (peers, peers6 []chihaya.Peer, err error)
	// GetLeechers gets all the leechers for a particular infoHash.
	GetLeechers(infoHash chihaya.InfoHash) (peers, peers6 []chihaya.Peer, err error)

	// NumSeeders gets the amount of seeders for a particular infoHash.
	NumSeeders(infoHash chihaya.InfoHash) int
	// NumLeechers gets the amount of leechers for a particular infoHash.
	NumLeechers(infoHash chihaya.InfoHash) int

	// Stopper provides the Stop method that stops the PeerStore.
	// Stop should shut down the PeerStore in a separate goroutine and send
	// an error to the channel if the shutdown failed. If the shutdown
	// was successful, the channel is to be closed.
	stopper.Stopper
}

PeerStore represents an interface for manipulating peers.

func OpenPeerStore 

func OpenPeerStore(cfg *DriverConfig) (PeerStore, error)

OpenPeerStore returns a PeerStore specified by a configuration.

type PeerStoreBenchmarker 

type PeerStoreBenchmarker interface {
	PutSeeder(*testing.B, *DriverConfig)
	PutSeeder1KInfohash(*testing.B, *DriverConfig)
	PutSeeder1KSeeders(*testing.B, *DriverConfig)
	PutSeeder1KInfohash1KSeeders(*testing.B, *DriverConfig)

	PutDeleteSeeder(*testing.B, *DriverConfig)
	PutDeleteSeeder1KInfohash(*testing.B, *DriverConfig)
	PutDeleteSeeder1KSeeders(*testing.B, *DriverConfig)
	PutDeleteSeeder1KInfohash1KSeeders(*testing.B, *DriverConfig)

	DeleteSeederNonExist(*testing.B, *DriverConfig)
	DeleteSeederNonExist1KInfohash(*testing.B, *DriverConfig)
	DeleteSeederNonExist1KSeeders(*testing.B, *DriverConfig)
	DeleteSeederNonExist1KInfohash1KSeeders(*testing.B, *DriverConfig)

	PutGraduateDeleteLeecher(*testing.B, *DriverConfig)
	PutGraduateDeleteLeecher1KInfohash(*testing.B, *DriverConfig)
	PutGraduateDeleteLeecher1KLeechers(*testing.B, *DriverConfig)
	PutGraduateDeleteLeecher1KInfohash1KLeechers(*testing.B, *DriverConfig)

	GraduateLeecherNonExist(*testing.B, *DriverConfig)
	GraduateLeecherNonExist1KInfohash(*testing.B, *DriverConfig)
	GraduateLeecherNonExist1KLeechers(*testing.B, *DriverConfig)
	GraduateLeecherNonExist1KInfohash1KLeechers(*testing.B, *DriverConfig)

	AnnouncePeers(*testing.B, *DriverConfig)
	AnnouncePeers1KInfohash(*testing.B, *DriverConfig)
	AnnouncePeersSeeder(*testing.B, *DriverConfig)
	AnnouncePeersSeeder1KInfohash(*testing.B, *DriverConfig)

	GetSeeders(*testing.B, *DriverConfig)
	GetSeeders1KInfohash(*testing.B, *DriverConfig)

	NumSeeders(*testing.B, *DriverConfig)
	NumSeeders1KInfohash(*testing.B, *DriverConfig)
}

PeerStoreBenchmarker is a collection of benchmarks for PeerStore drivers. Every benchmark expects a new, clean storage. Every benchmark should be called with a DriverConfig that ensures this.

func PreparePeerStoreBenchmarker 

func PreparePeerStoreBenchmarker(driver PeerStoreDriver) PeerStoreBenchmarker

PreparePeerStoreBenchmarker prepares a reusable suite for PeerStore driver benchmarks.

type PeerStoreDriver 

type PeerStoreDriver interface {
	New(*DriverConfig) (PeerStore, error)
}

PeerStoreDriver represents an interface for creating a handle to the storage of peers.

type PeerStoreTester 

type PeerStoreTester interface {
	// CompareEndpoints sets the function used to compare peers to a
	// comparison that only compares endpoints and omits PeerIDs.
	CompareEndpoints()

	TestPeerStore(*testing.T, *DriverConfig)
}

PeerStoreTester is a collection of tests for a PeerStore driver. Every benchmark expects a new, clean storage. Every benchmark should be called with a DriverConfig that ensures this.

func PreparePeerStoreTester 

func PreparePeerStoreTester(driver PeerStoreDriver) PeerStoreTester

PreparePeerStoreTester prepares a reusable suite for PeerStore driver tests. The tester will use PeerIDs and endpoints to compare peers.

type Store 

type Store struct {
	PeerStore
	IPStore
	StringStore
	// contains filtered or unexported fields
}

Store provides storage for a tracker.

func MustGetStore 

func MustGetStore() *Store

MustGetStore is used by middleware to access the store.

This function calls log.Fatal if a server hasn't been already created by the server package.

func (*Store) Start 

func (s *Store) Start()

Start starts the store drivers and blocks until all of them exit.

func (*Store) Stop 

func (s *Store) Stop()

Stop stops the store drivers and waits for them to exit.

type StringStore 

type StringStore interface {
	// PutString adds the given string to the StringStore.
	PutString(s string) error

	// HasString returns whether or not the StringStore contains the given
	// string.
	HasString(s string) (bool, error)

	// RemoveString removes the string from the string store.
	// Returns ErrResourceDoesNotExist if the given string is not contained
	// in the store.
	RemoveString(s string) error

	// Stopper provides the Stop method that stops the StringStore.
	// Stop should shut down the StringStore in a separate goroutine and send
	// an error to the channel if the shutdown failed. If the shutdown
	// was successful, the channel is to be closed.
	stopper.Stopper
}

StringStore represents an interface for manipulating strings.

func OpenStringStore 

func OpenStringStore(cfg *DriverConfig) (StringStore, error)

OpenStringStore returns a StringStore specified by a configuration.

type StringStoreBenchmarker 

type StringStoreBenchmarker interface {
	AddShort(*testing.B, *DriverConfig)
	AddLong(*testing.B, *DriverConfig)
	LookupShort(*testing.B, *DriverConfig)
	LookupLong(*testing.B, *DriverConfig)
	AddRemoveShort(*testing.B, *DriverConfig)
	AddRemoveLong(*testing.B, *DriverConfig)
	LookupNonExistShort(*testing.B, *DriverConfig)
	LookupNonExistLong(*testing.B, *DriverConfig)
	RemoveNonExistShort(*testing.B, *DriverConfig)
	RemoveNonExistLong(*testing.B, *DriverConfig)

	Add1KShort(*testing.B, *DriverConfig)
	Add1KLong(*testing.B, *DriverConfig)
	Lookup1KShort(*testing.B, *DriverConfig)
	Lookup1KLong(*testing.B, *DriverConfig)
	AddRemove1KShort(*testing.B, *DriverConfig)
	AddRemove1KLong(*testing.B, *DriverConfig)
	LookupNonExist1KShort(*testing.B, *DriverConfig)
	LookupNonExist1KLong(*testing.B, *DriverConfig)
	RemoveNonExist1KShort(*testing.B, *DriverConfig)
	RemoveNonExist1KLong(*testing.B, *DriverConfig)
}

StringStoreBenchmarker is a collection of benchmarks for StringStore drivers. Every benchmark expects a new, clean storage. Every benchmark should be called with a DriverConfig that ensures this.

func PrepareStringStoreBenchmarker 

func PrepareStringStoreBenchmarker(driver StringStoreDriver) StringStoreBenchmarker

PrepareStringStoreBenchmarker prepares a reusable suite for StringStore driver benchmarks.

type StringStoreDriver 

type StringStoreDriver interface {
	New(*DriverConfig) (StringStore, error)
}

StringStoreDriver represents an interface for creating a handle to the storage of strings.

type StringStoreTester 

type StringStoreTester interface {
	TestStringStore(*testing.T, *DriverConfig)
}

StringStoreTester is a collection of tests for a StringStore driver. Every benchmark expects a new, clean storage. Every benchmark should be called with a DriverConfig that ensures this.

func PrepareStringStoreTester 

func PrepareStringStoreTester(driver StringStoreDriver) StringStoreTester

PrepareStringStoreTester prepares a reusable suite for StringStore driver tests.

Source Files

View all Source files

Directories

Path Synopsis
middleware
client
infohash
ip
response
swarm

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.