bgpfinder

package module
v0.0.0-...-8c88e5d Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 21, 2022 License: BSD-2-Clause Imports: 9 Imported by: 0

README

bgpfinder

These are WIP notes about the bgpfinder project. Eventually this can be tidied up as user documentation.

Technology

  • Language: Go. It's what I know best these days, and it's easy to do HTTP things with it.
  • DB:
    • I'm open to suggestions
    • We don't need to pick just yet
    • Maybe a combo of redis and postgres?
    • Nothing newfangled.
  • Build & Deploy:
    • Let's get Github Actions set up ASAP
    • Use to run tests and build docker images

Packages

base package: finder (used as a library)
  • keep it simple. just support what we need. don’t make it too generic (hard coded stuff for a project is fine)
  • handle only http urls (deal with kafka etc elsewhere, if you want a local archive, do it with object storage)
  • defines Finder interface that other packages may implement
  • just hit website and do parsing. it’s up to the caller to figure out lengths etc
  • don’t worry much about edge cases where timestamps in files are off (leave it to the caller to ask for more time than they need if they care)
  • intervals are inclusive,exclusive
    • e.g., midnight-1am returns 00,15,45 for rv collectors
  • simple CLI that enables testing while development as well as use in scripts
    • e.g., find files matching a query and then pipe into xargs wget
Finder interface:
  • list of projects
  • list of collectors (ideally dynamic)
    • (optionally) overall approx time range for collector
  • give me all the URLs for a project/collector/time window
next package: server
  • wrap finder in http server
  • backwards compat with bgpstream broker API
  • still no db. just fire up finder for every request
  • designed to be run as sidecar to bgpstream instance(s)
  • allows bgpstream to work without extra infra (or if broker infra goes down)
  • hopefully will make production use more appealing
and then: cache/database
  • similar to current downloader
  • runs as daemon
  • uses finder package
  • periodically schedules finds across proj/coll/time
    • schedule finds for recent windows regularly, less frequent for old windows
  • compares found URLs to DB, updates accordingly.
  • implements finder interface so that server and/or other embedded users can benefit from cache
finally: public instance(s)
  • run finder server in several stable places
  • update bgpstream broker code to call finder server
  • all bgpstream features should work in this way

Documentation

Index

Constants

View Source 
const (
	RIS             = "ris"
	RIS_ARCHIVE_URL = "https://data.ris.ripe.net/"
	// XXX: it's tempting, but we can't use
	// https://www.ris.ripe.net/peerlist/ because it only lists
	// currently-active collectors.
	RIS_COLLECTORS_URL = RIS_ARCHIVE_URL
)
View Source 
const (
	ROUTEVIEWS                = "routeviews"
	ROUTEVIEWS_ARCHIVE_URL    = "http://archive.routeviews.org/"
	ROUTEVIEWS_COLLECTORS_URL = ROUTEVIEWS_ARCHIVE_URL
)

Variables

View Source 
var (
	ROUTEVIEWS_PROJECT = Project{Name: ROUTEVIEWS}

	// These are last-resort overrides to "fix" an out-of-pattern RV
	// collector name.
	ROUTEVIEWS_COLLECTOR_OVERRIDES = map[string][2]string{
		"": {"route-views2", "rv2"},
	}

	ROUTEVIEWS_DUMP_TYPES = map[DumpType]rvDumpType{
		DUMP_TYPE_RIB: {
			DumpType: DUMP_TYPE_RIB,
			Duration: time.Hour,
			URL:      "RIBS",
			Regexp:   rvRIBFileRe,
		},
		DUMP_TYPE_UPDATES: {
			DumpType: DUMP_TYPE_UPDATES,
			Duration: time.Minute * 15,
			URL:      "UPDATES",
			Regexp:   rvUpdatesFileRe,
		},
	}
)
View Source 
var DefaultFinder = mustInitDefaultFinder()

Global finder instance that includes all the built-in finder implementations (RV and RIS for now).

If you have a custom (private) finder, you can either register it with this finder instance, or use it directly.

View Source 
var (
	RIS_PROJECT = Project{Name: RIS}
)
View Source 
var ZeroCollector = Collector{}

Functions

This section is empty.

Types

type Collector 

type Collector struct {
	// Project name the collector belongs to
	Project Project `json:"project"`

	// Name of the collector
	Name string `json:"name"`

	// Project-internal name for this collector
	InternalName string `json:"internal_name"`
}

func Collectors 

func Collectors(project string) ([]Collector, error)

func GetCollector 

func GetCollector(name string) (Collector, error)

func (Collector) AsCSV 

func (c Collector) AsCSV() string

func (Collector) String 

func (c Collector) String() string

type DumpType 

type DumpType uint8
const (
	DUMP_TYPE_ANY     DumpType = 0 // any
	DUMP_TYPE_RIB     DumpType = 1 // rib
	DUMP_TYPE_UPDATES DumpType = 2 // updates
)

func DumpTypeString 

func DumpTypeString(s string) (DumpType, error)

DumpTypeString retrieves an enum value from the enum constants string name. Throws an error if the param is not part of the enum.

func DumpTypeValues 

func DumpTypeValues() []DumpType

DumpTypeValues returns all values of the enum

func (DumpType) IsADumpType 

func (i DumpType) IsADumpType() bool

IsADumpType returns "true" if the value is listed in the enum definition. "false" otherwise

func (DumpType) MarshalJSON 

func (i DumpType) MarshalJSON() ([]byte, error)

MarshalJSON implements the json.Marshaler interface for DumpType

func (DumpType) MarshalText 

func (i DumpType) MarshalText() ([]byte, error)

MarshalText implements the encoding.TextMarshaler interface for DumpType

func (DumpType) String 

func (i DumpType) String() string

func (*DumpType) UnmarshalJSON 

func (i *DumpType) UnmarshalJSON(data []byte) error

UnmarshalJSON implements the json.Unmarshaler interface for DumpType

func (*DumpType) UnmarshalText 

func (i *DumpType) UnmarshalText(text []byte) error

UnmarshalText implements the encoding.TextUnmarshaler interface for DumpType

type File 

type File struct {
	// URL of the file
	URL string

	// Collector that collected this file
	Collector Collector

	// Nominal dump duration
	Duration time.Duration

	// Dump type
	DumpType DumpType
}

Represents a single BGP file found by a Finder. TODO: better name for this. Dump is a candidate.

func Find 

func Find(query Query) ([]File, error)

type Finder 

type Finder interface {
	// Get the list of projects supported by this finder
	Projects() ([]Project, error)

	// Get a specific project
	Project(name string) (Project, error)

	// Get the list of collectors supported by the given project. All
	// projects if unset.
	Collectors(project string) ([]Collector, error)

	// Get a specific collector by name, returns the zero collector if there
	// is no such collector
	Collector(name string) (Collector, error)

	// Find all the BGP data URLs that match the given query
	Find(query Query) ([]File, error)
}

Just a sketch of what the base Finder interface might look like. Everything gets built on top of (or under, I guess) this.

type Logger 

type Logger struct {
	zerolog.Logger
}

func NewLogger 

func NewLogger(cfg LoggerConfig) (*Logger, error)

func (Logger) ModuleLogger 

func (l Logger) ModuleLogger(module string) Logger

Create a sub-logger with the given module name

type LoggerConfig 

type LoggerConfig struct {
	LogLevel string `help:"Log level" default:"info"`
}

type MultiFinder 

type MultiFinder struct {
	// contains filtered or unexported fields
}

Finder implementation that handles routing requests to a set of sub finder instances.

func NewMultiFinder 

func NewMultiFinder(finders ...Finder) (*MultiFinder, error)

func (*MultiFinder) AddFinder 

func (m *MultiFinder) AddFinder(f Finder) error

func (*MultiFinder) Collector 

func (m *MultiFinder) Collector(name string) (Collector, error)

func (*MultiFinder) Collectors 

func (m *MultiFinder) Collectors(project string) ([]Collector, error)

func (*MultiFinder) Find 

func (m *MultiFinder) Find(query Query) ([]File, error)

func (*MultiFinder) Project 

func (m *MultiFinder) Project(name string) (Project, error)

func (*MultiFinder) Projects 

func (m *MultiFinder) Projects() ([]Project, error)

type Project 

type Project struct {
	Name string `json:"name"`
}

func GetProject 

func GetProject(name string) (Project, error)

func Projects 

func Projects() ([]Project, error)

func (Project) AsCSV 

func (p Project) AsCSV() string

func (Project) String 

func (p Project) String() string

type Query 

type Query struct {
	// Collectors to search for. All collectors if unset/empty
	Collectors []Collector

	// Query window start time (inclusive)
	From time.Time

	// Query window end time (exclusive)
	Until time.Time

	// Dump type to search for. Any type if unset
	DumpType DumpType
}

TODO: think about how this should work -- just keep it simple! no complex query structures

TODO: add Validate method (e.g., From is before Until, IsADumpType, etc.)

type RISFinder 

type RISFinder struct {
	// contains filtered or unexported fields
}

func NewRISFinder 

func NewRISFinder() *RISFinder

func (*RISFinder) Collector 

func (f *RISFinder) Collector(name string) (Collector, error)

func (*RISFinder) Collectors 

func (f *RISFinder) Collectors(project string) ([]Collector, error)

func (*RISFinder) Find 

func (f *RISFinder) Find(query Query) ([]File, error)

func (*RISFinder) Project 

func (f *RISFinder) Project(name string) (Project, error)

func (*RISFinder) Projects 

func (f *RISFinder) Projects() ([]Project, error)

type RouteViewsFinder 

type RouteViewsFinder struct {
	// contains filtered or unexported fields
}

TODO: Finder implementation for the RouteViews archive TODO: refactor a this common caching-finder code out so that RIS and PCH can use it

func NewRouteViewsFinder 

func NewRouteViewsFinder() *RouteViewsFinder

func (*RouteViewsFinder) Collector 

func (f *RouteViewsFinder) Collector(name string) (Collector, error)

func (*RouteViewsFinder) Collectors 

func (f *RouteViewsFinder) Collectors(project string) ([]Collector, error)

func (*RouteViewsFinder) Find 

func (f *RouteViewsFinder) Find(query Query) ([]File, error)

func (*RouteViewsFinder) Project 

func (f *RouteViewsFinder) Project(name string) (Project, error)

func (*RouteViewsFinder) Projects 

func (f *RouteViewsFinder) Projects() ([]Project, error)

Source Files

View all Source files

Directories

Path Synopsis
cmd
bgpf

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.