measurex

package
v0.0.0-...-41ba115 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 30, 2023 License: GPL-3.0 Imports: 29 Imported by: 0

Documentation

Overview

Package measurex contains an heavily modified internal/measurex.

Index

Constants

View Source 
const (
	// DNSLookupFlagNS modifies the DNSLookupPlan to request resolving
	// the target domain's nameservers using NS.
	DNSLookupFlagNS = 1 << iota

	// DNSLookupFlagHTTPS modifies the DNSLookupPlan to request resolving
	// the target domain using HTTPSSvc.
	DNSLookupFlagHTTPS
)
View Source 
const (
	// DefaultDNSLookupTimeout is the default Options.DNSLookupTimeout value.
	DefaultDNSLookupTimeout = 4 * time.Second

	// DefaultDNSParallelism is the default Options.DNSParallelism value.
	DefaultDNSParallelism = 4

	// DefaultEndpointParallelism is the default Options.EndpointParallelism value.
	DefaultEndpointParallelism = 8

	// DefaultHTTPGETTimeout is the default Options.HTTPGETTimeout value.
	DefaultHTTPGETTimeout = 15 * time.Second

	// DefaultMaxAddressPerFamily is the default value of Options.MaxAddressesPerFamily. For
	// experiments like websteps, where we have a TH, we're actually going to test twice this
	// number when there are many IP addresses per domain, because we're also going to take
	// into account some TH-tested addresses. Going below the recommended value of 2 here
	// is not recommended for websteps. Because websteps uses more than one resolver and
	// because measurex tries to arrange addresses so that we intermix system-resolver and
	// non-system-resolver resolutions, if you use less than two here you are going to
	// only test the first IP address returned by the system resolver, which means you're
	// probably going to miss part of the censorship that's there.
	DefaultMaxAddressPerFamily = 2

	// DefaultMaxCrawlerDepth is the default value of Options.MaxCrawlerDepth.
	DefaultMaxCrawlerDepth = 3

	// DefaultMaxHTTPResponseBodySnapshotSize is the default value
	// of Options.MaxHTTPResponseBodySnapshotSize.
	DefaultMaxHTTPResponseBodySnapshotSize = 1 << 19

	// DefaultMaxHTTPSResponseBodySnapshotSizeConnectivity is the default value
	// of Options.MaxHTTPSResponseBodySnapshotSize when the URL path is "/",
	// where we assume we just want to check connectivity.
	DefaultMaxHTTPSResponseBodySnapshotSizeConnectivity = 1 << 12

	// DefaultMaxHTTPSResponseBodySnapshotSizeThrottling is the default value
	// of Options.MaxHTTPSResponseBodySnapshotSize when the URL path is no "/",
	// where we assume we want to detect signs of throttling.
	DefaultMaxHTTPSResponseBodySnapshotSizeThrottling = 1 << 19

	// DefaultQUICHandshakeTimeout is the default Options.QUICHandshakeTimeout value.
	DefaultQUICHandshakeTimeout = 10 * time.Second

	// DefaultTCPConnectTimeout is the default Options.TCPConnectTimeout value.
	DefaultTCPConnectTimeout = 15 * time.Second

	// DefaultTLSHandshakeTimeout is the default Options.TLSHandshakeTimeout value.
	DefaultTLSHandshakeTimeout = 10 * time.Second
)
View Source 
const (
	// URLAddressSupportsHTTP3 indicates that a given URL address supports HTTP3.
	URLAddressSupportsHTTP3 = 1 << iota

	// URLAddressAlreadyTestedHTTP indicates that this address has already
	// been tested using the cleartext HTTP protocol.
	URLAddressAlreadyTestedHTTP

	// URLAddressAlreadyTestedHTTPS indicates that this address has already
	// been tested using the encrypted HTTPS protocol.
	URLAddressAlreadyTestedHTTPS

	// URLAddressAlreadyTestedHTTP3 indicates that this address has already
	// been tested using the encrypted HTTP3 protocol.
	URLAddressAlreadyTestedHTTP3

	// URLAddressSystemResolver indicates that this entry has been
	// discovered through the system resolver.
	URLAddressSystemResolver
)
View Source 
const (
	// EndpointPlanningExcludeBogons excludes bogons from NewEndpointPlan's planning.
	EndpointPlanningExcludeBogons = 1 << iota

	// EndpointPlanningOnlyHTTP3 ensures we only test HTTP3.
	EndpointPlanningOnlyHTTP3

	// EndpointPlanningIncludeAll ensures that we include all the IP addresses
	// regardless on any options based restriction on the maximum number of
	// addresses per domain. This flag is used to ensure the TH receives the
	// whole list of IP addresses discovered by the client.
	EndpointPlanningIncludeAll

	// EndpointPlanningMeasureAgain ensures that we include endpoints that
	// we have already measured into the plan.
	EndpointPlanningMeasureAgain
)

Variables

View Source 
var (
	// ErrUnknownURLScheme means that the given
	// endpoint's URL scheme is neither HTTP nor HTTPS.
	ErrUnknownURLScheme = errors.New("unknown URL.Scheme")

	// ErrUnknownEndpointNetwork means that the given endpoint's
	// network is of a type that we don't know how to handle.
	ErrUnknownEndpointNetwork = errors.New("unknown Endpoint.Network")
)
View Source 
var (
	// ErrCrawlerDepth indicates we have reached the maximum crawler depth
	ErrCrawlerDepth = errors.New("reached maximum crawler depth")

	// ErrCrawlerEOF indicates we have measured all URLs.
	ErrCrawlerEOF = errors.New("measured all the provided URLs")
)
View Source 
var DefaultHTTP3Transport = &http3.RoundTripper{}

DefaultHTTP3Transport is the default HTTP3 transport used by this library.

View Source 
var ErrCannotDeterminePortFromURL = errors.New("cannot determine port from URL")

ErrCannotDeterminePortFromURL indicates that we could not determine the correct port from the URL authority and scheme.

View Source 
var ErrHTTPTooManyRedirects = errors.New("stopped after 10 redirects")

ErrHTTPTooManyRedirects is the unexported error that the standard library would return when hitting too many redirects.

Functions

func ALPNForHTTPSEndpoint 

func ALPNForHTTPSEndpoint(network archival.NetworkType) []string

ALPNForHTTPSEndpoint selects the correct ALPN for an HTTP endpoint given the network. On failure, we return an empty list.

func CanonicalURLString 

func CanonicalURLString(URL *SimpleURL) string

CanonicalURLString returns a representation of the given URL that should be more canonical than the random URLs returned by web services.

We need as canonical as possible URLs in URLRedirectDeque because their string representation is used to decide whether we need to follow redirects or not.

SPDX-License-Identifier: MIT

Adapted from: https://github.com/sekimura/go-normalize-url.

func GetWebPageTitle 

func GetWebPageTitle(webpage []byte) string

GetWebPageTitle returns the title or an empty string.

func NewCookieJar 

func NewCookieJar() http.CookieJar

NewCookieJar is a convenience factory for creating an http.CookieJar that is aware of the effective TLS / public suffix list. This means that the jar won't allow a domain to set cookies for another unrelated domain (in the public-suffix-list sense).

func NewHTTPGetRequest 

func NewHTTPGetRequest(ctx context.Context, URL string) (*http.Request, error)

NewHTTPGetRequest is a convenience factory for creating a new http.Request using the GET method and the given URL.

func NewHTTPRequestHeaderForMeasuring 

func NewHTTPRequestHeaderForMeasuring() http.Header

NewHTTPRequestHeaderForMeasuring returns an http.Header where the headers are the ones we use for measuring.

func NewHTTPRequestWithContext 

func NewHTTPRequestWithContext(ctx context.Context,
	method, URL string, body io.Reader) (*http.Request, error)

NewHTTPRequestWithContext is a convenience factory for creating a new HTTP request with the typical headers we use when performing measurements already set inside of req.Header.

func ParseCookies 

func ParseCookies(cookie ...string) []*http.Cookie

ParseCookies parses one or more serialized cookies into a list of *http.Cookie.

func PortFromURL 

func PortFromURL(URL *SimpleURL) (string, error)

PortFromURL returns the port determined from the URL or an error.

func SerializeCookies 

func SerializeCookies(in []*http.Cookie) (out []string)

SerializeCookies takes in input []*http.Cookie and returns a []string where each string is a serialized cookie.

func SerializeCookiesNames 

func SerializeCookiesNames(in []*http.Cookie) (out []string)

SerializeCookiesNames takes in input []*http.Cookie and returns a []string where each string is a cookie name.

func SortedSerializedCookiesNames 

func SortedSerializedCookiesNames(in []*http.Cookie) (out []string)

SortedSerializedCookiesNames returns a sorted copy of the cookies names.

func StringListSortUniq 

func StringListSortUniq(in []string) (out []string)

StringListSortUniq ensures a []string returns a sorted copy of the original list that does not contain any duplicate strings.

func URLAddressListToString 

func URLAddressListToString(ual []*URLAddress) string

URLAddressListToString transforms an URLAddressList to a flat list of IP addresses, which is useful for logging.

Types

type AbstractMeasurer 

type AbstractMeasurer interface {
	// DNSLookups behaves like Measurer.DNSLookups.
	DNSLookups(ctx context.Context,
		dnsLookups ...*DNSLookupPlan) <-chan *DNSLookupMeasurement

	// FlattenOptions returns flattened options.
	FlattenOptions() *Options

	// MeasureEndpoints behaves like Measurer.MeasureEndpoints.
	MeasureEndpoints(ctx context.Context,
		epnts ...*EndpointPlan) <-chan *EndpointMeasurement

	// NewURLMeasurement behaves like Measurer.NewURLMeasurement.
	NewURLMeasurement(input string) (*URLMeasurement, error)

	// NewURLRedirectDeque behaves like Measurer.NewURLRedirectDeque.
	NewURLRedirectDeque() *URLRedirectDeque

	// NextID behaves like Measurer.NextID.
	NextID() int64

	// Redirects behaves like Measurer.Redirects.
	Redirects(epnts []*EndpointMeasurement,
		opts *Options) ([]*URLMeasurement, bool)
}

AbstractMeasurer is an abstract view of a Measurer.

type ArchivalDNSLookupMeasurement 

type ArchivalDNSLookupMeasurement struct {
	// ID is the unique ID of this measurement.
	ID int64 `json:"id,omitempty"`

	// Domain is the domain this lookup refers to.
	Domain string `json:"domain"`

	// ReverseAddress is a convenience field to help analysis that is only
	// set when we're performing a reverse DNS lookup.
	ReverseAddress string `json:"reverse_address,omitempty"`

	// ResolverNetwork is the network used by this resolver.
	ResolverNetwork string `json:"resolver_network"`

	// ResolverAddress is the address used by this resolver.
	ResolverAddress string `json:"resolver_address"`

	// Failure is the failure that occurred.
	Failure *string `json:"failure"`

	// Addresses contains the discovered addresses.
	Addresses []string `json:"addresses"`

	// Queries contains the DNS lookup events.
	Queries []model.ArchivalDNSLookupResult `json:"queries"`
}

ArchivalDNSLookupMeasurement is an archival DNS lookup measurement.

func NewArchivalDNSLookupMeasurementList 

func NewArchivalDNSLookupMeasurementList(
	begin time.Time, in []*DNSLookupMeasurement) (out []ArchivalDNSLookupMeasurement)

NewArchivalDNSLookupMeasurementList converts a []*DNSLookupMeasurement into a []ArchivalDNSLookupMeasurement.

type ArchivalEndpointMeasurement 

type ArchivalEndpointMeasurement struct {
	// ID is the unique ID of this measurement.
	ID int64 `json:"id,omitempty"`

	// URL is the URL we're fetching.
	URL string `json:"url"`

	// Endpoint is the endpoint network.
	Network string `json:"network"`

	// Address is the endpoint address.
	Address string `json:"address"`

	// CookieNames contains the cookie names we sent.
	CookiesNames []string `json:"cookies_names"`

	// Failure is the error that occurred.
	Failure *string `json:"failure"`

	// FailedOperation is the operation that failed.
	FailedOperation *string `json:"failed_operation"`

	// StatusCode is the status code if any.
	StatusCode int64 `json:"status_code"`

	// Location is the redirect location if any.
	Location string `json:"location"`

	// BodyLength is the body length if any.
	BodyLength int64 `json:"body_length"`

	// Title is the webpage title if any.
	Title string `json:"title"`

	// NetworkEvent contains network events (if any).
	NetworkEvents []model.ArchivalNetworkEvent `json:"network_events"`

	// TCPConnect contains the TCP connect event (if any).
	TCPConnect *model.ArchivalTCPConnectResult `json:"tcp_connect"`

	// QUICTLSHandshake contains the QUIC/TLS handshake event (if any).
	QUICTLSHandshake *model.ArchivalTLSOrQUICHandshakeResult `json:"quic_tls_handshake"`

	// HTTPRoundTrip contains the HTTP round trip event (if any).
	HTTPRoundTrip *model.ArchivalHTTPRequestResult `json:"request"`
}

ArchivalEndpointMeasurement is the archival format of an endpoint measurement.

func NewArchivalEndpointMeasurementList 

func NewArchivalEndpointMeasurementList(begin time.Time,
	in []*EndpointMeasurement, bodyFlags int64) (out []ArchivalEndpointMeasurement)

NewArchivalEndpointMeasurementList converts a []*EndpointMeasurement into a []ArchivalEndpointMeasurement.

type ArchivalURLMeasurement 

type ArchivalURLMeasurement struct {
	// ID is the unique ID of this URLMeasurement.
	ID int64 `json:"id"`

	// EndpointIDs contains the ID of the EndpointMeasurement(s) that
	// generated this URLMeasurement through redirects.
	EndpointIDs []int64 `json:"endpoint_ids"`

	// URL is the underlying URL to measure.
	URL string `json:"url"`

	// Cookies contains the cookies.
	Cookies []string `json:"cookies"`

	// DNS contains a list of DNS measurements.
	DNS []ArchivalDNSLookupMeasurement `json:"dns"`

	// Endpoint contains a list of endpoint measurements.
	Endpoint []ArchivalEndpointMeasurement `json:"endpoint"`
}

ArchivalURLMeasurement is the archival format of an URL measurement.

type Cache 

type Cache struct {
	// DisableNetwork allows to disable network operations.
	DisableNetwork bool

	// DNS is a reference to the underlying DNS cache.
	DNS *caching.FSCache

	// Endpoint is a reference to the underlying endpoint cache.
	Endpoint *caching.FSCache
}

Cache is a cache for measurex DNS and endpoint measurements.

func NewCache 

func NewCache(dirpath string) *Cache

NewCache creates a new cache inside the given directory.

func (*Cache) FindDNSLookupMeasurement 

func (c *Cache) FindDNSLookupMeasurement(
	plan *DNSLookupPlan, policy CachingPolicy) (*DNSLookupMeasurement, bool)

FindDNSLookupMeasurement searches for a DNSLookupMeasurement compatible with the plan.

func (*Cache) FindEndpointMeasurement 

func (c *Cache) FindEndpointMeasurement(
	plan *EndpointPlan, policy CachingPolicy) (*EndpointMeasurement, bool)

FindEndpointMeasurement finds the endpoint measurement deriving from the given plan.

func (*Cache) StartTrimmer 

func (c *Cache) StartTrimmer(ctx context.Context)

StartTrimmer starts a background goroutine that runs until the given context is active and periodically trims the cache.

func (*Cache) StoreDNSLookupMeasurement 

func (c *Cache) StoreDNSLookupMeasurement(dlm *DNSLookupMeasurement) error

StoreDNSLookupMeasurement stores the given measurement into the cache.

func (*Cache) StoreEndpointMeasurement 

func (c *Cache) StoreEndpointMeasurement(em *EndpointMeasurement) error

StoreEndpointMeasurement stores the given measurement in cache.

func (*Cache) Trim 

func (c *Cache) Trim()

Trim removes old entries from the cache.

type CachedDNSLookupMeasurement 

type CachedDNSLookupMeasurement struct {
	T time.Time
	M *DNSLookupMeasurement
}

CachedDNSLookupMeasurement is the cached form of a DNSLookupMeasurement.

type CachedEndpointMeasurement 

type CachedEndpointMeasurement struct {
	T time.Time
	M *EndpointMeasurement
}

CachedEndpointMeasurement is the cached form of an EndpointMeasurement.

type CachingMeasurer 

type CachingMeasurer struct {
	// contains filtered or unexported fields
}

CachingMeasurer is a measurer using a local cache.

The cache works as follows:

1. it stores all the DNS lookup measurements using the same target domain into the same bucket and checks for equality using a strict definition of equality that includes not only the domain but also the lookup type to distinguish between similar DNS lookups for the same domain.

2. it stores all the endpoint measurements using as key their summary, which should avoid creating too large buckets.

On disk, the cache stores a list of records having the same ~unique sha256. Even when there should only be a single record with a given identifier (unlikely for domains but much more likely for endpoints) we use a list on disk just in case there's going to be any hash collision.

func NewCachingMeasurer 

func NewCachingMeasurer(mx AbstractMeasurer,
	cache *Cache, policy CachingPolicy) *CachingMeasurer

NewCachingMeasurer takes in input an existing measurer and the cache and returns a new instance of CachingMeasurer.

func (*CachingMeasurer) DNSLookups 

func (mx *CachingMeasurer) DNSLookups(ctx context.Context,
	dnsLookups ...*DNSLookupPlan) <-chan *DNSLookupMeasurement

DNSLookups implements AbstractMeasurer.DNSLookups.

func (*CachingMeasurer) FlattenOptions 

func (mx *CachingMeasurer) FlattenOptions() *Options

FlattenOptions implements AbstractMeasurer.FlattenOptions.

func (*CachingMeasurer) MeasureEndpoints 

func (mx *CachingMeasurer) MeasureEndpoints(ctx context.Context,
	epnts ...*EndpointPlan) <-chan *EndpointMeasurement

MeasureEndpoints implements AbstractMeasurer.MeasureEndpoints.

func (*CachingMeasurer) NewURLMeasurement 

func (mx *CachingMeasurer) NewURLMeasurement(input string) (*URLMeasurement, error)

NewURLMeasurement implements AbstractMeasurer.NewURLMeasurement.

func (*CachingMeasurer) NewURLRedirectDeque 

func (mx *CachingMeasurer) NewURLRedirectDeque() *URLRedirectDeque

NewURLRedirectDeque implements AbstractMeasurer.NewURLRedirectDeque.

func (*CachingMeasurer) NextID 

func (mx *CachingMeasurer) NextID() int64

NextID implements AbstractMeasurer.NextID.

func (*CachingMeasurer) Redirects 

func (mx *CachingMeasurer) Redirects(epnts []*EndpointMeasurement,
	opts *Options) ([]*URLMeasurement, bool)

Redirects implements AbstractMeasurer.Redirects.

type CachingPolicy 

type CachingPolicy interface {
	// StaleDNSLookupMeasurement returns whether a DNSLookupMeasurement is stale.
	StaleDNSLookupMeasurement(m *CachedDNSLookupMeasurement) bool

	// StaleEndpointMeasurement returns whether an EndpointMeasurement is stale.
	StaleEndpointMeasurement(m *CachedEndpointMeasurement) bool
}

CachingPolicy allows to customize che CachingMeasurer policy.

func CachingForeverPolicy 

func CachingForeverPolicy() CachingPolicy

CachingForeverPolicy returns a policy that caches entries forever.

func ReasonableCachingPolicy 

func ReasonableCachingPolicy() CachingPolicy

ReasonableCachingPolicy returns a reasonable caching policy.

type Crawler 

type Crawler struct {
	// Measurer is the measurer to use.
	Measurer AbstractMeasurer

	// Options contains options. If this field is nil, we will
	// end up using the default option values.
	Options *Options

	// Resolvers contains the resolvers to use.
	Resolvers []*DNSResolverInfo
}

Crawler starts from an input URL and visits the forest of all the URLs deriving from such an URL.

Please, either use NewCrawler to create a new instance or ensure you initialize all this struct's fields.

func NewCrawler 

func NewCrawler(mx AbstractMeasurer) *Crawler

NewCrawler creates a new instance of Crawler.

func (*Crawler) Crawl 

func (c *Crawler) Crawl(ctx context.Context, URL string) (<-chan *URLMeasurement, error)

Crawl visits the given URL.

type DNSLookupMeasurement 

type DNSLookupMeasurement struct {
	// ID is the unique ID of this measurement.
	ID int64 `json:",omitempty"`

	// URLMeasurementID is the ID of the parent URLMeasurement. We do not
	// emit this information to JSON because it is redundant, but it's still
	// handy to know it when we're processing measurements.
	URLMeasurementID int64 `json:"-"`

	// ReverseAddress is a convenience field holding the addr for
	// which we issued a reverse lookup, which only makes sense when
	// we're actually performing a reverse lookup.
	ReverseAddress string `json:",omitempty"`

	// Lookup contains the DNS lookup event. This field contains a summary
	// of the information discovered during this lookup. We recommend using
	// this structure for processing the results.
	Lookup *archival.FlatDNSLookupEvent

	// RoundTrip contains DNS round trips. This field contains one or
	// more round trips performed during the lookup. The system resolver
	// fakes out a round trip with query type ANY and all the info
	// that we could gather from calling getaddrinfo (or equivalent).
	RoundTrip []*archival.FlatDNSRoundTripEvent `json:",omitempty"`
}

DNSLookupMeasurement is a DNS lookup measurement.

func NewFakeHTTPSSvcDNSLookupMeasurement 

func NewFakeHTTPSSvcDNSLookupMeasurement(mx AbstractMeasurer,
	resolverNetwork archival.NetworkType, resolverAddress string,
	domain string, alpns []string, addresses []string) *DNSLookupMeasurement

NewFakeHTTPSSvcDNSLookupMeasurement creates a fake DNSLookupMeasurement from IP addresses obtained from an external source.

This factory is the best solution to fake a DNS lookup that can be compared with other DNS lookups. Because the returned lookup may also include ALPN information, we're faking an HTTPSSvc lookup result.

func (*DNSLookupMeasurement) ALPNs 

func (dlm *DNSLookupMeasurement) ALPNs() []string

ALPNs returns the list of ALPNs we discovered during the lookup.

func (*DNSLookupMeasurement) Addresses 

func (dlm *DNSLookupMeasurement) Addresses() []string

Addresses returns the list of addresses we discovered during the lookup.

func (*DNSLookupMeasurement) CNAME 

func (dlm *DNSLookupMeasurement) CNAME() string

CNAME returns the CNAME we discovered during the lookup.

func (*DNSLookupMeasurement) CouldDeriveFrom 

func (dlm *DNSLookupMeasurement) CouldDeriveFrom(p *DNSLookupPlan) bool

CouldDeriveFrom returns true if this measurement could have been the result of the plan provided as argument. This is true when the summary of the measurement is equal to the plan's summary.

func (*DNSLookupMeasurement) Describe 

func (dlm *DNSLookupMeasurement) Describe() string

Describe returns a compact human-readable description of this measurement.

func (*DNSLookupMeasurement) Domain 

func (dlm *DNSLookupMeasurement) Domain() string

Domain returns the domain for which we issued a DNS lookup.

func (*DNSLookupMeasurement) Failure 

func (dlm *DNSLookupMeasurement) Failure() archival.FlatFailure

Failure returns the failure that occurred.

func (*DNSLookupMeasurement) FinishedUnixNano 

func (dlm *DNSLookupMeasurement) FinishedUnixNano() int64

FinishedUnixNano returns the time when this measurement finished expressed in nanoseconds since the UNIX epoch.

func (*DNSLookupMeasurement) IsAnotherInstanceOf 

func (dlm *DNSLookupMeasurement) IsAnotherInstanceOf(o *DNSLookupMeasurement) bool

IsAnotherInstanceOf returns whether the current measurement is another instance of the other measurement, `o`. This is true when they have equal summary.

func (*DNSLookupMeasurement) IsCompatibleWith 

func (dlm *DNSLookupMeasurement) IsCompatibleWith(other *DNSLookupMeasurement) bool

IsCompatibleWith returns whether the current DNSLookupMeasurement can safely be compared with another DNSLookupMeasurement regardless of which specific DNS resolver has been used to perform the two measurements.

1. they are resolving the same domain;

2. they use the same lookup type.

A weaker definition of compatibility is provided by IsWeaklyCompatibleWith.

func (*DNSLookupMeasurement) IsWeaklyCompatibleWith 

func (dlm *DNSLookupMeasurement) IsWeaklyCompatibleWith(other *DNSLookupMeasurement) bool

IsWeaklyCompatibleWith returns whether the current DNSLookupMeasurement can safely be compared with another DNSLookupMeasurement regardless of which specific DNS resolver has been used to perform the two measurements.

We say that two DNSLookupMeasurements are weakly compatible if:

1. they are relative to the same domain;

2. the lookup type is compatible.

The following table shows when two lookup types are weakly compatible: 

+-------------+-------------+-------+--------+---------+
|             | getaddrinfo | https |   ns   | reverse |
+-------------+-------------+-------+--------+---------+
| getaddrinfo |     yes     |  yes  |   no   |   no    |
+-------------+-------------+-------+--------+---------+
|    https    |     yes     |  yes  |   no   |   no    |
+-------------+-------------+-------+--------+---------+
|      ns     |      no     |   no  |  yes   |   no    |
+-------------+-------------+-------+--------+---------+
|   reverse   |      no     |   no  |   no   |  yes    |
+-------------+-------------+-------+--------+---------+

In addition, two lookup types are _always_ weakly compatible when they're the same, even if they are not listed in the above table.

A stronger definition of compatibility is provided by IsCompatibleWith.

func (*DNSLookupMeasurement) LookupType 

func (dlm *DNSLookupMeasurement) LookupType() archival.DNSLookupType

LookupType returns the lookup type (e.g., getaddrinfo, NS).

func (*DNSLookupMeasurement) NS 

func (dlm *DNSLookupMeasurement) NS() []string

NS returns the list of NS we discovered during the lookup.

func (*DNSLookupMeasurement) PTRs 

func (dlm *DNSLookupMeasurement) PTRs() []string

PTRs returns the PTRs we discovered during the lookup.

func (*DNSLookupMeasurement) ResolverAddress 

func (dlm *DNSLookupMeasurement) ResolverAddress() string

ResolverAddress returns the resolver address (e.g., 8.8.8.8:53).

func (*DNSLookupMeasurement) ResolverNetwork 

func (dlm *DNSLookupMeasurement) ResolverNetwork() archival.NetworkType

ResolverNetwork returns the resolver network (e.g., udp, system).

func (*DNSLookupMeasurement) ResolverURL 

func (dlm *DNSLookupMeasurement) ResolverURL() string

ResolverURL returns a URL containing the resolver's network and address. For DoH resolvers, we just return the URL. For all the other resolvers, we use the network as the scheme and the address as the URL host.

func (*DNSLookupMeasurement) Runtime 

func (dlm *DNSLookupMeasurement) Runtime() (out time.Duration)

Runtime returns the time elapsed waiting for the lookup to complete.

func (*DNSLookupMeasurement) Summary 

func (dlm *DNSLookupMeasurement) Summary() string

Summary returns a string representing the DNS measurement's plan. Two measurements are ~same if they have the same summary.

The summary of a DNS lookup consists of these fields:

- domain - lookupType - resolver network - resolver address

If the measurement is nil, we return the empty string.

func (*DNSLookupMeasurement) SupportsHTTP3 

func (dlm *DNSLookupMeasurement) SupportsHTTP3() bool

SupportsHTTP3 returns whether this DNSLookupMeasurement includes the "h3" ALPN in the list of ALPNs for this domain.

func (*DNSLookupMeasurement) ToArchival 

func (m *DNSLookupMeasurement) ToArchival(begin time.Time) ArchivalDNSLookupMeasurement

ToArchival converts a DNSLookupMeasurement to ArchivalDNSLookupMeasurement.

func (*DNSLookupMeasurement) UsingResolverIPv6 

func (dlm *DNSLookupMeasurement) UsingResolverIPv6() (usingIPv6 bool)

UsingResolverIPv6 returns whether this DNS lookups used an IPv6 resolver.

type DNSLookupPlan 

type DNSLookupPlan struct {
	// URLMeasurementID is the ID of the original URLMeasurement.
	URLMeasurementID int64 `json:",omitempty"`

	// Domain is the domain to measure.
	Domain string

	// ReverseAddress is a convenience field holding the addr for
	// which we issued a reverse lookup, which only makes sense when
	// we're actually performing a reverse lookup.
	ReverseAddress string `json:",omitempty"`

	// LookupType is the type of lookup to perform.
	LookupType archival.DNSLookupType

	// Options contains the options. If nil we'll use default values.
	Options *Options

	// Resolver is the resolver to use.
	Resolver *DNSResolverInfo
}

DNSLookupPlan is a plan for performing a DNS lookup.

func NewDNSLookupPlans 

func NewDNSLookupPlans(domain string, options *Options,
	flags int64, ri ...*DNSResolverInfo) []*DNSLookupPlan

NewDNSLookupPlans creates a plan for measuring the given domain with the given options using the given list of resolvers. By default, we perform a getaddrinfo like lookup (i.e., A and AAAA). Use flags to add additional lookup types. For example, you can add a NS lookup using DNSLookupTypeNS.

func (*DNSLookupPlan) Clone 

func (dlp *DNSLookupPlan) Clone() (out *DNSLookupPlan)

Clone creates a DNSLookupPlan deep copy.

func (*DNSLookupPlan) CloneWithLookupType 

func (dlp *DNSLookupPlan) CloneWithLookupType(lt archival.DNSLookupType) (out *DNSLookupPlan)

CloneWithLookupType clones the original plan, configures the required lookup type, and returns the modified clone.

func (*DNSLookupPlan) Equals 

func (dlp *DNSLookupPlan) Equals(other *DNSLookupPlan) bool

Equals returns wheter a plan equals another plan. Two plans are equal if they have the same summary.

func (*DNSLookupPlan) ResolverAddress 

func (dlp *DNSLookupPlan) ResolverAddress() string

ResolverAddress returns the resolver address.

func (*DNSLookupPlan) ResolverNetwork 

func (dlp *DNSLookupPlan) ResolverNetwork() archival.NetworkType

ResolverNetwork returns the resolver network.

func (*DNSLookupPlan) Summary 

func (dlp *DNSLookupPlan) Summary() string

Summary returns a string representing the DNS lookup's plan. Two plans are ~same if they have the same summary.

The summary of a DNS lookup consists of these fields:

- domain - lookupType - resolver network - resolver address

If the plan is nil, we return the empty string.

type DNSResolverInfo 

type DNSResolverInfo struct {
	// Network is the resolver's network (e.g., "doh", "udp")
	Network archival.NetworkType

	// Address is the address (e.g., "1.1.1.1:53", "https://1.1.1.1/dns-query")
	Address string
}

DNSResolverInfo contains info about a DNS resolver.

func NewResolversHTTPS 

func NewResolversHTTPS(urls ...string) []*DNSResolverInfo

NewResolversHTTPS creates a list of HTTPS resolvers from a list of URLs.

func NewResolversUDP 

func NewResolversUDP(endpoints ...string) []*DNSResolverInfo

NewResolversUDP creates a list of UDP resolvers from a list of endpoints.

func (*DNSResolverInfo) Clone 

func (dri *DNSResolverInfo) Clone() (out *DNSResolverInfo)

Clone returns a clone of this resolver info.

func (*DNSResolverInfo) Equals 

func (dri *DNSResolverInfo) Equals(other *DNSResolverInfo) bool

Equals returns whether a DNSResolverInfo equals another one.

type EndpointMeasurement 

type EndpointMeasurement struct {
	// ID is the unique ID of this measurement.
	ID int64 `json:",omitempty"`

	// URLMeasurementID is the ID of the URLMeasurement that created us.
	URLMeasurementID int64 `json:"-"`

	// URL is the URL this measurement refers to.
	URL *SimpleURL

	// Network is the network of this endpoint.
	Network archival.NetworkType

	// Address is the address of this endpoint.
	Address string

	// Options contains the options used for the measurement.
	Options *Options

	// OrigCookies contains the cookies we originally used.
	OrigCookies []*http.Cookie `json:",omitempty"`

	// Finished is when this measurement is finished.
	Finished time.Time

	// Failure is the error that occurred.
	Failure archival.FlatFailure `json:",omitempty"`

	// FailedOperation is the operation that failed.
	FailedOperation FlatFailedOperation `json:",omitempty"`

	// NewCookies contains cookies the next redirection (if any) should use.
	NewCookies []*http.Cookie `json:",omitempty"`

	// Location is the URL we're redirected to (if any).
	Location *SimpleURL `json:",omitempty"`

	// HTTPTitle is the webpage title (if any).
	HTTPTitle string `json:",omitempty"`

	// NetworkEvent contains network events (if any).
	NetworkEvent []*archival.FlatNetworkEvent `json:",omitempty"`

	// TCPConnect contains the TCP connect event (if any).
	TCPConnect *archival.FlatNetworkEvent `json:",omitempty"`

	// QUICTLSHandshake contains the QUIC/TLS handshake event (if any).
	QUICTLSHandshake *archival.FlatQUICTLSHandshakeEvent `json:",omitempty"`

	// HTTPRoundTrip contains the HTTP round trip event (if any).
	HTTPRoundTrip *archival.FlatHTTPRoundTripEvent `json:",omitempty"`
}

EndpointMeasurement is an endpoint measurement.

func (*EndpointMeasurement) BodyIsTruncated 

func (em *EndpointMeasurement) BodyIsTruncated() bool

BodyIsTruncated returns whether the body is truncated. If there's no body, this function returns false.

func (*EndpointMeasurement) BodyLength 

func (em *EndpointMeasurement) BodyLength() int64

BodyLength returns the body length. If there's no body, it returns zero.

func (*EndpointMeasurement) CouldDeriveFrom 

func (em *EndpointMeasurement) CouldDeriveFrom(plan *EndpointPlan) bool

CouldDeriveFrom returns whether this endpoint could derive from the given plan. This happens when they have the same summary.

func (*EndpointMeasurement) Describe 

func (em *EndpointMeasurement) Describe() string

Describe describes this measurement.

func (*EndpointMeasurement) EndpointAddress 

func (em *EndpointMeasurement) EndpointAddress() string

EndpointAddress returns a string like "{address}/{network}".

func (*EndpointMeasurement) FinishedUnixNano 

func (em *EndpointMeasurement) FinishedUnixNano() int64

FinishedUnixNano returns the UnixNano time when this measurement was finished.

func (*EndpointMeasurement) IPAddress 

func (em *EndpointMeasurement) IPAddress() string

IPAddress returns the IP address used in this EndpointMeasurement.

func (*EndpointMeasurement) IsAnotherInstanceOf 

func (em *EndpointMeasurement) IsAnotherInstanceOf(other *EndpointMeasurement) bool

IsAnotherInstanceOf returns whether this EndpointMeasurement is another instance of the other EndpointMeasurement.

This happens when the two measurements have the same summary.

func (*EndpointMeasurement) IsHTTP3Measurement 

func (em *EndpointMeasurement) IsHTTP3Measurement() bool

IsHTTP3Measurement returns whether this EndpointMeasurement measures the encrypted HTTPS protocol using the QUIC network.

func (*EndpointMeasurement) IsHTTPMeasurement 

func (em *EndpointMeasurement) IsHTTPMeasurement() bool

IsHTTPMeasurement returns whether this EndpointMeasurement measures the cleartex HTTP protocol using the TCP network.

func (*EndpointMeasurement) IsHTTPRedirect 

func (em *EndpointMeasurement) IsHTTPRedirect() bool

IsHTTPRedirect returns whether this endpoint contains an HTTP redirect.

func (*EndpointMeasurement) IsHTTPSMeasurement 

func (em *EndpointMeasurement) IsHTTPSMeasurement() bool

IsHTTPSMeasurement returns whether this EndpointMeasurement measures the encrypted HTTPS protocol using the TCP network.

func (*EndpointMeasurement) LocationAsString 

func (em *EndpointMeasurement) LocationAsString() string

LocationAsString converts the location URL to string. If the location URL is nil, we return an empty string.

func (*EndpointMeasurement) RedirectLocationDomain 

func (em *EndpointMeasurement) RedirectLocationDomain() string

RedirectLocationDomain returns the domain of the redirect location.

func (*EndpointMeasurement) RedirectSummary 

func (em *EndpointMeasurement) RedirectSummary() (string, bool)

RedirectSummary is a summary of the endpoint's redirect. If there's no redirect, we return an empty string and false. Otherwise, we return a string that uniquely identifies this redirect and true.

Two redirects are ~same if they have the same redirect summary.

There is a redirect if the code is 301, 302, 303, 307, or 308 and there is a non-nil redirect location.

We use these fields for computing the summary:

- redirect location

- new cookies names (sorted)

func (*EndpointMeasurement) RequestHeaders 

func (em *EndpointMeasurement) RequestHeaders() http.Header

RequestHeaders returns the request headers. If there's no request we just return a set of empty headers.

func (*EndpointMeasurement) ResponseBody 

func (em *EndpointMeasurement) ResponseBody() []byte

ResponseBody returns the response body or empty byte array.

func (*EndpointMeasurement) ResponseBodyTLSH 

func (em *EndpointMeasurement) ResponseBodyTLSH() string

ResponseBodyTLSH returns the TLSH of the response body or empty string.

func (*EndpointMeasurement) ResponseHeaders 

func (em *EndpointMeasurement) ResponseHeaders() http.Header

ResponseHeaders returns the response headers. If there's no response we just return a set of empty headers.

func (*EndpointMeasurement) Scheme 

func (em *EndpointMeasurement) Scheme() string

Scheme returns the URL scheme or an empty string.

func (*EndpointMeasurement) SeemsLegitimateRedirect 

func (em *EndpointMeasurement) SeemsLegitimateRedirect() bool

SeemsLegitimateRedirect works as follows:

1. if this endpoint does not contain a redirect, return false;

2. if this endpoint's location is nil, return false;

3. otherwise returns whether the redirect domain is either equal to the original domain or has the same "public suffix".

func (*EndpointMeasurement) StatusCode 

func (em *EndpointMeasurement) StatusCode() int64

StatusCode returns the response status code. If there's no response we just return zero to the caller.

func (*EndpointMeasurement) Summary 

func (em *EndpointMeasurement) Summary() string

Summary returns a string representing the endpoint's summary. Two endpoints are ~same if they have the same summary.

The summary of an endpoint consists of these fields:

- URL

- Network

- Address

- relevant endpoint options

- original cookies names (sorted)

If the endpoint URL is nil, we return the empty string.

func (*EndpointMeasurement) SupportsAltSvcHTTP3 

func (em *EndpointMeasurement) SupportsAltSvcHTTP3() bool

SupportsAltSvcHTTP3 indicates whether the response in this EndpointMeasurement contains headers claiming the service also supports HTTP3.

func (*EndpointMeasurement) TCPQUICConnectRuntime 

func (em *EndpointMeasurement) TCPQUICConnectRuntime() (out time.Duration)

TCPQUICConnectRuntime returns the TCP/QUIC connect runtime depending on which network is used by this endpoint.

func (*EndpointMeasurement) ToArchival 

func (m *EndpointMeasurement) ToArchival(
	begin time.Time, bodyFlags int64) ArchivalEndpointMeasurement

ToArchival converts a EndpointMeasurement to ArchivalEndpointMeasurement.

func (*EndpointMeasurement) URLAddressList 

func (em *EndpointMeasurement) URLAddressList() ([]*URLAddress, bool)

URLAddressList converts this EndpointMeasurement to an URLAddress list.

func (*EndpointMeasurement) URLAsString 

func (em *EndpointMeasurement) URLAsString() string

URLAsString converts the endpoint URL to string. If such an URL is nil, we return an empty string.

func (*EndpointMeasurement) URLDomain 

func (em *EndpointMeasurement) URLDomain() string

URLDomain returns the domain used by the URL.

func (*EndpointMeasurement) UsingAddressIPv6 

func (em *EndpointMeasurement) UsingAddressIPv6() (usingIPv6 bool)

UsingAddressIPv6 returns true whether this specific endpoint has used an IPv6 destination address, false otherwise.

type EndpointPlan 

type EndpointPlan struct {
	// URLMeasurementID is the ID of the URLMeasurement that created us.
	URLMeasurementID int64

	// Domain is the endpoint domain (e.g., "dns.google").
	Domain string

	// Network is the network (e.g., "tcp" or "quic").
	Network archival.NetworkType

	// Address is the endpoint address (e.g., "8.8.8.8:443").
	Address string

	// URL is the endpoint URL.
	URL *SimpleURL

	// Options contains the options. A nil value implies that we're
	// going to use the default value of each option.
	Options *Options

	// Cookies contains the cookie to use when measuring.
	Cookies []*http.Cookie
}

EndpointPlan is the plan to measure an endpoint.

func (*EndpointPlan) IPAddress 

func (em *EndpointPlan) IPAddress() string

IPAddress returns the IP address used in this EndpointMeasurement.

func (*EndpointPlan) Summary 

func (e *EndpointPlan) Summary() string

Summary returns a string representing the endpoint's plan. Two plans are ~same if they have the same summary.

The summary of an endpoint consists of these fields:

- URL

- Network

- Address

- relevant endpoint options

- cookies names (sorted)

If the endpoint URL is nil, we return the empty string.

type FlatFailedOperation 

type FlatFailedOperation = archival.FlatFailure

FlatFailedOperation is a flat representation of a failed operation.

type IDGenerator 

type IDGenerator struct {
	// contains filtered or unexported fields
}

IDGenerator generates unique IDs for measurements.

func NewIDGenerator 

func NewIDGenerator() *IDGenerator

NewIDGenerator creates a new IDGenerator.

func (*IDGenerator) NextID 

func (g *IDGenerator) NextID() int64

NextID returns the next unique ID.

type Library 

type Library struct {
	// contains filtered or unexported fields
}

Library is a basic library for performing network measurements.

func NewDefaultLibrary 

func NewDefaultLibrary() *Library

NewDefaultLibrary creates a Library that uses netxlite.

func NewLibrary 

func NewLibrary(netx NetxliteLibrary) *Library

NewLibrary creates a new basic measurement library instance using a custom implementation of the nextlite library.

func (*Library) NewDialerWithoutResolver 

func (lib *Library) NewDialerWithoutResolver(saver *archival.Saver) model.Dialer

NewDialerWithoutResolver is a convenience factory for creating a dialer that saves measurements into the saver and that is not attached to any resolver (hence only works when passed IP addresses).

func (*Library) NewHTTPTransportWithConn 

func (lib *Library) NewHTTPTransportWithConn(saver *archival.Saver,
	conn net.Conn, maxBodySnapshotSize int64) model.HTTPTransport

NewHTTPTransportWithConn creates and wraps an HTTPTransport that does not dial and only uses the given conn.

func (*Library) NewHTTPTransportWithQUICSess 

func (lib *Library) NewHTTPTransportWithQUICSess(saver *archival.Saver,
	sess quic.EarlySession, maxBodySnapshotSize int64) model.HTTPTransport

NewHTTPTransportWithQUICSess creates and wraps an HTTPTransport that does not dial and only uses the given QUIC session.

func (*Library) NewHTTPTransportWithTLSConn 

func (lib *Library) NewHTTPTransportWithTLSConn(saver *archival.Saver,
	conn model.TLSConn, maxBodySnapshotSize int64) model.HTTPTransport

NewHTTPTransportWithTLSConn creates and wraps an HTTPTransport that does not dial and only uses the given conn.

func (*Library) NewQUICDialerWithoutResolver 

func (lib *Library) NewQUICDialerWithoutResolver(saver *archival.Saver) model.QUICDialer

NewQUICDialerWithoutResolver creates a new QUICDialer that is not attached to any resolver. This means that every attempt to dial any address containing a domain name will fail. This QUICDialer will save any event into the Saver. Any QUICConn created by it will likewise save any event into the Saver.

func (*Library) NewResolverDoH 

func (lib *Library) NewResolverDoH(saver *archival.Saver,
	clnt model.HTTPClient, network, address string) model.Resolver

NewResolverDoH is a convenience factory for creating a Resolver using DNS-over-HTTPS that saves measurements into the Saver.

Note that we'll not save HTTP related events (such as connecting, or handshaking) when issuing queries because the lifecycle of the HTTP client is much larger than the one of the saver.

The network argument should one of "doh" and "doh3". The former uses DNS-over-HTTPS and the latter DNS-over-HTTP3.

func (*Library) NewResolverSystem 

func (lib *Library) NewResolverSystem(saver *archival.Saver) model.Resolver

NewResolverSystem creates a system resolver and then wraps it using the WrapResolver function.

func (*Library) NewResolverUDP 

func (lib *Library) NewResolverUDP(saver *archival.Saver, address string) model.Resolver

NewResolverUDP is a convenience factory for creating a Resolver using UDP that saves measurements into the Saver.

func (*Library) NewTLSHandshakerStdlib 

func (lib *Library) NewTLSHandshakerStdlib() model.TLSHandshaker

NewTLSHandshakerStdlib creates a new TLS handshaker that uses the Go standard library by invoking the underlying netxlite library.

func (*Library) WrapHTTPClient 

func (lib *Library) WrapHTTPClient(clnt model.HTTPClient) model.HTTPClient

WrapHTTPClient wraps an HTTP client using the underlying netxlite library.

type Measurer 

type Measurer struct {
	// HTTP3ClientForDoH is MANDATORY and is like HTTPClientForDoH except
	// that this client only uses the HTTP3 protocol.
	HTTP3ClientForDoH model.HTTPClient

	// HTTPClientForDoH is the MANDATORY HTTPClient to use for any
	// DNS resolution using the DNS-over-HTTPS transport. This client
	// SHOULD use a transport that allows multiple connections per
	// host, so we can issue queries in parallel.
	HTTPClientForDoH model.HTTPClient

	// IDGenerator is the MANDATORY atomic variable used to generate
	// unique identifiers for measurements.
	IDGenerator *IDGenerator

	// Library is the MANDATORY network-measurement library.
	Library *Library

	// Options contains the options. If nil, we'll use default values.
	Options *Options

	// TLSHandshaker is the MANDATORY TLS handshaker.
	TLSHandshaker model.TLSHandshaker
}

Measurer performs measurements. If you don't use a factory for creating this type, make sure you set all the MANDATORY fields.

func NewMeasurer 

func NewMeasurer(library *Library) *Measurer

NewMeasurer creates a new Measurer instance using the default settings.

func NewMeasurerWithDefaultSettings 

func NewMeasurerWithDefaultSettings() *Measurer

NewMeasurerWithDefaultSettings creates a new measurer using apex/log's singleton as the logger and netxlite as the underlying library.

func NewMeasurerWithOptions 

func NewMeasurerWithOptions(library *Library, options *Options) *Measurer

NewMeasurerWithOptions creates a new Measurer instance with the given options.

func (*Measurer) DNSLookups 

func (mx *Measurer) DNSLookups(ctx context.Context,
	dnsLookups ...*DNSLookupPlan) <-chan *DNSLookupMeasurement

DNSLookups performs DNS lookups in parallel.

This function returns a channel where to read/ measurements from. The channel is closed when done.

func (*Measurer) FlattenOptions 

func (mx *Measurer) FlattenOptions() *Options

FlattenOptions implements AbstractMeasurer.FlattenOptions.

func (*Measurer) MeasureEndpoints 

func (mx *Measurer) MeasureEndpoints(
	ctx context.Context, epnts ...*EndpointPlan) <-chan *EndpointMeasurement

MeasureEndpoints measures some endpoints in parallel.

You can choose the parallelism with the parallelism argument. If this argument is zero, or negative, we use a small default value.

This function returns to the caller a channel where to read measurements from. The channel is closed when done.

func (*Measurer) NewURLMeasurement 

func (mx *Measurer) NewURLMeasurement(input string) (*URLMeasurement, error)

NewURLMeasurement creates a new URLMeasurement from a string URL.

func (*Measurer) NewURLRedirectDeque 

func (mx *Measurer) NewURLRedirectDeque() *URLRedirectDeque

NewURLRedirectDeque creates an URLRedirectDeque.

func (*Measurer) NextID 

func (mx *Measurer) NextID() int64

NextID returns the next measurement ID.

func (*Measurer) Redirects 

func (mx *Measurer) Redirects(
	epnts []*EndpointMeasurement, opts *Options) ([]*URLMeasurement, bool)

Redirects returns all the redirects seen in this URLMeasurement as a list of follow-up URLMeasurement instances. This function will return false if the returned list of follow-up measurements is empty.

type NetxliteLibrary 

type NetxliteLibrary interface {
	// NewDNSOverUDPTransport creates a new DNS-over-UDP DNS transport.
	NewDNSOverUDPTransport(dialer model.Dialer, address string) model.DNSTransport

	// NewDNSOverHTTPSTransport creates a new DNS-over-HTTPS DNS transport. The
	// network argument should be one of "doh" and "doh3".
	NewDNSOverHTTPSTransport(clnt model.HTTPClient, network, address string) model.DNSTransport

	// NewDNSSystemResolver creates a DNS resolver using the system resolver.
	NewDNSSystemResolver(txp model.DNSTransport) model.Resolver

	// NewDNSSystemTransport creates a DNS transport using the system resolver.
	NewDNSSystemTransport() model.DNSTransport

	// NewDialerWithResolver creates a new dialer using the given resolver
	NewDialerWithResolver(reso model.Resolver) model.Dialer

	// NewDialerWithoutResolver creates a new dialer not using any resolver.
	NewDialerWithoutResolver() model.Dialer

	// NewHTTP3Transport creates a new HTTPTransport using HTTP3.
	NewHTTP3Transport(dialer model.QUICDialer, tlsConfig *tls.Config) model.HTTPTransport

	// NewHTTPTransport creates a new HTTPTransport.
	NewHTTPTransport(dialer model.Dialer, tlsDialer model.TLSDialer) model.HTTPTransport

	// NewNullDialer creates a new "null" Dialer.
	NewNullDialer() model.Dialer

	// NewNullTLSDialer creates a new "null" TLSDialer.
	NewNullTLSDialer() model.TLSDialer

	// NewQUICDialerWithoutResolver creates a new QUIC dialer
	// that is not attached to any resolver.
	NewQUICDialerWithoutResolver(ql model.UDPListener) model.QUICDialer

	// NewUDPListener creates a new UDP listener.
	NewUDPListener() model.UDPListener

	// NewUnwrappedParallelResolver creates a new "parallel" resolver. (Note that
	// this resolver needs to be wrapped.)
	NewUnwrappedParallelResolver(txp model.DNSTransport) model.Resolver

	// NewSingleUseDialer creates a new "single use" dialer.
	NewSingleUseDialer(conn net.Conn) model.Dialer

	// NewSingleUseQUICDialer creates a new "single use" QUIC dialer.
	NewSingleUseQUICDialer(sess quic.EarlySession) model.QUICDialer

	// NewSingleUseTLSDialer creates a new "single use" TLS dialer.
	NewSingleUseTLSDialer(conn model.TLSConn) model.TLSDialer

	// NewTLSHandshakerStdlib creates a new TLS handshaker using the stdlib.
	NewTLSHandshakerStdlib() model.TLSHandshaker

	// WrapHTTPClient wraps an HTTP client.
	WrapHTTPClient(clnt model.HTTPClient) model.HTTPClient

	// WrapResolver wraps a resolver.
	WrapResolver(reso model.Resolver) model.Resolver
}

NetxliteLibrary abstracts the netxlite dependency.

type OperationLogger 

type OperationLogger struct {
	// contains filtered or unexported fields
}

OperationLogger logs about an in-progress operation

func NewOperationLogger 

func NewOperationLogger(format string, v ...interface{}) *OperationLogger

NewOperationLogger creates a new logger that logs about an in-progress operation.

func (*OperationLogger) Stop 

func (ol *OperationLogger) Stop(err error)

type Options 

type Options struct {
	// ALPN allows to override the QUIC/TLS ALPN we'll use.
	ALPN []string `json:",omitempty"`

	// DNSLookupTimeout is the maximum time we're willing to wait
	// for any DNS lookup to complete.
	DNSLookupTimeout time.Duration `json:",omitempty"`

	// DNSParallelism is the number of parallel goroutines we will
	// use for measuring HTTP/HTTPS/HTTP3 endpoints.
	DNSParallelism int64 `json:",omitempty"`

	// EndpointParallellism is the number of parallel goroutines we will
	// use for measuring HTTP/HTTPS/HTTP3 endpoints.
	EndpointParallelism int64 `json:",omitempty"`

	// HTTPGetTimeout is the maximum time we're willing to wait
	// for any HTTP GET operation to complete.
	HTTPGetTimeout time.Duration `json:",omitempty"`

	// HTTPHostHeader allows to override the Host header we'll use.
	HTTPHostHeader string `json:",omitempty"`

	// HTTPRequestHeaders controls the HTTP request headers we'll use in
	// the first HTTP request. Subsequent requests following redirects
	// will use the same headers of the first request.
	HTTPRequestHeaders http.Header `json:",omitempty"`

	// DoNotInitiallyForceHTTPAndHTTPS controls whether we're going to
	// initially force using both HTTP and HTTPS for the first URL.
	//
	// TODO(bassosimone): per the latest spec, we should rename this
	// variable to become DoNotFollowHTTPAndHTTPS (or maybe we can even
	// find a better name that is even more explanatory).
	DoNotInitiallyForceHTTPAndHTTPS bool `json:",omitempty"`

	// MaxAddressesPerFamily controls the maximum number of IP addresses
	// per family (i.e., A and AAAA) we'll test.
	MaxAddressesPerFamily int64 `json:",omitempty"`

	// MaxCrawlerDepth is the maximum exploration depth. Every different
	// redirection is another depth level. We will stop exploring when we'll
	// have reached the maximum depth.
	MaxCrawlerDepth int64 `json:",omitempty"`

	// MaxHTTPResponseBodySnapshotSize is the maximum response body
	// snapshot size for cleartext requests (HTTP).
	MaxHTTPResponseBodySnapshotSize int64 `json:",omitempty"`

	// MaxHTTPSResponseBodySnapshotSizeConnectivity is the maximum response body
	// snapshot size for encrypted requests (HTTPS/HTTP3), used when we're just
	// ensuring that we can speak with a given HTTPS endpoint.
	MaxHTTPSResponseBodySnapshotSizeConnectivity int64 `json:",omitempty"`

	// MaxHTTPSResponseBodySnapshotSizeThrottling is the maximum response body
	// snapshot size for encrypted requests (HTTPS/HTTP3), used when we want
	// to measure the download speed by downloading a sizable body chunk.
	MaxHTTPSResponseBodySnapshotSizeThrottling int64 `json:",omitempty"`

	// Parent is the parent Options data structure. By setting this field
	// you can layer new Options on top of existing Options.
	Parent *Options `json:",omitempty"`

	// QUICHandshakeTimeout is the maximum time we're willing to wait
	// for any QUIC handshake to complete.
	QUICHandshakeTimeout time.Duration `json:",omitempty"`

	// TCPConnectTimeout is the maximum time we're willing to wait
	// for any TCP connect attempt to complete.
	TCPconnectTimeout time.Duration `json:",omitempty"`

	// TLSHandshakeTimeout is the maximum time we're willing to wait
	// for any TLS handshake to complete.
	TLSHandshakeTimeout time.Duration `json:",omitempty"`

	// SNI allows to override the QUIC/TLS SNI we'll use.
	SNI string `json:",omitempty"`
}

Options contains options. Every field in this structure is optional and all the methods works as intended when *Options is nil.

func (*Options) Chain 

func (parent *Options) Chain(child *Options) *Options

Chain returns child configured to use the current parent as its parent for the purpose of resolving options. If child is nil, this function just returns the parent options.

func (*Options) Flatten 

func (cur *Options) Flatten() *Options

Flatten generates a new Options that contains all the currently configured options (or default values) inside it.

type SimpleURL 

type SimpleURL struct {
	// Scheme is the URL scheme.
	Scheme string `json:",omitempty"`

	// Host is the host (possily containing a port)
	Host string

	// Path is the URL path.
	Path string `json:",omitempty"`

	// RawQuery contains the unparsed query.
	RawQuery string `json:",omitempty"`
}

SimpleURL is a simpler URL representation.

func NewSimpleURL 

func NewSimpleURL(URL *url.URL) (out *SimpleURL)

NewSimpleURL creates a simple URL from an URL.

func ParseSimpleURL 

func ParseSimpleURL(URL string) (*SimpleURL, error)

ParseSimpleURL parses a simple URL and returns it.

func (*SimpleURL) Clone 

func (su *SimpleURL) Clone() *SimpleURL

Clone creates a copy of this SimpleURL.

func (*SimpleURL) Hostname 

func (su *SimpleURL) Hostname() string

Hostname is like url.URL.Hostname.

func (*SimpleURL) Port 

func (su *SimpleURL) Port() string

Port is like url.URL.Port.

func (*SimpleURL) Query 

func (su *SimpleURL) Query() url.Values

Query is like url.URL.Query.

func (*SimpleURL) String 

func (su *SimpleURL) String() string

String is like url.URL.String.

func (*SimpleURL) ToURL 

func (su *SimpleURL) ToURL() *url.URL

ToURL converts SimpleURL back to stdlib URL.

type URLAddress 

type URLAddress struct {
	// URLMeasurementID is the ID of the parent URLMeasurement.
	URLMeasurementID int64

	// Address is the target IPv4/IPv6 address.
	Address string

	// Domain is the domain of the URL.
	Domain string

	// Flags contains URL flags.
	Flags int64
}

URLAddress is an address associated with a given URL.

func EndpointMeasurementListToURLAddressList 

func EndpointMeasurementListToURLAddressList(
	domain string, eml ...*EndpointMeasurement) ([]*URLAddress, bool)

EndpointMeasurementListToURLAddressList takes in input a list of EndpointMeasurement and produces an URLAddressList.

The domain filter selects only the measurements that are actually valid for the given domain.

Note: do not use this function if you have better ways of creating an URLAddress list, such as, URLMeasurement.URLAddressList.

func MergeURLAddressListStable 

func MergeURLAddressListStable(in ...[]*URLAddress) []*URLAddress

MergeURLAddressListStable takes in input a list of []*URLAddress and it returns a new list where each IP address appears just once and the flags of all its duplicates in input have been merged. The first step of the algorithm is creating a unique list concatenating all the lists provided in input. Then, it will proceed to remove duplicates. In doing so, it will preserve the original order in which unique IP addresses appear in the concatenated list. For example, if IP A appears before IP B in the concatenated list, the same will hold for the return list. Note that the elements of the returned list are clones of the original. So this is a non-destructive and data-race-safe operation.

func NewURLAddressList 

func NewURLAddressList(ID int64, domain string, dns []*DNSLookupMeasurement,
	endpoint []*EndpointMeasurement) ([]*URLAddress, bool)

NewURLAddressList generates a list of URLAddresses based on DNS lookups and Endpoint measurements, all relative to the given ID. We'll _only_ include into the result the IP addresses relative to the given domain. The boolean return value indicates whether we have at least one IP address in the result.

func (*URLAddress) AlreadyTestedHTTP 

func (ua *URLAddress) AlreadyTestedHTTP() bool

AlreadyTestedHTTP returns whether we've already tested this IP address using HTTP.

func (*URLAddress) AlreadyTestedHTTP3 

func (ua *URLAddress) AlreadyTestedHTTP3() bool

AlreadyTestedHTTP3 returns whether we've already tested this IP address using HTTP3.

func (*URLAddress) AlreadyTestedHTTPS 

func (ua *URLAddress) AlreadyTestedHTTPS() bool

AlreadyTestedHTTPS returns whether we've already tested this IP address using HTTPS.

func (*URLAddress) Clone 

func (ua *URLAddress) Clone() *URLAddress

Clone creates a clone of this URLAddressList.

func (*URLAddress) SupportsHTTP3 

func (ua *URLAddress) SupportsHTTP3() bool

SupportsHTTP3 returns whether we think this address supports HTTP3.

type URLAddressListDiff 

type URLAddressListDiff struct {
	// ModifiedFlags contains all the entries in A that have different
	// flags compared to the respective entries in B.
	ModifiedFlags []*URLAddress

	// NewEntries contains all the entries in A that do not appear in B.
	NewEntries []*URLAddress
}

URLAddressListDiff is the diff of two []*URLAddress.

func NewURLAddressListDiff 

func NewURLAddressListDiff(A, B []*URLAddress) *URLAddressListDiff

NewURLAddressListDiff takes in input two []*URLAddress A and B and returns in output all the elements of A that do not appear in B or appear in B with different flags than they appear in A. If the two lists are equal, we just return to the caller an empty diff. The output will always be a valid struct containing lists with pointers to the original matching elements in the A list. The relative order of elements in the lists of the returned diff struct is consistent with the order they appear in the A list.

type URLMeasurement 

type URLMeasurement struct {
	// ID is the unique ID of this URLMeasurement.
	ID int64

	// EndpointIDs contains the ID of the EndpointMeasurement(s) that
	// generated this URLMeasurement through redirects.
	EndpointIDs []int64 `json:",omitempty"`

	// Options contains options. If nil, we'll use default values.
	Options *Options `json:",omitempty"`

	// URL is the underlying URL to measure.
	URL *SimpleURL

	// Cookies contains the list of cookies to use.
	Cookies []*http.Cookie `json:",omitempty"`

	// DNS contains a list of DNS measurements.
	DNS []*DNSLookupMeasurement

	// Endpoint contains a list of endpoint measurements.
	Endpoint []*EndpointMeasurement
}

URLMeasurement is the (possibly interim) result of measuring an URL.

func (*URLMeasurement) AddFromExternalDNSLookup 

func (um *URLMeasurement) AddFromExternalDNSLookup(mx AbstractMeasurer,
	resolverNetwork, resolverAddress string, alpns []string, addrs ...string)

AddFromExternalDNSLookup adds the result of an "external" DNS lookup (i.e., a lookup not performed using measurex) to the URLMeasurement.DNS list. You can use this functionality, for example, for pre-filling the DNS list with selected IP addresses.

Each IP address will be added to a single entry. We will skip strings that are not valid IP addresses representations. The fake entry will use the given resolver network and address (you may want to, e.g., set them to "probe" or "th").

The entry will fake an HTTPSvc lookup because that also allows you to include ALPN, which you may know, into the generated fake lookup entry. If you don't know the ALPN, pass nil as the alpns argument; we will convert it to an empty list for you.

If there are duplicate entries, they will be collapsed by this function.

func (*URLMeasurement) Domain 

func (um *URLMeasurement) Domain() string

Domain is the domain inside the input URL.

func (*URLMeasurement) IsHTTP 

func (um *URLMeasurement) IsHTTP() bool

IsHTTP returns whether this URL is HTTP.

func (*URLMeasurement) IsHTTPS 

func (um *URLMeasurement) IsHTTPS() bool

IsHTTPS returns whether this URL is HTTPS.

func (*URLMeasurement) NewDNSLookupPlans 

func (um *URLMeasurement) NewDNSLookupPlans(
	flags int64, ri ...*DNSResolverInfo) []*DNSLookupPlan

NewDNSLookupPlans is a convenience function for calling the NewDNSLookupPlans free function in the context of this URLMeasurement.

func (*URLMeasurement) NewDNSReverseLookupPlans 

func (um *URLMeasurement) NewDNSReverseLookupPlans(
	addrs []string, ri ...*DNSResolverInfo) []*DNSLookupPlan

NewDNSReverseLookupPlans generates a []*DNSLookupPlan for performing a reverse lookup for the given list of addresses and the given resolvers.

func (*URLMeasurement) NewEndpointPlan 

func (um *URLMeasurement) NewEndpointPlan(flags int64) ([]*EndpointPlan, bool)

NewEndpointPlan is a convenience function that calls um.URLAddressList and passes the resulting list to um.NewEndpointPlanWithAddressList.

func (*URLMeasurement) NewEndpointPlanWithAddressList 

func (um *URLMeasurement) NewEndpointPlanWithAddressList(
	addrs []*URLAddress, flags int64) ([]*EndpointPlan, bool)

NewEndpointPlanWithAddressList creates a new plan for measuring all the endpoints derived from the given address list compatibly with options constraints.

Note that the returned list will include HTTP, HTTPS, and HTTP3 plans related to the original URL regardless of its scheme.

The flags argument allows to specify flags that modify the planning algorithm. The EndpointPlanningExcludeBogons flag is such that we will not include any bogon IP address into the returned plan.

func (*URLMeasurement) ToArchival 

func (m *URLMeasurement) ToArchival(begin time.Time, bodyFlags int64) ArchivalURLMeasurement

ToArchival converts URLMeasurement to ArchivalURLMeasurement.

func (*URLMeasurement) URLAddressList 

func (um *URLMeasurement) URLAddressList() ([]*URLAddress, bool)

URLAddressList calls NewURLAddressList using um.ID, um.DNS, and um.Endpoint.

type URLRedirectDeque 

type URLRedirectDeque struct {
	// contains filtered or unexported fields
}

URLRedirectDeque is the type we use to manage the redirection queue and to follow a reasonable number of redirects.

func (*URLRedirectDeque) Append 

func (r *URLRedirectDeque) Append(um ...*URLMeasurement)

Append appends one or more URLMeasurement to the right of the deque.

func (*URLRedirectDeque) Depth 

func (r *URLRedirectDeque) Depth() int64

Depth returns the number or redirects we followed so far.

func (*URLRedirectDeque) MaxDepth 

func (r *URLRedirectDeque) MaxDepth() int64

MaxDepth returns the maximum depth.

func (*URLRedirectDeque) PopLeft 

func (r *URLRedirectDeque) PopLeft() (*URLMeasurement, error)

PopLeft removes the first element in the redirect deque. Returns true if we returned an element and false when the deque is empty.

func (*URLRedirectDeque) RememberVisitedURLs 

func (r *URLRedirectDeque) RememberVisitedURLs(epnts []*EndpointMeasurement)

RememberVisitedURLs register the URLs we've already visited so that we're not going to visit them again.

func (*URLRedirectDeque) String 

func (r *URLRedirectDeque) String() string

String returns a string representation of the deque.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.