wayback

package

v1.1.3 Latest Latest Go to latest Published: Jun 4, 2023 License: MIT Imports: 4 Imported by: 0

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

View Source

const CRAWL_STORAGE = "https://web.archive.org/web"

View Source

const INDEX_SERVER = "https://web.archive.org/cdx/search/cdx"

This section is empty.

This section is empty.

type Wayback struct {
	MaxTimeout int // Request timeout
	MaxRetries int // Max number of request retries if timeouted
}

func New(timeout, retries int) (*Wayback, error)

func (wb *Wayback) FetchPages(config common.RequestConfig, results chan []*common.CdxResponse, errors chan error)

FetchPages ... Concurrent way to GetPages. Makes request to WebArchive CDX API and return observations in a channel.

func (wb *Wayback) GetFile(page *common.CdxResponse) ([]byte, error)

Download file from WebArchive using a link from CDX response

func (wb *Wayback) GetNumPages(url string) (int, error)

Return the number of pages located in WebArchive for given url

func (wb *Wayback) GetPages(config common.RequestConfig) ([]*common.CdxResponse, error)

GetPages ... Makes request to WebArchive CDX API to gather all url observations

func (Wayback) Name() string

func (wb *Wayback) ParseResponse(resp []byte) ([]*common.CdxResponse, error)