crawler

package
v0.1.4-0...-39de252 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 24, 2021 License: MIT Imports: 12 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Crawler 

type Crawler struct {
	// contains filtered or unexported fields
}

Crawler is used to store crawling configuration

func NewCrawler 

func NewCrawler(instances []*url.URL, numWorkers uint, tokens Tokens, output string) *Crawler

NewCrawler creates new `Crawler` instance

func (*Crawler) Start 

func (c *Crawler) Start()

Start initializes crawling job

type SiteData 

type SiteData struct {
	Link *url.URL
	Body []byte
}

SiteData is used to store web site data

type Tokens 

type Tokens map[string]string

Tokens is used to convert JSON tokens into a map type

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.