scraper

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 8, 2022 License: MIT Imports: 4 Imported by: 0

Documentation

Overview

scraper holds all the scraping logic. All subpackage must implement “scraper.ScrapeIterator“

Index

Constants

This section is empty.

Variables

View Source 
var Finish = errors.New("end of scrape queue")

Functions

This section is empty.

Types

type Doer 

type Doer interface {
	Do(*http.Request) (*http.Response, error)
}

type ScrapData 

type ScrapData struct {
	Index   int
	Content *goquery.Selection
}

type ScrapeIterator 

type ScrapeIterator interface {
	// Returns the next data. If there's no more data to fetch. “scraper.Finish“ error will be returned.
	Next(ctx context.Context, url string) (data *ScrapData, err error)
}

ScrapeIterator is an interface to fetch data queue in the implementor. Implementor must return Finish error if there's no more data to fetch.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.