scrape

package
v0.0.0-...-c1f4902 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 18, 2024 License: MIT Imports: 15 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Filter 

type Filter struct {
	// contains filtered or unexported fields
}

func NewFilter 

func NewFilter(conf config.FilterRule) *Filter

func (*Filter) ContentFilter 

func (f *Filter) ContentFilter(content string) string

func (*Filter) Pass 

func (f *Filter) Pass(urlStr string) bool

Pass checks if the url can be scraped

type Scrape 

type Scrape struct {
	// contains filtered or unexported fields
}

func NewScrape 

func NewScrape(headless, rewiseDomain bool) *Scrape

func (*Scrape) AddPipeline 

func (s *Scrape) AddPipeline(fn func(string) string)

func (*Scrape) BatchRun 

func (s *Scrape) BatchRun(ctx context.Context, urlList []string) ([]ScrapeResult, error)

func (*Scrape) Run 

func (s *Scrape) Run(ctx context.Context, rawUrl string) (*ScrapeResult, error)

type ScrapeResult 

type ScrapeResult struct {
	Url     string `json:"url"`
	Title   string `json:"title"`
	Content string `json:"content"`
	Error   string `json:"error,omitempty"`
}

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.