crawler

package
v1.2.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 28, 2021 License: MIT Imports: 11 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Collector

func Collector(ctx context.Context, url string, projectPath string, cookieJar *cookiejar.Jar, proxyString string, userAgent string) error

Collector searches for css, js, and images within a given link TODO improve for better performance

func Crawl

func Crawl(ctx context.Context, site string, projectPath string, cookieJar *cookiejar.Jar, proxyString string, userAgent string) error

Crawl asks the necessary crawlers for collecting links for building the web page

func Extractor

func Extractor(link string, projectPath string)

Extractor visits a link determines if its a page or sublink downloads the contents to a correct directory in project folder TODO add functionality for determining if page or sublink

func HTMLExtractor

func HTMLExtractor(link string, projectPath string)

HTMLExtractor ...

Types

This section is empty.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL