http

package

v0.0.0-...-8fe63c4 Latest Latest Go to latest Published: Jun 10, 2019 License: GPL-3.0 Imports: 8 Imported by: 0

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

github.com/goncalopereira/gocrawler

Links

Open Source Insights

Documentation ¶

Index ¶

func GetURL(r data.Request, parser BodyParser) (data.Response, error)
func Server(port string, filterRequests chan<- data.Request) error
func WebClient(parser BodyParser, webRequests <-chan data.Request, ...) error
type BasicBodyParser
- func (r BasicBodyParser) Parse(body io.Reader, request data.Request) data.Response
type BodyParser
type NewRequestHandler
- func (nrh NewRequestHandler) Post(w http.ResponseWriter, r *http.Request)

Constants ¶

This section is empty.

Variables ¶

This section is empty.

Functions ¶

func GetURL ¶

func GetURL(r data.Request, parser BodyParser) (data.Response, error)

GetURL sends all links found in page back http.Client misbehaves and might create too many goroutines

func Server ¶

func Server(port string, filterRequests chan<- data.Request) error

Server starts new HTTP server that receives external requests sends request to be filtered on Crawler

func WebClient ¶

func WebClient(parser BodyParser, webRequests <-chan data.Request, responses chan<- data.Response, shutdown <-chan bool) error

WebClient that runs through external requests to crawl

Types ¶

type BasicBodyParser ¶

type BasicBodyParser struct {
	MaxLinks int
}

BasicBodyParser basic implementation for a href

func (BasicBodyParser) Parse ¶

func (r BasicBodyParser) Parse(body io.Reader, request data.Request) data.Response

Parse handling links from body reader out to Response moving ioReader to chan would create issues with pointer links are A tag, Link Tag and Nav tag but we check A no JS parsing

type BodyParser ¶

type BodyParser interface {
	Parse(body io.Reader, request data.Request) data.Response
}

BodyParser interface for new ways to parse body

type NewRequestHandler ¶

type NewRequestHandler struct {
	FilterRequests chan<- data.Request
}

NewRequestHandler Parses and sends requests to Crawler

func (NewRequestHandler) Post ¶

func (nrh NewRequestHandler) Post(w http.ResponseWriter, r *http.Request)

Post Parses Post and sends to Crawler queue

Source Files ¶

View all Source files

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL