colly: github.com/gocolly/colly/extensions Index | Files

package extensions

import "github.com/gocolly/colly/extensions"

Package extensions implements various helper addons for Colly

Index

Package Files

extensions.go random_user_agent.go referer.go url_length_filter.go

func RandomUserAgent Uses

func RandomUserAgent(c *colly.Collector)

RandomUserAgent generates a random browser user agent on every request

func Referer Uses

func Referer(c *colly.Collector)

Referer sets valid Referer HTTP header to requests. Warning: this extension works only if you use Request.Visit from callbacks instead of Collector.Visit.

func URLLengthFilter Uses

func URLLengthFilter(c *colly.Collector, URLLengthLimit int)

URLLengthFilter filters out requests with URLs longer than URLLengthLimit

Package extensions imports 3 packages (graph). Updated 2019-01-23. Refresh now. Tools for package owners.