crawler

package

v0.0.0-...-58590c3 Latest Latest Go to latest Published: Mar 9, 2018 License: MIT Imports: 14 Imported by: 0

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

github.com/cavaliercoder/spodermen

Links

Open Source Insights

Documentation ¶

Index ¶

type Crawler
- func New(options ...Option) (*Crawler, error)
- func (c *Crawler) Crawl(q *Queue, workers int) *Stats
- func (c *Crawler) Do(req *Request) (*Response, error)
type Option
type Queue
- func NewQueue(size int, ctx context.Context) *Queue
type Request
- func NewRequest(urlStr string, follow bool) (*Request, error)
- func (c *Request) String() string
type Response
- func (c *Response) String() string
type Stats
- func (c *Stats) AddResponse(resp *Response)
- func (c *Stats) JSON() string

Constants ¶

This section is empty.

Variables ¶

This section is empty.

Functions ¶

This section is empty.

Types ¶

type Crawler ¶

type Crawler struct {
	// contains filtered or unexported fields
}

func New ¶

func New(options ...Option) (*Crawler, error)

func (*Crawler) Crawl ¶

func (c *Crawler) Crawl(q *Queue, workers int) *Stats

func (*Crawler) Do ¶

func (c *Crawler) Do(req *Request) (*Response, error)

type Option ¶

type Option func(*Crawler) error

func FollowHosts ¶

func FollowHosts(hosts ...string) Option

func NoFollow ¶

func NoFollow() Option

func UserAgentString ¶

func UserAgentString(s string) Option

func WithContext ¶

func WithContext(ctx context.Context) Option

type Queue ¶

type Queue struct {
	sync.Mutex
	// contains filtered or unexported fields
}

func NewQueue ¶

func NewQueue(size int, ctx context.Context) *Queue

func (*Queue) Close ¶

func (q *Queue) Close()

func (*Queue) Dequeue ¶

func (q *Queue) Dequeue() (*Request, bool)

func (*Queue) Done ¶

func (q *Queue) Done()

Done decrements the number of queue items in-process.

func (*Queue) Enqueue ¶

func (q *Queue) Enqueue(reqs ...*Request)

type Request ¶

type Request struct {
	URL    *url.URL
	Follow bool
}

A Request represents a single URL in a crawling queue.

func NewRequest ¶

func NewRequest(urlStr string, follow bool) (*Request, error)

func (*Request) String ¶

func (c *Request) String() string

type Response ¶

type Response struct {
	Request       *Request
	Duration      time.Duration
	StatusCode    int
	ContentLength int64
	ContentType   string
	URLs          []string
}

A Response is represents the outcome of a CrawlRequest.

func (*Response) String ¶

func (c *Response) String() string

type Stats ¶

type Stats struct {
	sync.Mutex
	TotalRequests int32
	TotalBytes    int64
	StatusCodes   map[int]int32
	MimeTypes     map[string]int32
}

Stats collects transfer statistics for a Crawler.

func (*Stats) AddResponse ¶

func (c *Stats) AddResponse(resp *Response)

AddResponse increments all statistics according to the given response.

func (*Stats) JSON ¶

func (c *Stats) JSON() string

JSON returns a JSON representation of the crawler statistics.

Source Files ¶

View all Source files

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL