scour: github.com/brettdonohoo/scour Index | Files

package scour

import "github.com/brettdonohoo/scour"

Package scour provides convenience functions for web scraping. Inspired by https://github.com/fabpot/goutte

Index

Package Files

client.go crawler.go scour.go selection.go

type Client Uses

type Client struct {
    *http.Client
    // contains filtered or unexported fields
}

func NewClient Uses

func NewClient() *Client

Create a new Client with and embedded http.Client type

func (*Client) Click Uses

func (c *Client) Click(l *Link) (ret *Crawler, err error)

Click is a convenience function simulating a client browser link click Returns a scour crawler object

func (*Client) Request Uses

func (c *Client) Request(method string, url string) (ret *Crawler, err error)

Request is used to initiate a client request and return a crawler object

func (*Client) SetProxy Uses

func (c *Client) SetProxy(proxy_str string) error

SetProxy is a convenience function for setting a proxy url used by the client object

type Crawler Uses

type Crawler struct {
    *goquery.Document
}

The Crawler type wraps goquery's document type - adding new convenience functions

func NewCrawler Uses

func NewCrawler(resp *http.Response) *Crawler

Create the new crawler type from the Response

func (c *Crawler) SelectLink(text string) (l *Link, err error)

Look for specific anchor tag within document

type Link struct {
    // contains filtered or unexported fields
}

Link is used for holding link data

func (*Link) Method Uses

func (l *Link) Method() string

func (*Link) Url Uses

func (l *Link) Url() string

Create the new path for Link TODO: Flesh this out

type Selection Uses

type Selection struct {
    *goquery.Selection
    // contains filtered or unexported fields
}

Selection embeds goquery's Selection object to provide a current URL

Package scour imports 5 packages (graph). Updated 2016-07-14. Refresh now. Tools for package owners. This is an inactive package (no imports and no commits in at least two years).