spiders

package
v0.0.0-...-f2a16fb Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 5, 2016 License: MIT Imports: 2 Imported by: 32

Documentation

Index

Constants

View Source
const (
	BASE_PARSE_NAME     = "base"
	SPIDERS_STATUS_INIT = iota
	SPIDERS_STATUS_RUNNING
	SPIDERS_STATUS_STOP
	SPIDERS_BASIC_COOKIE
)

Variables

This section is empty.

Functions

This section is empty.

Types

type Spider

type Spider struct {
	BeforeMethod  func()
	InitStartUrls func()
	AfterMethod   func()
	Name          string
	StartUrls     []string
	ExtData       map[string]interface{}
	ParseMap      map[string]func(response *http.Response) ([]*http.Request, error) // defined you own parse function to scrapy response,make sure it return []response
}

what a spider do * make start request * define basic parse func

func (*Spider) MakeStartRequests

func (this *Spider) MakeStartRequests() []*http.Request

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL