Documentation ¶
Overview ¶
Package app interface for graphical user interface. The basic business execution order is: New () -> [SetLog (io.Writer) ->] Init () -> SpiderPrepare () -> Run ()
Index ¶
- Variables
- type App
- type Logic
- func (logic *Logic) CountNodes() int
- func (logic *Logic) GetAppConf(k ...string) interface{}
- func (logic *Logic) GetMode() int
- func (logic *Logic) GetOutputLib() []string
- func (logic *Logic) GetSpiderByName(name string) *spider.Spider
- func (logic *Logic) GetSpiderLib() []*spider.Spider
- func (logic *Logic) GetSpiderQueue() crawler.SpiderQueue
- func (logic *Logic) GetTaskJar() *distribute.TaskJar
- func (logic *Logic) Init(mode int, port int, master string, w ...io.Writer) App
- func (logic *Logic) IsPause() bool
- func (logic *Logic) IsRunning() bool
- func (logic *Logic) IsStopped() bool
- func (logic *Logic) LogGoOn() App
- func (logic *Logic) LogRest() App
- func (logic *Logic) PauseRecover()
- func (logic *Logic) ReInit(mode int, port int, master string, w ...io.Writer) App
- func (logic *Logic) Run()
- func (logic *Logic) SetAppConf(k string, v interface{}) App
- func (logic *Logic) SetLog(w io.Writer) App
- func (logic *Logic) SpiderPrepare(original []*spider.Spider) App
- func (logic *Logic) Status() int
- func (logic *Logic) Stop()
Constants ¶
This section is empty.
Variables ¶
var LogicApp = New()
LogicApp is variable ... The only global core interface instance
Functions ¶
This section is empty.
Types ¶
type App ¶
type App interface { SetLog(io.Writer) App // set the global log real-time display terminal LogGoOn() App // Continue log printing LogRest() App // pause log printing Init(mode int, port int, master string, w ...io.Writer) App // Must be initialized before using App, except for SetLog () ReInit(mode int, port int, master string, w ...io.Writer) App // Switch the run mode and reset the log print target GetAppConf(k ...string) interface{} // Get global parameters SetAppConf(k string, v interface{}) App // Set the global parameters (client mode does not call the method) SpiderPrepare(original []*spider.Spider) App // must be called after the global run parameter is run () (not called in client mode) Run() // Blocked operation until task completes (must be invoked after all configuration items should be configured) Stop() // Offline mode to terminate the task halfway (external blocking operation until the current task is terminated) IsRunning() bool // Check if the task is running IsPause() bool // Check if the task is in a paused state IsStopped() bool // Check if the task has been terminated PauseRecover() // Offline mode pause \ resume task Status() int // returns the current state GetSpiderLib() []*spider.Spider // Get all spider species GetSpiderByName(string) *spider.Spider // get a spider by name GetSpiderQueue() crawler.SpiderQueue // Get the spider queue interface instance GetOutputLib() []string // Get all output mode GetTaskJar() *distribute.TaskJar // return to the task library distribute.Distributer // Implement a distributed interface }
App inteface ...
type Logic ¶
type Logic struct { *cache.AppConf // global configuration *spider.SpiderSpecies // all spider species crawler.SpiderQueue // the current task of the spider queue *distribute.TaskJar // The repository that delivers the task between the server and the client crawler.Pool // crawling the pool teleport.Teleport // socket long connection duplex communication interface, json data transmission sync.RWMutex // contains filtered or unexported fields }
Logic struct ...
func (*Logic) CountNodes ¶
CountNodes is a function ... The number of nodes returned in the server client mode
func (*Logic) GetAppConf ¶
GetAppConf is a function ... Get global parameters
func (*Logic) GetOutputLib ¶
GetOutputLib is a function ... Get all output mode
func (*Logic) GetSpiderByName ¶
GetSpiderByName is a function ... get a spider by name
func (*Logic) GetSpiderLib ¶
GetSpiderLib is a function ... Get all spider species
func (*Logic) GetSpiderQueue ¶
func (logic *Logic) GetSpiderQueue() crawler.SpiderQueue
GetSpiderQueue is a function ... Get the spider queue interface instance
func (*Logic) GetTaskJar ¶
func (logic *Logic) GetTaskJar() *distribute.TaskJar
GetTaskJar is a function ... return to the task library
func (*Logic) Init ¶
Init is a function ... use Init before you must use Init to initialize (except SetLog ())
func (*Logic) PauseRecover ¶
func (logic *Logic) PauseRecover()
PauseRecover is a function ... Offline mode pause \ resume task
func (*Logic) SetAppConf ¶
SetAppConf is a function ... set global parameters
func (*Logic) SpiderPrepare ¶
SpiderPrepare must be executed immediately after setting the global run parameters, Run () original is the original spider of the spider package that has not been assigned an assignment A spider that has been explicitly assigned will no longer reassign KeyIn client mode does not call this method
Directories ¶
Path | Synopsis |
---|---|
aid
|
|
surfer
Package surfer s a Go language prepared by the high concurrent web downloader, support GET/POST/HEAD Method and method http/https Protocol, while supporting the fixed UserAgent automatically save the cookie with a random large number of UserAgent disabled cookie two modes, a high degree of simulation of the browser behavior, enabling analog login and other functions.
|
Package surfer s a Go language prepared by the high concurrent web downloader, support GET/POST/HEAD Method and method http/https Protocol, while supporting the fixed UserAgent automatically save the cookie with a random large number of UserAgent disabled cookie two modes, a high degree of simulation of the browser behavior, enabling analog login and other functions. |
surfer/agent
Package agent generates user agents strings for well known browsers and for custom browsers.
|
Package agent generates user agents strings for well known browsers and for custom browsers. |
data collection
|
data collection |
collector
result collection and output
|
result collection and output |