Documentation
¶
Overview ¶
Mobal is a pure Go language designed to support the distributed high concurrent, heavyweight crawler software, located in the Internet data collection, for a certain Go or JS programming based on the people to provide a need to focus on custom rules The powerful reptile tool. (Mysql / mongodb / kafka / csv / excel, etc.). It has three kinds of operating modes, such as Web, GUI and command line. The rules are simple and flexible. ), There are a large number of Demo sharing; In addition it also supports horizontal and vertical two crawl mode, support for simulated login and task pause, cancel and a series of advanced features. (official QQ group: Go large data 42731170).
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package app interface for graphical user interface.
|
Package app interface for graphical user interface. |
|
downloader/surfer
Package surfer s a Go language prepared by the high concurrent web downloader, support GET/POST/HEAD Method and method http/https Protocol, while supporting the fixed UserAgent automatically save the cookie with a random large number of UserAgent disabled cookie two modes, a high degree of simulation of the browser behavior, enabling analog login and other functions.
|
Package surfer s a Go language prepared by the high concurrent web downloader, support GET/POST/HEAD Method and method http/https Protocol, while supporting the fixed UserAgent automatically save the cookie with a random large number of UserAgent disabled cookie two modes, a high degree of simulation of the browser behavior, enabling analog login and other functions. |
|
downloader/surfer/agent
Package agent generates user agents strings for well known browsers and for custom browsers.
|
Package agent generates user agents strings for well known browsers and for custom browsers. |
|
pipeline
data collection
|
data collection |
|
pipeline/collector
result collection and output
|
result collection and output |
|
[spider frame (golang)] Mobal (Phantom Spider) is a pure Go language prepared by the high-level, distributed, heavyweight reptile software, support stand-alone, server, client three operating modes, with Web, GUI (Mysql / mongodb / csv / excel, etc.), a large number of Demo sharing; at the same time she also supports horizontal and vertical two crawling mode, support for simulated login (mysql / mongodb / csv / excel, etc.), a large number of Demo shared; And task suspension, cancellation and a series of advanced features; (official QQ group: Go large data 42731170, welcome to join our discussion).
|
[spider frame (golang)] Mobal (Phantom Spider) is a pure Go language prepared by the high-level, distributed, heavyweight reptile software, support stand-alone, server, client three operating modes, with Web, GUI (Mysql / mongodb / csv / excel, etc.), a large number of Demo sharing; at the same time she also supports horizontal and vertical two crawling mode, support for simulated login (mysql / mongodb / csv / excel, etc.), a large number of Demo shared; And task suspension, cancellation and a series of advanced features; (official QQ group: Go large data 42731170, welcome to join our discussion). |
|
common
|
|
|
config
Package config is used to parse config Usage: import( "github.com/astaxie/beego/config" ) cnf, err := config.NewConfig("ini", "config.conf") cnf APIS: cnf.Set(key, val string) error cnf.String(key string) string cnf.Strings(key string) []string cnf.Int(key string) (int, error) cnf.Int64(key string) (int64, error) cnf.Bool(key string) (bool, error) cnf.Float(key string) (float64, error) cnf.DefaultString(key string, defaultVal string) string cnf.DefaultStrings(key string, defaultVal []string) []string cnf.DefaultInt(key string, defaultVal int) int cnf.DefaultInt64(key string, defaultVal int64) int64 cnf.DefaultBool(key string, defaultVal bool) bool cnf.DefaultFloat(key string, defaultVal float64) float64 cnf.DIY(key string) (interface{}, error) cnf.GetSection(section string) (map[string]string, error) cnf.SaveConfigFile(filename string) error more docs http://beego.me/docs/module/config.md
|
Package config is used to parse config Usage: import( "github.com/astaxie/beego/config" ) cnf, err := config.NewConfig("ini", "config.conf") cnf APIS: cnf.Set(key, val string) error cnf.String(key string) string cnf.Strings(key string) []string cnf.Int(key string) (int, error) cnf.Int64(key string) (int64, error) cnf.Bool(key string) (bool, error) cnf.Float(key string) (float64, error) cnf.DefaultString(key string, defaultVal string) string cnf.DefaultStrings(key string, defaultVal []string) []string cnf.DefaultInt(key string, defaultVal int) int cnf.DefaultInt64(key string, defaultVal int64) int64 cnf.DefaultBool(key string, defaultVal bool) bool cnf.DefaultFloat(key string, defaultVal float64) float64 cnf.DIY(key string) (interface{}, error) cnf.GetSection(section string) (map[string]string, error) cnf.SaveConfigFile(filename string) error more docs http://beego.me/docs/module/config.md |
|
goquery
Package goquery implements features similar to jQuery, including the chainable syntax, to manipulate and query an HTML document.
|
Package goquery implements features similar to jQuery, including the chainable syntax, to manipulate and query an HTML document. |
|
mahonia
This package is a character-set conversion library for Go.
|
This package is a character-set conversion library for Go. |
|
pool
Package pool is a package ...
|
Package pool is a package ... |
|
websocket
Package websocket implements a client and server for the WebSocket protocol as specified in RFC 6455.
|
Package websocket implements a client and server for the WebSocket protocol as specified in RFC 6455. |
|
runtime
|
|
|
Package web ==> [spider frame (golang)] Mobal (Phantom Spider) is a pure Go language prepared by the high-level, distributed, heavyweight reptile software, support stand-alone, server, client three operating modes, with Web, GUI (Mysql / mongodb / csv / excel, etc.), a large number of Demo sharing; at the same time she also supports horizontal and vertical two crawling mode, support for simulated login (mysql / mongodb / csv / excel, etc.), a large number of Demo shared; And task suspension, cancellation and a series of advanced features; (official QQ group: Go large data 42731170, welcome to join our discussion).
|
Package web ==> [spider frame (golang)] Mobal (Phantom Spider) is a pure Go language prepared by the high-level, distributed, heavyweight reptile software, support stand-alone, server, client three operating modes, with Web, GUI (Mysql / mongodb / csv / excel, etc.), a large number of Demo sharing; at the same time she also supports horizontal and vertical two crawling mode, support for simulated login (mysql / mongodb / csv / excel, etc.), a large number of Demo shared; And task suspension, cancellation and a series of advanced features; (official QQ group: Go large data 42731170, welcome to join our discussion). |
Click to show internal directories.
Click to hide internal directories.