Documentation ¶
Index ¶
- func GetSocks5Proxy(ctx context.Context, url string) (ret string, err error)
- func RecordData(data interface{}) (err error)
- func RunWithCrawler(crawler Crawler, opts ...chromedp.ExecAllocatorOption)
- func SetStorageTarget(file *os.File)
- func StartStorageServer(ctx context.Context)
- func ToCrawl(action CrawlAction)
- type CrawlAction
- type Crawler
- type ProxyInfo
- type ProxyResponse
- type Storage
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func GetSocks5Proxy ¶
获取SOCKS5代理地址
func RecordData ¶
func RecordData(data interface{}) (err error)
写入已经json序列化好的字符数组,字符串或者是可以进行json序列化的对象
func RunWithCrawler ¶
func RunWithCrawler(crawler Crawler, opts ...chromedp.ExecAllocatorOption)
func StartStorageServer ¶
需要使用go关键字,单独起一个协程运行,只有开启了该服务,才能存储数据
func ToCrawl ¶
func ToCrawl(action CrawlAction)
Types ¶
type CrawlAction ¶
type CrawlAction = func(proxyReqUrl string, isHeadless bool, customMap map[string]string, outputFile *os.File)
func GetAction ¶
func GetAction(crawler Crawler) CrawlAction
type ProxyResponse ¶
type Storage ¶
type Storage struct {
// contains filtered or unexported fields
}
func GetStorageInstance ¶
func GetStorageInstance() *Storage
Click to show internal directories.
Click to hide internal directories.