README
¶
nsq app
This library is simply a command line application that can be constructed using your own spiders. Example spider can be found in crawl repo.
Skeleton
To create your own nsq crawler you can simply copy skeleton/
directory and add your own spiders.
After copying it should be enough to replace all occurences of github.com/crackcomm/crawl/skeleton to new path of the application.
If you want CircleCI to deploy docker image for you change crawl/skeleton to your image name in circle.yaml.
Command-line Usage
$ skeleton --help
NAME:
crawler - nsq crawl consumer
USAGE:
crawler [global options] command [command options] [arguments...]
VERSION:
0.0.1
COMMANDS:
help, h Shows a list of commands or help for one command
GLOBAL OPTIONS:
--topic "crawl_requests" crawl requests nsq topic (required) [$TOPIC]
--channel "default" crawl requests nsq channel (required) [$CHANNEL]
--nsq-addr [--nsq-addr option --nsq-addr option] [$NSQ_ADDR]
--nsqlookup-addr [--nsqlookup-addr option --nsqlookup-addr option] [$NSQLOOKUP_ADDR]
--concurrency "100" [$CONCURRENCY]
--help, -h show help
--version, -v print the version
Documentation
¶
Overview ¶
Package consumer implements command line crawl consumer from nsq.
Index ¶
- Variables
- func New(opts ...Option) *cli.App
- type App
- type Option
- func WithBefore(fnc func(*App) error) Option
- func WithCrawler(crawler crawl.Crawler) Option
- func WithCrawlerConstructor(fnc func(*App) crawl.Crawler) Option
- func WithHandler(name string, h crawl.Handler) Option
- func WithMiddlewares(middlewares ...crawl.Middleware) Option
- func WithSpiderConstructor(fnc func(*App) Spider) Option
- func WithSpiders(spiders ...Spider) Option
- type Spider
Constants ¶
This section is empty.
Variables ¶
var Flags = []cli.Flag{ clinsq.AddrFlag, clinsq.LookupAddrFlag, clinsq.TopicFlag, clinsq.ChannelFlag, &cli.IntFlag{ Name: "concurrency", Value: 100, EnvVars: []string{"CONCURRENCY"}, }, &cli.IntFlag{ Name: "timeout", Usage: "default timeout in seconds", Value: 30, EnvVars: []string{"TIMEOUT"}, }, }
Flags - Consumer app flags.
Functions ¶
Types ¶
type App ¶
type App struct {
// Ctx - Cli context, set on action.
Ctx *cli.Context
// Queue - NSQ queue. Constructed on first Action() call.
*nsqcrawl.Queue
// contains filtered or unexported fields
}
App - Consumer command line application structure.
type Option ¶
type Option func(*App)
Option - Consumer app option setter.
func WithBefore ¶
WithBefore - Overwrites flag checking before action.
func WithCrawler ¶
WithCrawler - Registers spider on a crawler.
func WithCrawlerConstructor ¶
WithCrawlerConstructor - Constructs a crawler on action.
func WithHandler ¶
WithHandler - Registers crawler handler. It has to be set after WithCrawler (if any).
func WithMiddlewares ¶
func WithMiddlewares(middlewares ...crawl.Middleware) Option
WithMiddlewares - Registers middlewares on a crawler. It has to be set after WithCrawler (if any).
func WithSpiderConstructor ¶
WithSpiderConstructor - Constructs a spider on action.
func WithSpiders ¶
WithSpiders - Registers spider on a crawler. It has to be set after WithCrawler (if any).
Source Files