crawler

package
v0.0.0-...-301f747 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 21, 2015 License: MIT Imports: 13 Imported by: 0

Documentation

Overview

Package crawler is a crawler that runs periodically for each user and updates the database if it finds new results. It can also warn the user by email when there is new results.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func StartWebservice

func StartWebservice(scheduler *Scheduler, userStore user.Store, port string)

StartWebservice starts the crawler webservice.

Types

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client implements a rpc client for the crawler.

func NewClient

func NewClient(url string) *Client

NewClient creates a new client

func (*Client) Refresh

func (c *Client) Refresh(userID string) error

Refresh calls the crawler webservice Queue method for the specified userID

type ResultGetter

type ResultGetter interface {
	// Run fetches results for a user.
	Run(user *User) []RunResult
}

ResultGetter is an interface for something that fetches results.

type ResultGetterClient

type ResultGetterClient interface {
	Do(*http.Request) (*http.Response, error)
}

ResultGetterClient interface for sending a request to get results.

type RunResult

type RunResult struct {
	ClassIndex int
	Class      *api.Class
	Err        error
}

RunResult contains the result of a ResultGetter run for a class.

type Scheduler

type Scheduler struct {
	// contains filtered or unexported fields
}

Scheduler handles scheduling crawler runs for every user.

func NewScheduler

func NewScheduler(config *SchedulerConfig) *Scheduler

NewScheduler creates a new scuduler object.

func (*Scheduler) Queue

func (s *Scheduler) Queue(user *api.User)

Queue tells the scheduler do a run for a user

func (*Scheduler) QueueAsync

func (s *Scheduler) QueueAsync(user *api.User, doneCh chan bool)

QueueAsync tells the scheduler do a run for a user async

func (*Scheduler) Start

func (s *Scheduler) Start()

Start starts the scheduler

func (*Scheduler) Stop

func (s *Scheduler) Stop()

Stop stops the scheduler.

type SchedulerConfig

type SchedulerConfig struct {
	ResultGetters      []ResultGetter
	UserStore          user.Store
	CrawlerConfigStore crawlerconfig.Store
	UserResultsStore   results.Store
	Sender             tools.Sender
}

SchedulerConfig initializes the scheduler.

type User

type User struct {
	ID      string
	Classes []api.Class
	Nip     string
	Code    string
	Name    string
	Email   string
	DoneCh  chan bool
}

User contains info about the user of a ResultGetter run.

type Webservice

type Webservice struct {
	// contains filtered or unexported fields
}

Webservice is the exported type for rpc.

func (*Webservice) Queue

func (ws *Webservice) Queue(userID string, ret *int) error

Queue starts the crawler for the specified userID. This method is available to clients of the webservice throught the rpc package.

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL