Documentation
¶
Index ¶
- Variables
- type ErrMailDeliveryFailed
- type FeedCrawlRequest
- type FeedCrawlResponse
- type FeedWatcher
- func (fw *FeedWatcher) CrawlFeed() (r *FeedCrawlResponse)
- func (fw *FeedWatcher) Crawling() (r bool)
- func (fw *FeedWatcher) LoadGuidsFromDb(max int) (map[string]bool, error)
- func (fw *FeedWatcher) PollFeed() bool
- func (fw *FeedWatcher) Polling() bool
- func (fw *FeedWatcher) StopPoll()
- func (fw *FeedWatcher) UpdateFeed(resp *FeedCrawlResponse) error
Constants ¶
This section is empty.
Variables ¶
After allows for stubbing out in test
var ErrAlreadyCrawlingFeed = errors.New("already crawling feed")
ErrAlreadyCrawlingFeed is when you are already Crawling a feed
var ErrCrawlerNotAvailable = errors.New("no crawler available")
ErrCrawlerNotAvailable is when the crawler channel would block
Functions ¶
This section is empty.
Types ¶
type ErrMailDeliveryFailed ¶
type ErrMailDeliveryFailed struct {
// contains filtered or unexported fields
}
ErrMailDeliveryFailed captures the specific reason for a mail failure
func (*ErrMailDeliveryFailed) Error ¶
func (e *ErrMailDeliveryFailed) Error() string
type FeedCrawlRequest ¶
type FeedCrawlRequest struct {
URI string
ResponseChan chan *FeedCrawlResponse
}
FeedCrawlRequest is how to request a URL to be crawled by a crawler instance.
type FeedCrawlResponse ¶
type FeedCrawlResponse struct {
URI string
Body []byte
Feed *gofeed.Feed
Items []*gofeed.Item
HTTPResponseStatus string
HTTPResponseStatusCode int
Error error
}
FeedCrawlResponse is we get back from the crawler.
Body, Feed, Items and HTTPResponseStatus are only guaranteed to be non zero value if Error is non nil.
type FeedWatcher ¶
type FeedWatcher struct {
FeedInfo db.FeedInfo
GUIDCache map[string]bool
LastCrawlResponse *FeedCrawlResponse
After func(d time.Duration) <-chan time.Time // Allow for mocking out in test.
Logger logrus.FieldLogger
// contains filtered or unexported fields
}
FeedWatcher controlls the crawling of a Feed. It keeps state about the GUIDs it's seen, sends crawl requests and deals with responses.
func NewFeedWatcher ¶
func NewFeedWatcher( feedInfo db.FeedInfo, crawlChan chan *FeedCrawlRequest, mailChan chan *mail.Request, dbh db.Service, GUIDCache []string, minSleep int64, maxSleep int64, ) *FeedWatcher
NewFeedWatcher returns a new FeedWatcher instance.
func (*FeedWatcher) CrawlFeed ¶
func (fw *FeedWatcher) CrawlFeed() (r *FeedCrawlResponse)
CrawlFeed will send a request to crawl a feed over it's crawl channel
func (*FeedWatcher) Crawling ¶
func (fw *FeedWatcher) Crawling() (r bool)
Crawling returns true if the FeedWatcher is crawling a feed (actually waiting for a response from the crawler).
func (*FeedWatcher) LoadGuidsFromDb ¶
func (fw *FeedWatcher) LoadGuidsFromDb(max int) (map[string]bool, error)
LoadGuidsFromDb populates the internal cache of GUIDs by getting the most recent GUIDs it knows about. To retrieve all GUIDs set max to -1.
func (*FeedWatcher) PollFeed ¶
func (fw *FeedWatcher) PollFeed() bool
PollFeed is designed to be called as a Goroutine. Use UpdateFeed to just update once.
func (*FeedWatcher) Polling ¶
func (fw *FeedWatcher) Polling() bool
Polling returns true if the FeedWatcher is Polling a feed.
func (*FeedWatcher) StopPoll ¶
func (fw *FeedWatcher) StopPoll()
StopPoll will cause the PollFeed loop to exit.
func (*FeedWatcher) UpdateFeed ¶
func (fw *FeedWatcher) UpdateFeed(resp *FeedCrawlResponse) error
UpdateFeed will crawl the feed, check for new items, mail them out and update the database with the new information
Source Files