smeago

package

v0.0.0-...-da67eab Latest Latest Go to latest Published: Sep 30, 2017 License: MIT Imports: 10 Imported by: 0

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

github.com/dbalduini/smeago

Links

Open Source Insights

Documentation ¶

Index ¶

type Crawler
- func NewCrawler(d string) *Crawler
- func (c *Crawler) Crawl(j Job)
type CrawlerSupervisor
- func NewCrawlerSupervisor(c *Crawler) *CrawlerSupervisor
type Job
- func NewJob(id int, path string) *Job
type Result
- func ReadString(rd io.Reader) (*Result, error)
- func ReadStringSize(rd io.Reader, n int) (*Result, error)
type Sitemap
- func (s *Sitemap) Write(w io.Writer) error
- func (s *Sitemap) WriteToFile() error

Constants ¶

This section is empty.

Variables ¶

This section is empty.

Functions ¶

This section is empty.

Types ¶

type Crawler ¶

type Crawler struct {
	Domain  string
	Results chan Job
	Retries chan Job
}

func NewCrawler ¶

func NewCrawler(d string) *Crawler

NewCrawler creates a crawler for the given domain

func (*Crawler) Crawl ¶

func (c *Crawler) Crawl(j Job)

Crawl the job path and retries in case of failures

type CrawlerSupervisor ¶

type CrawlerSupervisor struct {
	// contains filtered or unexported fields
}

CrawlerSupervisor control the execution of the crawler

func NewCrawlerSupervisor ¶

func NewCrawlerSupervisor(c *Crawler) *CrawlerSupervisor

NewCrawlerSupervisor returns a new CrawlerSupervisor

func (*CrawlerSupervisor) AddJobToBuffer ¶

func (cs *CrawlerSupervisor) AddJobToBuffer(path string)

AddJobToBuffer creates a new job for the given path and adds it to the buffer

func (*CrawlerSupervisor) BuffSize ¶

func (cs *CrawlerSupervisor) BuffSize() int

BuffSize returns the len of the buffer

func (*CrawlerSupervisor) CompleteJob ¶

func (cs *CrawlerSupervisor) CompleteJob(j Job)

CompleteJob Removes the job from pending list

func (*CrawlerSupervisor) CrawlJobs ¶

func (cs *CrawlerSupervisor) CrawlJobs()

CrawlJobs crawls all jobs in the buffer concurrently

func (*CrawlerSupervisor) GetVisitedLinks ¶

func (cs *CrawlerSupervisor) GetVisitedLinks() []string

GetVisitedLinks returns a set of all visited links

func (*CrawlerSupervisor) HasPending ¶

func (cs *CrawlerSupervisor) HasPending() bool

HasPending returns true if there are jobs in the pending list

func (*CrawlerSupervisor) Start ¶

func (cs *CrawlerSupervisor) Start(done chan bool)

Start crawls buffered jobs until pending list is empty

type Job ¶

type Job struct {
	ID         int
	Path       string
	Links      []string
	Completed  bool
	RetryCount int
}

func NewJob ¶

func NewJob(id int, path string) *Job

NewJob creates a new Job

type Result ¶

type Result struct {
	Links []string
}

func ReadString ¶

func ReadString(rd io.Reader) (*Result, error)

func ReadStringSize ¶

func ReadStringSize(rd io.Reader, n int) (*Result, error)

type Sitemap ¶

type Sitemap struct {
	Filename string
	Links    []string
	Path     string
}

func (*Sitemap) Write ¶

func (s *Sitemap) Write(w io.Writer) error

func (*Sitemap) WriteToFile ¶

func (s *Sitemap) WriteToFile() error

WriteToFile writes the sitemap into a file

Source Files ¶

View all Source files

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL