README
¶
Goccer
Go concurrent crawler(s) library
Usage
package main
import (
"log"
"github.com/oglinuk/goccer"
)
func main() {
wp := goccer.NewWorkerpool()
collected := wp.Queue([]string{"https://fourohfournotfound.com"})
for _, c := range collected {
log.Println(c)
}
}
Examples
See examples directory.
Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Workerpool ¶
type Workerpool struct {
// contains filtered or unexported fields
}
Workerpool is a wrapper for the jobs chan to send seeds, the wg sync.WaitGroup to wait for goroutines, the w(riter) which is an in-memory implementation (only at the moment, see TODO above crawler), the (c)rawler, and a mu(tex) to prevent concurrent map writes
func (*Workerpool) Queue ¶ added in v0.2.0
func (wp *Workerpool) Queue(ps []string) []string
Queue (p)ath(s) to be crawled
Source Files
Directories
Click to show internal directories.
Click to hide internal directories.