crawler

package module

v0.0.0-...-9a3108e Latest Latest Go to latest Published: Dec 24, 2020 License: MIT Imports: 13 Imported by: 0

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

github.com/agrewal/crawler

README ¶

crawler

Documentation ¶

Index ¶

Variables
func ReaderToStoringFetchable(ctx context.Context, reader io.Reader, channelBufferSize int, store *Store, ...) <-chan fetcher.Fetchable
type Store
- func NewStore(dirname string) (*Store, error)
type StoringFetchable
- func NewStoringFetchable(urlStr string, store *Store, minInterval time.Duration) *StoringFetchable

Constants ¶

This section is empty.

Variables ¶

View Source

var (
	ErrAlreadyCrawledRecently = errors.New("error: already crawled recently")
)

Functions ¶

func ReaderToStoringFetchable ¶

func ReaderToStoringFetchable(ctx context.Context, reader io.Reader, channelBufferSize int, store *Store, minInterval time.Duration) <-chan fetcher.Fetchable

Types ¶

type Store ¶

type Store struct {
	// contains filtered or unexported fields
}

func NewStore ¶

func NewStore(dirname string) (*Store, error)

func (*Store) Close ¶

func (s *Store) Close()

func (*Store) Exists ¶

func (s *Store) Exists(url string) (bool, error)

func (*Store) Get ¶

func (s *Store) Get(url string) (*crawled_url.CrawledUrl, io.Closer, error)

func (*Store) LastCrawlTs ¶

func (s *Store) LastCrawlTs(url string) (int64, error)

func (*Store) Save ¶

func (s *Store) Save(url string, resp *http.Response) error

type StoringFetchable ¶

type StoringFetchable struct {
	// contains filtered or unexported fields
}

func NewStoringFetchable ¶

func NewStoringFetchable(urlStr string, store *Store, minInterval time.Duration) *StoringFetchable

func (*StoringFetchable) HandleResponse ¶

func (s *StoringFetchable) HandleResponse(resp *http.Response) error

func (*StoringFetchable) Id ¶

func (s *StoringFetchable) Id() string

func (*StoringFetchable) Request ¶

func (s *StoringFetchable) Request() (*http.Request, error)

func (*StoringFetchable) Validate ¶

func (s *StoringFetchable) Validate() error

Source Files ¶

View all Source files

store.go

Directories ¶

Path	Synopsis
cmd
crawler
crawled_url
fetcher

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL