Package fetchbot provides a simple and flexible web crawler that follows the robots.txt policies and crawl delays.
Package gocrawl is a polite, slim and concurrent web crawler written in Go.
Package pipeline is the persistent and offline process part of crawler.
Package log handles logging for the crawler and frontend
Package standard implements the functionality for a non-headless crawler.
Package crawler defines all the functionality for page crawling
Package model contains the general data models and interfaces for the crawler.
Package crawlerdetect provides a Go version of the https://github.com/JayBizzle/Crawler-Detect PHP library It can be used to detect crawlers based on their HTTP User Agent header
This package helps the backend determine which clients are bots or crawlers.