robots

package

v0.0.0-...-3990c93 Latest Latest Go to latest Published: Jan 16, 2020 License: AGPL-3.0 Imports: 5 Imported by: 0

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

robots is a library written in golang for parsing the robots.txt file format.

First install the library with

go get xojoc.pw/crawl/robots

...then run...

go test

if something fails open an issue.

See godoc for the complete documentation.

robots was written by Alexandru Cojocaru (https://xojoc.pw).

Package robots parses a robots.txt file as specified by Wikipedia https://en.wikipedia.org/wiki/Robots.txt

This section is empty.

This section is empty.

This section is empty.

type Txt struct {
	CrawlDelay map[string]int
	Allow      map[string][]string
	Disallow   map[string][]string

	Sitemaps []string
}

Txt contains the robots.txt rules.

func Parse(r io.Reader) (*Txt, error)

Parse parses a robots.txt file.

func (t *Txt) Allowed(ua string, path string) bool

Allowed returns true if user agent ua can access path. False otherwise.

func (t *Txt) Delay(ua string) int

Delay returns the number of seconds to wait between successive accesses to the same host. Returns 0 if no delay is specified.