robots

package

v0.0.0-...-4337d54 Latest Latest Go to latest Published: Jul 3, 2013 License: MIT Imports: 6 Imported by: 1

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

github.com/ChuckHa/Squid

Links

Open Source Insights

Documentation ¶

Overview ¶

A package for parsing robots.txt

Index ¶

func CleanInput(s string) string
func GetRobotsTxtUrl(rawurl string) string
type RobotsTxt
- func NewRobotsTxtFromUrl(rawurl string) *RobotsTxt
type Rules
- func GetRules(contents string) Rules
- func (r Rules) Add(key, value string)
Bugs

Constants ¶

This section is empty.

Variables ¶

This section is empty.

Functions ¶

func CleanInput ¶

func CleanInput(s string) string

func GetRobotsTxtUrl ¶

func GetRobotsTxtUrl(rawurl string) string

GetRobotsTxtUrl returns the location of robots.txt given a URL that points to somewhere on the server.

Types ¶

type RobotsTxt ¶

type RobotsTxt struct {
	DisallowAll, AllowAll bool
	// User-agents to disallowed URLs
	Rules Rules
	Url   *url.URL
	// contains filtered or unexported fields
}

func NewRobotsTxtFromUrl ¶

func NewRobotsTxtFromUrl(rawurl string) *RobotsTxt

func (*RobotsTxt) Allowed ¶

func (r *RobotsTxt) Allowed(ua, rawurl string) bool

Ask if a specific UserAgent and URL that it wants to crawl is an allowed action. BUG(ChuckHa): Will fail when UserAgent: * and Disallow: / followed by UserAgent: Squidbot and Disallow:

func (*RobotsTxt) GetRobotsTxtFromUrl ¶

func (r *RobotsTxt) GetRobotsTxtFromUrl(robotsUrl string)

Actually get the contents from some robots.txt url.

func (*RobotsTxt) NotAllowed ¶

func (r *RobotsTxt) NotAllowed(ua, rawurl string) bool

type Rules ¶

type Rules map[string][]string

func GetRules ¶

func GetRules(contents string) Rules

func (Rules) Add ¶

func (r Rules) Add(key, value string)

Notes ¶

Bugs ¶

Will fail when UserAgent: * and Disallow: / followed by UserAgent: Squidbot and Disallow:

Source Files ¶

View all Source files

robots.go

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL