Documentation
¶
Overview ¶
Package robots handles caching robots.txt files
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Cacher ¶
type Cacher interface {
IndexExists() (bool, error)
Setup() error
Put(*Robots)
Get(host string) (*Robots, error)
}
Cacher handles the caching backend for robots.txt files
type ElasticSearch ¶
type ElasticSearch struct {
Client *elastic.Client
Index string
Type string
Bulk *elastic.BulkProcessor
}
ElasticSearch holds our Elasticsearch connection and index information
func (*ElasticSearch) Get ¶
func (e *ElasticSearch) Get(sh string) (*Robots, error)
Get retrieves a single cached robots.txt file
func (*ElasticSearch) IndexExists ¶
func (e *ElasticSearch) IndexExists() (bool, error)
IndexExists returns true if the index exists
func (*ElasticSearch) Mapping ¶
func (e *ElasticSearch) Mapping() string
Mapping is the mapping of our robots Index
func (*ElasticSearch) Setup ¶
func (e *ElasticSearch) Setup() error
Setup creates an index for caching robots.txt files
type Robots ¶
type Robots struct {
SchemeHost string `json:"-"`
StatusCode int `json:"status"`
Body string `json:"body"`
Expires string `json:"expires"`
Cached bool `json:"-"`
}
Robots is a single robots.txt response
func New ¶
New creates a new Robot & sets the ID to the host Robots are handled on a per-host basis https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt
func (*Robots) SetBody ¶
SetBody saves the body of a robots.txt file Since Group has unexported fields we must save the entire body, not just the group
func (*Robots) SetExpires ¶
SetExpires marks the date the robots.txt file should expire from cache
func (*Robots) SetStatusCode ¶
SetStatusCode sets the HTTP response code
Source Files