sitemap

package module

v0.0.0-...-3607cf1 Latest Latest Go to latest Published: Jan 2, 2017 License: MIT Imports: 8 Imported by: 0

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

github.com/qubyte/sitemap

Links

Open Source Insights

README ¶

sitemap

sitemap is a library that can crawl a given URL and produce a map of sites within that domain and subdomains. It will honour rel="nofollow" attributes on links.

sitemap will attempt to gather information on links, scripts, images, videos, and audio.

Documentation ¶

Index ¶

type Site
- func (s *Site) Crawl()
- func (s *Site) MarshalJSON() ([]byte, error)
type SiteMap
- func NewSiteMap(origin *url.URL) *SiteMap
- func (m *SiteMap) Crawl(workers int, done chan<- bool)

Constants ¶

This section is empty.

Variables ¶

This section is empty.

Functions ¶

This section is empty.

Types ¶

type Site ¶

type Site struct {
	URL           url.URL
	Links         []url.URL
	NoFollowLinks []url.URL
	Scripts       []url.URL
	Images        []url.URL
	Videos        []url.URL
	Audio         []url.URL
	CSS           []url.URL
}

Site contains information about a site, including its URL, and the URLs of other sites it links to, scripts, and images.

func (*Site) Crawl ¶

func (s *Site) Crawl()

Crawl populates the resources of this Site instance by loading the associated link and processing the page. After processing the document, links are pushed into a channel for processing into the sitemap.

func (*Site) MarshalJSON ¶

func (s *Site) MarshalJSON() ([]byte, error)

MarshalJSON is a helper method which makes Site instances more JSON friendly.

type SiteMap ¶

type SiteMap struct {
	Sites map[string]*Site
	// contains filtered or unexported fields
}

SiteMap is a container for a set of URLs for assets used by the site, and a Mutex to allow safe access over go routines.

func NewSiteMap ¶

func NewSiteMap(origin *url.URL) *SiteMap

NewSiteMap is a constructor function used to return a SiteMap instance with an initialized sites map.

func (*SiteMap) Crawl ¶

func (m *SiteMap) Crawl(workers int, done chan<- bool)

Crawl begins the crawl. It hides the use of a WaitGroup behind a synchronization channel.

Source Files ¶

View all Source files

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL