sitemap

func ParseIndex ¶

func ParseIndex(data []byte) (Index, error)

ParseIndex create Index data from text

func ForceGet ¶ added in v0.3.0

func ForceGet(URL string, options interface{}) (Sitemap, error)

ForceGet is fetch and parse sitemap.xml/sitemapindex.xml. The difference with the Get function is that it ignores some errors.

Errors to Ignore:

・When sitemapindex.xml contains a sitemap.xml URL that cannot be retrieved. ・When sitemapindex.xml contains a sitemap.xml that is empty ・When sitemapindex.xml contains a sitemap.xml that has format problems.

Errors not to Ignore:

・When sitemap.xml/sitemapindex.xml could not retrieved. ・When sitemap.xml/sitemapindex.xml is empty. ・When sitemap.xml/sitemapindex.xml has format problems.

If you want **not** to ignore some errors, use the Get function.

func Get ¶

func Get(URL string, options interface{}) (Sitemap, error)

Get is fetch and parse sitemap.xml/sitemapindex.xml

If sitemap.xml or sitemapindex.xml has some problems, This function return error.

・When sitemap.xml/sitemapindex.xml could not retrieved. ・When sitemap.xml/sitemapindex.xml is empty. ・When sitemap.xml/sitemapindex.xml has format problems. ・When sitemapindex.xml contains a sitemap.xml URL that cannot be retrieved. ・When sitemapindex.xml contains a sitemap.xml that is empty ・When sitemapindex.xml contains a sitemap.xml that has format problems.

If you want to ignore these errors, use the ForceGet function.

smap, err := Get("https://issueoverflow.com/sitemap.xml", nil)
if err != nil {
	fmt.Println(err)
}

for _, URL := range smap.URL {
	fmt.Println(URL.Loc)
}

Output:

SetFetch(func(URL string, options interface{}) ([]byte, error) {
	req, err := http.NewRequest("GET", URL, nil)
	if err != nil {
		return []byte{}, err
	}

	// Set User-Agent
	req.Header.Set("User-Agent", "MyBot")

	// Set timeout
	timeout := time.Duration(10 * time.Second)
	client := http.Client{
		Timeout: timeout,
	}

	// Fetch data
	res, err := client.Do(req)
	if err != nil {
		return []byte{}, err
	}
	defer res.Body.Close()

	body, err := io.ReadAll(res.Body)
	if err != nil {
		return []byte{}, err
	}

	return body, err
})

smap, err := Get("https://issueoverflow.com/sitemap.xml", nil)
if err != nil {
	fmt.Println(err)
}

for _, URL := range smap.URL {
	fmt.Println(URL.Loc)
}

Output:

func Parse ¶

func Parse(data []byte) (Sitemap, error)

Parse create Sitemap data from text