crawler

package

v0.0.0-...-fef445f Latest Latest Go to latest Published: Mar 20, 2024 License: MIT Imports: 9 Imported by: 0

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

This section is empty.

This section is empty.

This section is empty.

type Article struct {
	URL         string
	Header      string
	Content     string
	Author      string
	IsSponsored bool
}

type Crawler struct {
	Retries         int
	RetriesInterval time.Duration
	URL             string
	// contains filtered or unexported fields
}

func New(timeout time.Duration, retries int) *Crawler

func (c *Crawler) ParseArticles() ([]Article, error)

Fetches and parses the articles

Articles is contained in tables with class 'el-item item '. They have same structure regardless of content table > tbody > tr > td > (content)