fetcher

package
v0.0.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 27, 2021 License: MIT Imports: 22 Imported by: 0

README

Fetcher

Fetch news from a entrance url, filter update time in the range, and storage them into a json. So, Microservice can read this fetched data directly.

Skeleton

  • fetcher.go: Fetch action implemented here
  • links.go: Fetch links from an entrance url.
  • article.go: Domain Object here to implement functions that microservice needed and invoked, such as Get, List and Search.
  • db.go: This is for fetched data persistence, load and save data directly.

Fetch kernal methods

There are kernal methods used to modified to match and fetch better, you can clone and rewrite the parts to implement your microservice.

  1. func (a *Article) fetchTitle() (string, error) {}
  2. func (a *Article) fetchUpdateTime() (*timestamppb.Timestamp, error) {}
  3. func (a *Article) fetchContent() (string, error) {}

Documentation

Index

Constants

This section is empty.

Variables

View Source
var ErrIgnoreCate error = errors.New("article is ignored by category")
View Source
var ErrIgnoreVIP error = errors.New("article is ignored by vip required")
View Source
var ErrTimeOverDays error = errors.New("article update time out of range")

Functions

func Fetch

func Fetch() error

Fetch fetch and storage all stuffs to `db/articles.json`

Types

type Article

type Article struct {
	Id            string
	Title         string
	Content       string
	WebsiteId     string
	WebsiteDomain string
	WebsiteTitle  string
	UpdateTime    *timestamppb.Timestamp
	U             *url.URL
	// contains filtered or unexported fields
}

func NewArticle

func NewArticle() *Article

func (*Article) Get

func (a *Article) Get(id string) (*Article, error)

Get read database and return the data by rawurl.

func (*Article) List

func (a *Article) List() ([]*Article, error)

List get all articles from database

func (*Article) Search

func (a *Article) Search(keyword ...string) ([]*Article, error)

type ByUpdateTime

type ByUpdateTime []*Article

func (ByUpdateTime) Len

func (u ByUpdateTime) Len() int

func (ByUpdateTime) Less

func (u ByUpdateTime) Less(i, j int) bool

func (ByUpdateTime) Swap

func (u ByUpdateTime) Swap(i, j int)

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL