v1

package

v1.0.2 Latest Latest Go to latest Published: Apr 25, 2019 License: MIT Imports: 15 Imported by: 0

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

github.com/rnidev/go-webscraper

Links

Open Source Insights

Documentation ¶

Index ¶

Variables
func AddProductToCache(c *redis.Client, scrapedProduct *AmazonProduct, duration time.Duration) (err error)
func ConvertHTMLEntities(htmlString string) (clean string)
func NewScraperServer(client *redis.Client) v1.WebScraperServer
func StoreProduct(c *redis.Client, scrapedProduct *AmazonProduct) (err error)
type AmazonProduct
- func FetchProduct(c *redis.Client, asin string) (product AmazonProduct, err error)
- func GetProductFromCache(c *redis.Client, asin string) (product AmazonProduct, err error)
- func (product *AmazonProduct) GetProductInfoByASIN() (res *colly.Response, err error)

Constants ¶

This section is empty.

Variables ¶

View Source

var (
	//ErrMissingASIN is shared error message, returns if asin is missing
	ErrMissingASIN = errors.New("missing ASIN in request")
)

Functions ¶

func AddProductToCache ¶

func AddProductToCache(c *redis.Client, scrapedProduct *AmazonProduct, duration time.Duration) (err error)

AddProductToCache cache AmazonProduct with key cacheProduct:{ASIN}

func ConvertHTMLEntities ¶

func ConvertHTMLEntities(htmlString string) (clean string)

ConvertHTMLEntities converts scraped html entities and returns clean string

func NewScraperServer ¶

func NewScraperServer(client *redis.Client) v1.WebScraperServer

NewScraperServer takes a new redis client for scraper server

func StoreProduct ¶

func StoreProduct(c *redis.Client, scrapedProduct *AmazonProduct) (err error)

StoreProduct save AmazonProduct into Redis with key product:{ASIN}

Types ¶

type AmazonProduct ¶

type AmazonProduct struct {
	Asin       string   `json:"asin"`
	Name       string   `json:"name"`
	Categories []string `json:"categories"`
	Ranks      []string `json:"ranks"`
	Dimensions []string `json:"dimensions"`
	CreatedAt  string   `json:"created_at"`
}

AmazonProduct is the default product struct for ASIN service

func FetchProduct ¶

func FetchProduct(c *redis.Client, asin string) (product AmazonProduct, err error)

FetchProduct get AmazonProduct from Redis Todo: Fetch product will be used in Phase 2 for limited caching

func GetProductFromCache ¶

func GetProductFromCache(c *redis.Client, asin string) (product AmazonProduct, err error)

GetProductFromCache tries to grab cached product from cacheProduct with key cacheProduct:{ASIN}

func (*AmazonProduct) GetProductInfoByASIN ¶

func (product *AmazonProduct) GetProductInfoByASIN() (res *colly.Response, err error)

GetProductInfoByASIN takes asin, build target url, and returns product info

Source Files ¶

View all Source files

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL