README
¶
gopartpicker
A scraper for pcpartpicker.com for Go.
It is implemented using Colly.
Features
- Extract data from part list URLs
- Search for parts
- Extract data from part URLs
- Match PCPartPicker URLs using RegExp
- Support for multiple regions and currencies
Installation
It is assumed that you have Go 1.17 installed.
go get github.com/quakecodes/gopartpicker
Usage
Import the library.
import "github.com/quakecodes/gopartpicker"
Create a new scraper instance.
scraper := gopartpicker.NewScraper()
Fetch a part via URL.
part, err := scraper.GetPart("https://uk.pcpartpicker.com/product/g94BD3/amd-ryzen-5-5600x-37-ghz-6-core-processor-100-100000065box")
if err != nil {
log.Fatal(err)
}
fmt.Println(part.Name)
Fetch a part list via URL.
partList, err := scraper.GetPartList("https://uk.pcpartpicker.com/list/LNqWbh")
if err != nil {
log.Fatal(err)
}
fmt.Println(partList.Parts[0].Name)
Search for parts via a search term. The second argument is the region to search with.
parts, err := scraper.SearchParts("ryzen 5 3600", "uk")
// Some searches redirect to a product page, if you know that what you are searching will not redirect
// then you do not need to do the type assertion and if statement
_, ok := err.(*gopartpicker.RedirectError)
if ok {
// RedirectError.Error returns the URL of the redirect
part, err := scraper.GetPart(err.Error())
if err != nil {
log.Fatal(err)
}
fmt.Println(part.Name)
} else if err != nil {
log.Fatal(err)
} else {
fmt.Println(parts[0].Name)
}
Set headers for subsequent requests.
// "global" sets headers for all requests
scraper.SetHeaders("global", map[string]string{
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.71 Safari/537.36",
})
// Alternatively, make headers site specific by passing in a domain
scraper.SetHeaders("pcpartpicker.com", map[string]string{
"accept": "accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
})
// Make sure you set headers before calling request methods
scraper.GetPart("...")
Documentation
As of right now, there is no full documentation.
Feel free to ask for help by asking QuaKe in this Discord server.
Documentation
¶
Index ¶
- func ConvertListURL(URL string) string
- func ExtractPartListURLs(text string) []string
- func ExtractVendorName(URL string) string
- func MatchPCPPURL(URL string) bool
- func MatchPartListURL(URL string) bool
- func MatchProductURL(URL string) bool
- func StringPriceToFloat(price string) (float64, string, error)
- type CompNote
- type Part
- type PartList
- type PartListPart
- type Price
- type Rating
- type RedirectError
- type Scraper
- type SearchPart
- type Spec
- type Vendor
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func ConvertListURL ¶
Converts certain PCPartPicker list URLs to a specific format in order to prevent client side JS loading.
func ExtractPartListURLs ¶
Returns a list of all valid PCPartPicker part list URLs in the provided text.
func ExtractVendorName ¶
Extracts the name of a vendor from a PCPartPicker affiliate link.
func MatchPCPPURL ¶
Checks if a URL is a PCPartPicker URL, making sure to check all regional subdomains.
func MatchPartListURL ¶
Check if a URL is a PCPartPicker part list URL.
func MatchProductURL ¶
Checks if a URL is PCPartPicker product URL.
Types ¶
type CompNote ¶
type CompNote struct {
// The compatibility note's message.
Message string
// The level or severity of the note, e.g. Warning, Note.
Level string
}
A compatibility note for a part list.
type Part ¶
type Part struct {
// The part's type, e.g. CPU, Motherboard.
Type string
// The part's name.
Name string
// A slice of full image URLs.
Images []string
// The full URL for the part's PCPartPicker page.
URL string
// A slice of vendors supplying the part.
Vendors []Vendor
// The specifications of the part.
Specs []Spec
// The part's rating.
Rating Rating
}
A full part object with multiple vendors and reviews.
type PartList ¶
type PartList struct {
// The full URL for the part list.
URL string
// The parts in the part list.
Parts []PartListPart
// The price of all the part list's parts combined.
Price Price
// The total estimated wattage for the part list.
Wattage string
// Compatibility notes for the part list.
Compatibility []CompNote
}
Represents a PCPartPicker part list.
type PartListPart ¶
type PartListPart struct {
// The part's type, e.g. CPU, Motherboard.
Type string
// The part's name.
Name string
// The full URL for the part's image.
Image string
// The full URL for the part's PCPartPicker page.
URL string
// The vendor for the part that is offering it for the lowest price.
Vendor Vendor
}
A part from a part list. It has some data missing, such as all available vendors and specs.
type Price ¶
type Price struct {
// The base price of the item, without shipping, discounts or tax.
Base float64
// The price of shipping for the item.
Shipping float64
// The price of tax for the item.
Tax float64
// The price of discounts for the item.
Discounts float64
// The total price of the item.
Total float64
// The currency of the price of the item, e.g. £, $.
Currency string
// A string representing the item's total price, e.g. $1000 or 5500 RON.
TotalString string
}
Represents the total price of an item as well as additional fees.
type RedirectError ¶
type RedirectError struct {
URL string
}
func (RedirectError) Error ¶
func (r RedirectError) Error() string
type Scraper ¶
type Scraper struct {
// The Colly collector used for scraping.
Collector *colly.Collector
Headers map[string]map[string]string
}
func (Scraper) GetPartList ¶
Fetches data on a part list via URL.
func (Scraper) SearchParts ¶
func (s Scraper) SearchParts(searchTerm string, region string) ([]SearchPart, error)
Searches for parts using PCPartPicker's search function.
type SearchPart ¶
type SearchPart struct {
// The part's name.
Name string
// The full URL for the part's image.
Image string
// The full URL for the part's PCPartPicker page.
URL string
// The vendor for the part that is offering it for the lowest price.
Vendor Vendor
}
A part from PCPartPicker search results. It has a lot of data missing, such as full pricing information.
type Vendor ¶
type Vendor struct {
// The name of the vendor.
Name string
// The full URL for the vendor's image.
Image string
// Whether the vendor has the part in stock or not.
InStock bool
// The vendor's price for the part.
Price Price
// The full, affiliate URL to buy the part from.
URL string
}
The seller of a part.
Source Files