Documentation
¶
Index ¶
- Constants
- Variables
- type RateLimitManager
- type RateLimiter
- type RateRules
- type ScrapedData
- type Scraper
- func (s *Scraper) CallAPI(apiURL string) ([]byte, error)
- func (s *Scraper) EnableCache()
- func (s *Scraper) GetLeagues() ([]*models.League, error)
- func (s *Scraper) ScrapCharacterInventory(charName string) (*models.CharacterInventory, error)
- func (s *Scraper) ScrapCharacterSkills(charName string) (*models.CharacterSkills, error)
- func (s *Scraper) ScrapCharacters() ([]*models.Character, error)
- func (s *Scraper) ScrapEverything() (*ScrapedData, error)
- func (s *Scraper) ScrapStash(indexID int) (*models.StashTab, error)
- func (s *Scraper) ScrapWholeStash() ([]models.Tab, []*models.StashTab, error)
- func (s *Scraper) SetDemo(isDemo bool)
- func (s *Scraper) SetVerbosity(v int)
Constants ¶
const ( // StashURL is the official URL for the getting of a user account stash. StashURL = "https://www.pathofexile.com/character-window/get-stash-items?accountName=%s&realm=%s&league=%s&tabs=%d&tabIndex=%d" // ViewProfileURL is the official URL for the getting a user account main profile information. ViewProfileURL = "http://www.pathofexile.com/account/view-profile/%s" // ProfileCharactersURL is the official URL for the getting a user account characters. ProfileCharactersURL = "https://pathofexile.com/character-window/get-characters?accountName=%s" // ProfileCharacterItemsURL is the official URL for the getting a user account inventories. ProfileCharacterItemsURL = "https://www.pathofexile.com/character-window/get-items?accountName=%s&realm=%s&character=%s" // ProfileCharacterSkillsURL is the official URL for getting a user skills and jewels/abyss put in it. ProfileCharacterSkillsURL = "https://www.pathofexile.com/character-window/get-passive-skills?character=%s&accountName=%s" // LeaguesURL is the official URL for getting the list of all leagues. LeaguesURL = "http://api.pathofexile.com/leagues?type=main&compact=1" // DataDir is where all data are. DataDir = "data/" // DataCacheDir is the cache directory. DataCacheDir = DataDir + "cache/" // DemoDir is the cache directory. DemoDir = "demo/" )
Variables ¶
var ( // DefaultRateLimiter is 45 queries in 60 second. // So roughly less than 1 query per second (1350ms between queries). DefaultRateLimiter = NewRateLimiter(45, 60*time.Second) )
Functions ¶
This section is empty.
Types ¶
type RateLimitManager ¶
type RateLimitManager struct {
// contains filtered or unexported fields
}
RateLimitManager holds RateLimiter Every URL has its own rate limit, so it's maintained separately.
func NewPoeRateLimitManager ¶
func NewPoeRateLimitManager(poesessid string) RateLimitManager
NewPoeRateLimitManager returns a new rate limit manager already fill for path of exile API.
GGG rules I got so far:
- https://www.pathofexile.com/character-window/get-stash-items x-rate-limit-ip: 45:60:60,200:120:900 No more than 45 queries by minute (1 min ban) => 0.75 q/s No more than 200 queries by 2 minutes (15 min ban) => 1.3 q/s
- https://www.pathofexile.com/character-window/get-characters x-rate-limit-ip: 60:60:60,200:120:900 No more than 60 queries by minute (1 min ban) => 1 q/s No more than 200 queries by 2 minutes (15 min ban) => 1.3 q/s
- https://www.pathofexile.com/character-window/get-passive-skills No limit defined, so let's put 120:60:0
func NewRateLimitManager ¶
func NewRateLimitManager() RateLimitManager
NewRateLimitManager returns a new rate limit manager.
func (*RateLimitManager) GetRateLimiter ¶
func (r *RateLimitManager) GetRateLimiter(poesessid, url string) *RateLimiter
GetRateLimiter returns the best rate limiter given the session and the url provided. If none match, a default rate limiter is given.
func (*RateLimitManager) UpdateRateLimiter ¶
func (r *RateLimitManager) UpdateRateLimiter(poesessid, url string, rules, state RateRules)
UpdateRateLimiter updates or creates the corresponding rate limiter.
type RateLimiter ¶
type RateLimiter struct {
// NbMaxQuery is the number of queries allowed.
NbMaxQuery int
// Window is the window used to count the number of queries.
Window time.Duration
// NbQuery is the number of query currently made.
NbQuery int
// contains filtered or unexported fields
}
RateLimiter holds rule for limiting queries in time. Strategy is as follows: While we are under the rate limiting rule, just download everything. When hitting the limit, wait and query at a slower pace. The goal here is to not wait for few queries, but still be gentle for the one with a lot of call to make.
func NewRateLimiter ¶
func NewRateLimiter(nbMaxQuery int, window time.Duration) *RateLimiter
NewRateLimiter returns a new rate limiter.
func (*RateLimiter) EnableSlowMode ¶
func (r *RateLimiter) EnableSlowMode()
EnableSlowMode enables slow mode. This mode allows to query slowly instead of querying by batch.
func (*RateLimiter) NextQuery ¶
func (r *RateLimiter) NextQuery() (time.Duration, func())
NextQuery returns the time to wait before performing the next query. A closure function is returns which has to be called after the query has been called.
type RateRules ¶
type RateRules struct {
NbQuery int
WindowSeconds int // In seconds
BanTimeSeconds int // In seconds
}
RateRules are the rate limit rules
func ExtractFirstRuleFromString ¶
ExtractFirstRuleFromString extracts rate limit rules from a string. Format is: <NbQueries>:<MaxQueries>:<BanTimeSeconds> coma separated. Example:
2:60:60,2:240:900
Not that it get only the very first rule.
type ScrapedData ¶
type ScrapedData struct {
Demo bool
Account string
League string
Realm string
Date time.Time
Characters []*models.CharacterInventory
Skills []*models.CharacterSkills
Stash []*models.StashTab
TabsDesc []models.Tab
Wealth models.WealthBreakdown
}
ScrapedData holds everything scrapped.
type Scraper ¶
type Scraper struct {
// contains filtered or unexported fields
}
Scraper scraps path of exile site using its API.
func NewScraper ¶
NewScraper returns a configured scraper.
func (*Scraper) EnableCache ¶
func (s *Scraper) EnableCache()
EnableCache enable caching of queries. Useful for debug, do not enable it in production.
func (*Scraper) GetLeagues ¶
GetLeagues retrieves all available league names.
func (*Scraper) ScrapCharacterInventory ¶
func (s *Scraper) ScrapCharacterInventory(charName string) (*models.CharacterInventory, error)
ScrapCharacterInventory scraps the inventory of a given character.
func (*Scraper) ScrapCharacterSkills ¶
func (s *Scraper) ScrapCharacterSkills(charName string) (*models.CharacterSkills, error)
ScrapCharacterSkills scraps the inventory of a given character.
func (*Scraper) ScrapCharacters ¶
ScrapCharacters scraps all characters owned by a user.
func (*Scraper) ScrapEverything ¶
func (s *Scraper) ScrapEverything() (*ScrapedData, error)
ScrapEverything scraps items, characters, profile, inventory and so on...
func (*Scraper) ScrapStash ¶
ScrapStash scraps a stash from the official website.
func (*Scraper) ScrapWholeStash ¶
ScrapWholeStash scraps all tabs in a stash from the official website.
func (*Scraper) SetVerbosity ¶
SetVerbosity set verbosity of logs.
Source Files