Documentation
¶
Index ¶
- Constants
- type Anonymity
- type BloomCachedChan
- type CachedChan
- type GeoInfo
- type GeoInfoFetcher
- type Proxy
- func (p *Proxy) AddScore(delta int8)
- func (p *Proxy) DetectAnonymity(g utils.RequestHeadersGetter) (err error)
- func (p *Proxy) DetectGeoInfo(f GeoInfoFetcher) (err error)
- func (p *Proxy) DetectLatency()
- func (p *Proxy) DetectSpeed()
- func (p *Proxy) Equal(to *Proxy) bool
- func (p *Proxy) String() string
- func (p *Proxy) URL() string
Constants ¶
const ( // Unknown 探测不到匿名度 Unknown Anonymity = 0 // Transparent 透明：服务器知道你使用了代理，并且能查到原始IP Transparent Anonymity = 1 // Anonymous 普通匿名(较为少见)：服务器知道你使用了代理，但是查不到原始IP Anonymous Anonymity = 2 // Elite 高级匿名：服务器不知道你使用了代理 Elite Anonymity = 3 // 高匿名 // MaximumScore 代理最大得分 MaximumScore int8 = 100 )
const ( // NameOfIPAPIFetcher name of ip-api fetcher implements. NameOfIPAPIFetcher string = "ip-api" )
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Anonymity ¶
type Anonymity uint8
Anonymity 匿名度, 请求`https://httpbin.org/get?show_env=1` 根据ResponseBody中的 `X-Real-Ip` 和 `Via`字段判断。 另外如果代理支持HTTPS，访问https网站是没有匿名度的概念的， 因为此时代理只负责传输数据，并不能解析替换RequestHeaders。
type BloomCachedChan ¶
type BloomCachedChan struct {
// contains filtered or unexported fields
}
BloomCachedChan excludes proxy that are already sent to channel by placing a bloom filter in front of the channel.
func (*BloomCachedChan) Recv ¶
func (cc *BloomCachedChan) Recv() <-chan *Proxy
func (*BloomCachedChan) Send ¶
func (cc *BloomCachedChan) Send(ip, port string)
type CachedChan ¶
CachedChan provides a channel to transport proxies from spiders.
func NewBloomCachedChan ¶
func NewBloomCachedChan() CachedChan
NewBloomCachedChan returns a default bloom cached chan.
type GeoInfo ¶
type GeoInfo struct {
CountryName string `ip-api-json:"country"` // e.g. China
CountryCode string `ip-api-json:"countryCode"` // e.g. CN
RegionName string `ip-api-json:"regionName"` // e.g. Jiangsu
RegionCode string `ip-api-json:"region"` // e.g. JS
City string `ip-api-json:"city"` // e.g. Nanjing
Lat float32 `ip-api-json:"lat"` // e.g. 32.0617
Lon float32 `ip-api-json:"lon"` // e.g. 118.7778
ISP string `ip-api-json:"isp"` // e.g. Chinanet
}
GeoInfo 包括ip的地理位置相关信息，包括国家省市，运营商，经纬度等等 ip-api-json tag用于ip-api fetcher从 `http://www.ip-api.com/docs/api:json` 中拉取信息
type GeoInfoFetcher ¶
type GeoInfoFetcher interface {
// Do returns the geo information for ip.
// Use the fetcher to make a fetching request,
// and parse response body.
Do(ip string) (info *GeoInfo, err error)
// contains filtered or unexported methods
}
GeoInfoFetcher have some functions which use to fetch geo information from specify url. init(): you can init url formatter or some other initial work. fetch(): you can request to url with ip. unmarshal(): unmarshal to GeoInfo struct accord to the tag name which define in the struct.
func NewGeoInfoFetcher ¶
func NewGeoInfoFetcher(name string) (f GeoInfoFetcher)
NewGeoInfoFetcher returns a fetcher for name.
type Proxy ¶
type Proxy struct {
IP net.IP `json:"ip"`
Port uint32 `json:"port"`
GeoInfo *GeoInfo `json:"geo_info"`
Anon Anonymity `json:"anonymity"`
Latency uint32 `json:"latency"` // unit: ms
Speed uint32 `json:"speed"` // unit: kb/s
Score int8 `json:"score"` // [0-100]
CreatedAt time.Time `json:"created_at"`
CheckedAt time.Time `json:"checked_at"`
// contains filtered or unexported fields
}
Proxy IP Proxy data model.
func NewProxy ¶
NewProxy passes in the ip, port, calculates the other field values, and returns an initialized Proxy object
func (*Proxy) DetectAnonymity ¶
func (p *Proxy) DetectAnonymity(g utils.RequestHeadersGetter) (err error)
DetectAnonymity use a `utils.RequestHeadersGetter` to get a http request headers, and then use the following logic to determine the anonymity
If `X-Real-Ip` is equal to the public ip, the anonymity is `Transparent`. If `X-Real-Ip` is not equal to the public ip, and `Via` field exists, the anonymity is `Anonymous`. Otherwise, the anonymity is `Elite`.
func (*Proxy) DetectGeoInfo ¶
func (p *Proxy) DetectGeoInfo(f GeoInfoFetcher) (err error)
DetectGeoInfo set the GeoInfo field value by calling `NewGeoInfo`
func (*Proxy) DetectLatency ¶
func (p *Proxy) DetectLatency()
DetectLatency TODO: detect proxy lentency by request one website N times, and calculate average response time.
func (*Proxy) DetectSpeed ¶
func (p *Proxy) DetectSpeed()
DetectSpeed TODO: detect proxy speed by download a large file, and calculate speed `kb_of_file_size / download_cost_time = n kb/s`
Source Files