README
¶
Read the docs.
Documentation
¶
Overview ¶
A Golang port(?) of https://github.com/Anorov/cloudflare-scrape
Index ¶
- Variables
- func Get(url string) (resp *http.Response, err error)
- func IsCloudflareChallenge(resp *http.Response) bool
- func NewRoundTripper(options *Options) http.RoundTripper
- func NodeExecute(javascript string) (string, error)
- type ChallengeAnswer
- type CookieFilter
- type FilteringJar
- type JSExecutor
- type JSExecutorFunc
- type Options
Constants ¶
This section is empty.
Variables ¶
var DefaultClient *http.Client
var ( // This error is returned if we could not solve cloudflare's challenge. // Most likely, this means Cloudflare changed their "challenge" ErrCouldNotSolve = errors.New("Could not solve cloudflare challenge") )
var NodeExecutor = JSExecutorFunc(NodeExecute)
Functions ¶
func IsCloudflareChallenge ¶
func NewRoundTripper ¶
func NewRoundTripper(options *Options) http.RoundTripper
Creates a RoundTripper that solves Cloudflare "I Am Under Attack" / "anti-bot" page.
To use this: create an http.Client with a Transport field set to the RoundTripper that this function creates.
A nil value for options will set reasonable defaults for some definition of reasonable.
But in the spirit of Go, where it is *actually* encouraged for each package to define their own set of random global variables, you may find an existing http.Client defined already as DefaultClient.
func NodeExecute ¶
Executes a javascript expression with node. This executor makes no security guarantees.
Types ¶
type ChallengeAnswer ¶
type ChallengeAnswer struct {
Challenge string
ChallengePass string
JsAnswer string
ExecutedJavascript string
}
func SolveChallenge ¶
func SolveChallenge(hostname string, body []byte, executor JSExecutor) (*ChallengeAnswer, error)
Given a hostname, the cloudflare anti bot page, and a javascript executor, this returns the "answer" to cloudflare's challenge.
type CookieFilter ¶
func CookiesWithName ¶
func CookiesWithName(names ...string) CookieFilter
type FilteringJar ¶
type FilteringJar struct {
Filter CookieFilter
*cookiejar.Jar
}
func NewFilteringJar ¶
func NewFilteringJar(filter CookieFilter) *FilteringJar
func (*FilteringJar) SetCookies ¶
func (j *FilteringJar) SetCookies(u *url.URL, cookies []*http.Cookie)
type JSExecutor ¶
A JSExecutor executes a javascript expression, and returns the resulting string. Note that the javascript that it must execute can be arbitrary.
type JSExecutorFunc ¶
type Options ¶
type Options struct {
// If nil, this uses the default executor, which uses node.js
Executor JSExecutor
// If nil, this uses the default http Transport
RoundTripper http.RoundTripper
}
Source Files
Directories