README
¶
gahttp
Async/concurrent HTTP requests for Go with rate-limiting.
Work in progress.
Example
package main
import (
"fmt"
"net/http"
"time"
"github.com/tomnomnom/gahttp"
)
func printStatus(req *http.Request, resp *http.Response, err error) {
if err != nil {
return
}
fmt.Printf("%s: %s\n", req.URL, resp.Status)
}
func main() {
p := gahttp.NewPipeline()
p.SetConcurrency(20)
p.SetRateLimit(time.Second * 1)
urls := []string{
"http://example.com",
"http://example.com",
"http://example.com",
"http://example.net",
"http://example.org",
}
for _, u := range urls {
p.Get(u, gahttp.Wrap(printStatus, gahttp.CloseBody))
}
p.Done()
p.Wait()
}
TODO
DoneAndWait()func?- Helper for writing responses to channel? (e.g.
func ChanWriter() (chan *Response, procFn))- For when you don't want to do the work concurrently
- Actually handle timeouts / provide context interface for cancellation etc?
Documentation
¶
Index ¶
- func NewClient(opts ClientOptions) *http.Client
- func NewDefaultClient() *http.Client
- type ClientOptions
- type Pipeline
- func (p *Pipeline) Do(r *http.Request, fn ProcFn)
- func (p *Pipeline) Done()
- func (p *Pipeline) Get(u string, fn ProcFn) error
- func (p *Pipeline) Post(u string, body io.Reader, fn ProcFn) error
- func (p *Pipeline) Run()
- func (p *Pipeline) SetClient(c *http.Client)
- func (p *Pipeline) SetConcurrency(c int)
- func (p *Pipeline) SetRateLimit(d time.Duration)
- func (p *Pipeline) SetRateLimitMillis(m int)
- func (p *Pipeline) Wait()
- type ProcFn
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func NewClient ¶
func NewClient(opts ClientOptions) *http.Client
NewClient returns a new client with the specified options
func NewDefaultClient ¶
NewDefaultClient returns the default HTTP client
Types ¶
type ClientOptions ¶
type ClientOptions int
ClientOptions are a bitmask of options for HTTP clients
const ( // Don't follow redirects NoRedirects ClientOptions = 1 << iota // Skip verification of TLS certificates SkipVerify )
type Pipeline ¶
type Pipeline struct {
// contains filtered or unexported fields
}
a Pipeline is the main component of the gahttp package. It orchestrates making requests, optionally rate limiting them
func NewPipeline ¶
func NewPipeline() *Pipeline
New returns a new *Pipeline for the provided concurrency level
func NewPipelineWithClient ¶
NewWithClient returns a new *Pipeline for the provided concurrency level, and uses the provided *http.Client to make requests
func (*Pipeline) Do ¶
Do is the pipeline's generic request function; similar to http.DefaultClient.Do(), but it also accepts a ProcFn which will be called when the request has been executed
func (*Pipeline) Done ¶
func (p *Pipeline) Done()
Done should be called to signal to the pipeline that all requests that will be made have been enqueued. This closes the internal channel used to send requests to the workers that are executing the HTTP requests.
func (*Pipeline) Get ¶
Get is a convenience wrapper around the Do() function for making HTTP GET requests. It accepts a URL and the ProcFn to process the response.
func (*Pipeline) Post ¶
Post is a convenience wrapper around the Do() function for making HTTP POST requests. It accepts a URL, an io.Reader for the POST body, and a ProcFn to process the response.
func (*Pipeline) Run ¶
func (p *Pipeline) Run()
Run puts the pipeline into a running state. It launches the worker processes that execute the HTTP requests. Run() is called automatically by Do(), Get() and Post(), so it's often not necessary to call it directly.
func (*Pipeline) SetClient ¶
SetClient sets the HTTP client used by the pipeline to make HTTP requests. It can only be set before the pipeline is running
func (*Pipeline) SetConcurrency ¶
SetConcurrency sets the concurrency level for the pipeline. It can only be set before the pipeline is running
func (*Pipeline) SetRateLimit ¶
SetRateLimit sets the delay between requests to a given hostname
func (*Pipeline) SetRateLimitMillis ¶
SetRateLimitMillis sets the delay between request to a given hostname in milliseconds. This function is provided as a convenience, to make it easy to accept integer values as command line arguments.
type ProcFn ¶
a ProcFn is a function that processes an HTTP response. The HTTP request is provided for context, along with any error that occurred.
func CloseBody ¶
CloseBody wraps a ProcFn and returns a version of it that automatically closed the response body
Source Files