README
¶
Package cloudeng.io/webapi/operations
import cloudeng.io/webapi/operations
Package operations provides support for invoking various operations on web APIs.
Functions
Func RunCrawl
func RunCrawl[ScanT, EndpointT any](ctx context.Context, crawler *Crawler[ScanT, EndpointT], handler CrawlHandler[EndpointT]) error
RunCrawl is a convenience function that runs a crawler and calls the supplied handler for each Object crawled.
Func ScanDownloaded
func ScanDownloaded[EndpointT any](ctx context.Context, handler CrawlHandler[EndpointT]) error
Types
Type Auth
type Auth interface {
// WithAuthorization adds an authorization header or other required
// authorization information to the provided http.Request.
WithAuthorization(context.Context, *http.Request) error
}
Auth represents an authorization mechanism.
Type CrawlHandler
type CrawlHandler[EndpointT any] func(context.Context, content.Object[EndpointT, Response]) error
Type Crawler
type Crawler[ScanT any, EndpointT any] struct {
// contains filtered or unexported fields
}
Crawler is a generic crawler that can be used to iterate over a paginated API endpoint (using a Scanner) that enumerates objects that can be downloaded using a Fetcher. The Fetcher is responsible for decoding the results of each response from the paginated API and downloading the objects.
Functions
func NewCrawler[ScanT, EndpointT any](scanner *Scanner[ScanT], fetcher Fetcher[ScanT, EndpointT]) *Crawler[ScanT, EndpointT]
NewCrawler creates a new crawler that scans the API using the provided Scanner with the result of each scan being passed to the Fetcher to download each item returned by the scan.
Methods
func (c *Crawler[ScanT, EndpointT]) Run(ctx context.Context, ch chan<- []content.Object[EndpointT, Response]) error
Run runs the crawler. It consists of a scan loop that calls a Fetcher for each scan response.
Type DownloadHandler
type DownloadHandler[EndpointT any] func(ctx context.Context, path string, obj content.Object[EndpointT, Response]) error
Type Encoding
type Encoding int
Encoding represents the encoding scheme used for the response body.
Constants
JSONEncoding
JSONEncoding Encoding = iota
Type Endpoint
type Endpoint[T any] struct {
// contains filtered or unexported fields
}
Endpoint represents an API endpoint that whose response body is unmarshaled, by default using json.Unmarshal, into the specified type.
Functions
func NewEndpoint[T any](opts ...Option) *Endpoint[T]
NewEndpoint returns a new endpoint for the specified type.
Methods
func (ep *Endpoint[T]) Get(ctx context.Context, url string) (T, []byte, Encoding, error)
Get invokes a GET request on this endpoint (without a body).
func (ep *Endpoint[T]) GetUsingRequest(ctx context.Context, req *http.Request) (T, []byte, Encoding, *http.Response, error)
GetUsingRequest invokes a Get request on this endpoint using the supplied http.Request. The Body in the http.Response has already been read and its contents returned as the second return value.
Type Error
type Error struct {
Err error
Status string
StatusCode int
Attempts int
}
Methods
func (err *Error) Error() string
Type Fetcher
type Fetcher[ScanT, EndpointT any] interface {
Fetch(context.Context, ScanT, chan<- []content.Object[EndpointT, Response]) error
}
Fetcher is a generic interface for fetching objects from an API endpoint as part of a crawl. The Fetcher extracts/decodes items to be fetched from the result of a Scan and then downloads each item.
Type Option
type Option func(o *options)
Option represents an option that can be used when creating new Endpoints and Streams.
Functions
func WithAuth(a Auth) Option
WithAuth specifies the instance of Auth to use when making requests.
func WithRateController(rc *ratecontrol.Controller, statusCodes ...int) Option
WithRateController sets the rate controller to use to enforce rate control and backoff.
func WithUnmarshal(u Unmarshal, e Encoding) Option
WithUnmarshal specifies a custom unmarshaling function to use for decoding response bodies. The default is json.Unmarshal.
Type Paginator
type Paginator[T any] interface {
// Next is called with the returned type and response for that operation.
// The first URL to use in a scan is generated by calling Next with an empty
// payload and nil *http.Response
Next(ctx context.Context, t T, r *http.Response) (req *http.Request, done bool, err error)
}
Paginator represents the ability to generate the next request (URL with optional body) given the response from the previous request. Paginators are typically used with Scanners to iterate over a paginated API.
Type Response
type Response struct {
// The raw bytes of the response.
Bytes []byte
// The encoding used for Bytes.
Encoding Encoding
// When the response was received.
When time.Time
// Fields copied from the http.Response.
Headers http.Header
Trailers http.Header
ContentLength int64
StatusCode int
ProtoMajor, ProtoMinir int
TransferEncoding []string
// Any error encountered during the operation.
Error error
// Checkpoint is an opaque value that can be used to resume an
// operation at a later time. This is used generally used by implementations
// of Crawler/Fetcher.
Checkpoint []byte
// Current and Total, if non-zero, provide an indication of progress.
Current int64
Total int64
}
Response contains metadata for the result of an operation and is used for the response field of the content.Object returned by the Fetcher.
Methods
func (r *Response) FromHTTPResponse(hr *http.Response)
Type Scanner
type Scanner[T any] struct {
// contains filtered or unexported fields
}
Scanner provides the ability to iterate over a paginated API a page at a time.
Functions
func NewScanner[T any](paginator Paginator[T], opts ...Option) *Scanner[T]
NewScanner creates a new Scanner using the supplied paginator. The options are used to create the underlying Endpoint.
Methods
func (sc *Scanner[T]) Body() []byte
Body returns the body for the current page.
func (sc *Scanner[T]) Err() error
Err returns the first error encountered during scanning.
func (sc *Scanner[T]) Response() T
Response returns the response for the current page.
func (sc *Scanner[T]) Scan(ctx context.Context) bool
Scan iterates over the paginated API. It returns true if there is another page to scan, false otherwise.
Type Unmarshal
type Unmarshal func([]byte, any) error
Unmarshal represents a function that can be used to unmarshal a response body.
Documentation
¶
Overview ¶
Package operations provides support for invoking various operations on web APIs.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type Auth ¶
type Auth interface {
// WithAuthorization adds an authorization header or other required
// authorization information to the provided http.Request.
WithAuthorization(context.Context, *http.Request) error
}
Auth represents an authorization mechanism.
type CrawlHandler ¶
type Crawler ¶
Crawler is a generic crawler that can be used to iterate over a paginated API endpoint (using a Scanner) that enumerates objects that can be downloaded using a Fetcher. The Fetcher is responsible for decoding the results of each response from the paginated API and downloading the objects.
func NewCrawler ¶
func NewCrawler[ScanT, EndpointT any](scanner *Scanner[ScanT], fetcher Fetcher[ScanT, EndpointT]) *Crawler[ScanT, EndpointT]
NewCrawler creates a new crawler that scans the API using the provided Scanner with the result of each scan being passed to the Fetcher to download each item returned by the scan.
type Encoding ¶
type Encoding int
Encoding represents the encoding scheme used for the response body.
type Endpoint ¶
type Endpoint[T any] struct { // contains filtered or unexported fields }
Endpoint represents an API endpoint that whose response body is unmarshaled, by default using json.Unmarshal, into the specified type.
func NewEndpoint ¶
NewEndpoint returns a new endpoint for the specified type.
func (*Endpoint[T]) IssueRequest ¶
func (ep *Endpoint[T]) IssueRequest(ctx context.Context, req *http.Request) (T, []byte, Encoding, *http.Response, error)
IssueRequest invokes an arbitrary request on this endpoint using the supplied http.Request. The Body in the http.Response has already been read and its contents returned as the second return value.
type FS ¶
FS defines a filesystem interface to be broadly used by webapi packages and clients. It is defined in operations for convenience.
type Fetcher ¶
type Fetcher[ScanT, EndpointT any] interface { Fetch(context.Context, ScanT, chan<- []content.Object[EndpointT, Response]) error }
Fetcher is a generic interface for fetching objects from an API endpoint as part of a crawl. The Fetcher extracts/decodes items to be fetched from the result of a Scan and then downloads each item.
type Option ¶
type Option func(o *options)
Option represents an option that can be used when creating new Endpoints and Streams.
func WithRateController ¶
func WithRateController(rc *ratecontrol.Controller, statusCodes ...int) Option
WithRateController sets the rate controller to use to enforce rate control and backoff.
func WithUnmarshal ¶
WithUnmarshal specifies a custom unmarshaling function to use for decoding response bodies. The default is json.Unmarshal.
type Paginator ¶
type Paginator[T any] interface { // Next is called with the returned type and response for that operation. // The first URL to use in a scan is generated by calling Next with an empty // payload and nil *http.Response Next(ctx context.Context, t T, r *http.Response) (req *http.Request, done bool, err error) }
Paginator represents the ability to generate the next request (URL with optional body) given the response from the previous request. Paginators are typically used with Scanners to iterate over a paginated API.
type Response ¶
type Response struct {
// The raw bytes of the response.
Bytes []byte
// The encoding used for Bytes.
Encoding Encoding
// When the response was received.
When time.Time
// Fields copied from the http.Response.
Headers http.Header
Trailers http.Header
ContentLength int64
StatusCode int
ProtoMajor, ProtoMinir int
TransferEncoding []string
// Any error encountered during the operation.
Error error
// Checkpoint is an opaque value that can be used to resume an
// operation at a later time. This is used generally used by implementations
// of Crawler/Fetcher.
Checkpoint []byte
// Current and Total, if non-zero, provide an indication of progress.
Current int64
Total int64
}
Response contains metadata for the result of an operation and is used for the response field of the content.Object returned by the Fetcher.
func (*Response) FromHTTPResponse ¶
type Scanner ¶
type Scanner[T any] struct { // contains filtered or unexported fields }
Scanner provides the ability to iterate over a paginated API a page at a time.
func NewScanner ¶
NewScanner creates a new Scanner using the supplied paginator. The options are used to create the underlying Endpoint.
func (*Scanner[T]) HTTPResponse ¶
HTTPResponse returns the response for the current page.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package apicrawlcmd provides support for building command line tools that implement API crawls.
|
Package apicrawlcmd provides support for building command line tools that implement API crawls. |