fetching

package
v1.8.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 21, 2023 License: MIT Imports: 27 Imported by: 0

Documentation

Index

Constants

View Source 
const MaxConcurrentDownloads = 5

Variables

View Source 
var DoForClientFunc = DoForClient

Changed only during tests.

Functions

func BitRateToByteDuration 

func BitRateToByteDuration(bitRate int64) time.Duration

BitRateToByteDuration returns the duration between bytes for a given bit rate in bits per second, rounded down to nanoseconds.

func DoForClient 

func DoForClient(client *http.Client, req *http.Request) (*http.Response, error)

func GetProxyLoggingFunc 

func GetProxyLoggingFunc() func(req *http.Request) (*url.URL, error)

func MakeClient 

func MakeClient() *http.Client

func ParseRange 

func ParseRange(rangeHeader string, endMax int64) (rangeStart, rangeEnd int64, err error)

func ReadExactly 

func ReadExactly(r io.Reader, byteCount int) (data []byte, err error)

Types

type ConsoleDownloadProgressHandler added in v1.5.0 

type ConsoleDownloadProgressHandler struct {
}

func (*ConsoleDownloadProgressHandler) HandleBadHttpResponse added in v1.5.0 

func (handler *ConsoleDownloadProgressHandler) HandleBadHttpResponse(fromURL string, code int)

func (*ConsoleDownloadProgressHandler) HandleFailDownload added in v1.5.0 

func (handler *ConsoleDownloadProgressHandler) HandleFailDownload(fromURL string, workerId int, err error)

func (*ConsoleDownloadProgressHandler) HandleFinishDownload added in v1.5.0 

func (handler *ConsoleDownloadProgressHandler) HandleFinishDownload(fromURL string, workerId int)

func (*ConsoleDownloadProgressHandler) HandleHttpGetError added in v1.5.0 

func (handler *ConsoleDownloadProgressHandler) HandleHttpGetError(fromURL string, err error)

func (*ConsoleDownloadProgressHandler) HandleProgress added in v1.5.0 

func (handler *ConsoleDownloadProgressHandler) HandleProgress(fromURL string, workerId int, receivedBytes uint64)

func (*ConsoleDownloadProgressHandler) HandleReadError added in v1.5.0 

func (handler *ConsoleDownloadProgressHandler) HandleReadError(fromURL string, err error, receivedByteCount int64)

func (*ConsoleDownloadProgressHandler) HandleStartDownload added in v1.5.0 

func (handler *ConsoleDownloadProgressHandler) HandleStartDownload(fromURL string, workerId int)

type Download 

type Download struct {
	// contains filtered or unexported fields
}

Download wraps the retrieval of a resource at a URL using HTTP GET requests and exposes the received data through its implementation of the io.Reader interface.

The Read() method will only return a non-nil error other than io.EOF when the Content-Length of the requested resource changes during Download's attempts to retrieve it or the remote signals that it cannot serve a range-request made in an attempt to resume if the first GET was interrupted.

See Download.handler for Download's behavior in other error scenarios.

func NewDownload 

func NewDownload(ctx context.Context, resourceUrl string) *Download

func NewDownloadForConcurrentUse 

func NewDownloadForConcurrentUse(ctx context.Context, resourceUrl string, client *http.Client, handler DownloadProgressHandler, workerId int) *Download

func NewHandledDownload 

func NewHandledDownload(ctx context.Context, resourceUrl string, handler DownloadProgressHandler) *Download

func (*Download) Close 

func (dl *Download) Close() error

Close closes the underlying HTTP response body of the Download if one currently exists. If you Read() until you receive a non-nil error, you do not have to call Close(); otherwise you must.

func (*Download) Read 

func (dl *Download) Read(p []byte) (n int, err error)

Read reads some data of the requested resource into p. Calling Read() again after it returned a non-nil error results in undefined behaviour.

func (*Download) URL 

func (dl *Download) URL() string

type DownloadError 

type DownloadError string

func (DownloadError) Error 

func (err DownloadError) Error() string

type DownloadProgressHandler 

type DownloadProgressHandler interface {
	HandleStartDownload(fromURL string, workerId int)                  // The download of resource at given URL will be started.
	HandleProgress(fromURL string, workerId int, receivedBytes uint64) // The download of resource at given URL made some progress.
	HandleFinishDownload(fromURL string, workerId int)                 // The download of resource at given URL finished successfully.
	HandleFailDownload(fromURL string, workerId int, err error)        // The download of resource at given URL failed irrecoverably.

	HandleHttpGetError(fromURL string, err error)                    // The HTTP GET request to download the resource did not receive an HTTP response.
	HandleBadHttpResponse(fromURL string, code int)                  // A bad HTTP response code was received.
	HandleReadError(fromURL string, err error, firstByteIndex int64) // An error occurred while reading the response body (data) of the resource at the given URL.
}

DownloadProgressHandler is an interface which defines callbacks for typical events which will or may occur during a resource download via an HTTP GET request, such as receiving bytes, the connection being interrupted or a bad HTTP response code being received.

type Downloader 

type Downloader struct {
	// contains filtered or unexported fields
}

Downloader has helper functions for common use cases of Download, such as writing a resource to a file while downloading it, downloading multiple resources in parallel and verifying the hashsum or signature of downloading resources.

func NewDownloader 

func NewDownloader(ctx context.Context, handler DownloadProgressHandler) *Downloader

func (*Downloader) DownloadFile 

func (downloader *Downloader) DownloadFile(url string, fileInfo *config.FileInfo, filePath string) error

func (*Downloader) DownloadResource 

func (downloader *Downloader) DownloadResource(url string, processDownload func(dl *Download) error) error

func (*Downloader) DownloadResources 

func (downloader *Downloader) DownloadResources(urls []string, processDownload func(dl *Download) error) error

DownloadResources makes a goroutined call of processDownload with a *Download ready for Read()s for every url in urls, never allowing more than MaxConcurrentDownloads to be processed at the same time and returning a non-nil error if and only if any of the calls to processDownload do return a non-nil error or the context is cancelled.

func (*Downloader) DownloadSignedResource 

func (downloader *Downloader) DownloadSignedResource(fromURL string, keys []*rsa.PublicKey) ([]byte, error)

func (*Downloader) DownloadSignedResources 

func (downloader *Downloader) DownloadSignedResources(urls []string, keys []*rsa.PublicKey) (map[string][]byte, error)

func (*Downloader) DownloadToDirectory 

func (downloader *Downloader) DownloadToDirectory(baseUrl string, fileMap config.FileInfoMap, localDirPath string) error

func (*Downloader) DownloadToRAM 

func (downloader *Downloader) DownloadToRAM(fileMap config.FileInfoMap) (fileData map[string][]byte, dlErr error)

func (*Downloader) MustDownloadToDirectory 

func (downloader *Downloader) MustDownloadToDirectory(baseUrl string, fileMap config.FileInfoMap, localDirPath string)

func (*Downloader) MustDownloadToTempDirectory 

func (downloader *Downloader) MustDownloadToTempDirectory(baseUrl string, fileMap config.FileInfoMap, localDirPath string) (tempDirectoryPath string)

type EmptyHandler 

type EmptyHandler struct {
}

func (*EmptyHandler) HandleBadHttpResponse 

func (handler *EmptyHandler) HandleBadHttpResponse(fromURL string, code int)

func (*EmptyHandler) HandleFailDownload 

func (handler *EmptyHandler) HandleFailDownload(fromURL string, workerId int, err error)

func (*EmptyHandler) HandleFinishDownload 

func (handler *EmptyHandler) HandleFinishDownload(fromURL string, workerId int)

func (*EmptyHandler) HandleHttpGetError 

func (handler *EmptyHandler) HandleHttpGetError(fromURL string, err error)

func (*EmptyHandler) HandleProgress 

func (handler *EmptyHandler) HandleProgress(fromURL string, workerId int, receivedBytes uint64)

func (*EmptyHandler) HandleReadError 

func (handler *EmptyHandler) HandleReadError(fromURL string, err error, receivedByteCount int64)

func (*EmptyHandler) HandleStartDownload 

func (handler *EmptyHandler) HandleStartDownload(fromURL string, workerId int)

type LowercaseHeaders added in v1.5.0 

type LowercaseHeaders map[string][]string

LowercaseHeaders models http.Header with each header name being lower-case.

func NewLowercaseHeaders added in v1.5.0 

func NewLowercaseHeaders(headers http.Header) LowercaseHeaders

NewLowercaseHeaders returns a map of headers for the given http.Header with all header names in lower-case. Header names which overlap after becoming lower-case have their values merged into one header in an unspecified order.

func (LowercaseHeaders) Get added in v1.5.0 

func (h LowercaseHeaders) Get(header string) string

Get is analogous to http.Header.Get()

func (LowercaseHeaders) Values added in v1.5.0 

func (h LowercaseHeaders) Values(header string) []string

Values is analogous to http.Header.Values()

type TimeoutingReader 

type TimeoutingReader struct {
	Reader  io.Reader
	Timeout time.Duration
}

TimeoutingReader wraps an io.Reader with a configurable timeout.

func (*TimeoutingReader) Read 

func (trc *TimeoutingReader) Read(p []byte) (n int, err error)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.