ranger

package module
v1.0.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jan 24, 2020 License: BSD-2-Clause-Views Imports: 9 Imported by: 0

README

ranger - io.ReaderAt with range requests! coverage report

INSTALL

$ go get howett.net/ranger

OVERVIEW

Package ranger provides an implementation of io.ReaderAt and io.ReadSeeker which makes partial document requests. Ranger ships with a range fetcher that operates on an HTTP resource using the Range: header.

USE

package main

import (
	"archive/zip"
	"io"
	"howett.net/ranger"
	"net/url"
	"os"
)

func main() {
	url, _ := url.Parse("http://example.com/example.zip")

	reader, _ := ranger.NewReader(&ranger.HTTPRanger{URL: url})
	length, _ := reader.Length()
	zipreader, _ := zip.NewReader(reader, length)

	data := make([]byte, zipreader.File[0].UncompressedSize64)
	rc, _ := zipreader.File[0].Open()
	io.ReadFull(rc, data)
	rc.Close()
}

Documentation

Overview

Package ranger implements an io.ReaderAt and io.ReadSeeker-compliant implementation of a caching HTTP range request client.

Index

Examples

Constants

View Source
const DefaultBlockSize int = 128 * 1024

DefaultBlockSize is the default size for the blocks that are downloaded from the server and cached.

Variables

View Source
var (
	// ErrResourceChanged is the error returned by Read when the underlying resource's integrity can no longer be verified.
	// In the case of HTTP, this usually happens when the remote document's validator has changed.
	ErrResourceChanged = errors.New("unsatisfiable range request; resource may have changed")

	// ErrResourceNotFound is returned by the first Read operation that determines that a resource is inaccessible.
	ErrResourceNotFound = errors.New("resource not found")
)

Functions

This section is empty.

Types

type Block

type Block struct {
	Length int64
	Data   []byte
}

Block represents a block returned from a ranged read

type ByteRange

type ByteRange struct {
	Start, End int64
}

ByteRange represents a not-yet-fetched range of bytes

type HTTPClient

type HTTPClient interface {
	Do(*http.Request) (*http.Response, error)
	Get(string) (*http.Response, error)
	Head(string) (*http.Response, error)
}

HTTPClient is an interface describing the methods required from net/http.Client

type HTTPRanger

type HTTPRanger struct {
	URL                            *url.URL
	Client                         HTTPClient
	UserAgent                      string
	DisableAcceptRangesHeaderCheck bool
	// contains filtered or unexported fields
}

HTTPRanger is a RangeFetcher that uses the HTTP Range: header to fetch blocks.

HTTPRanger first makes a HEAD request and then between 0 and Length()/BlockSize GET requests, attempting whenever possible to optimize for a lower number of requests.

No network requests are made until the first I/O-related function call.

func (*HTTPRanger) ExpectedLength

func (r *HTTPRanger) ExpectedLength() (int64, error)

ExpectedLength returns the length, in bytes, of the ranged-over file.

func (*HTTPRanger) FetchRanges

func (r *HTTPRanger) FetchRanges(ranges []ByteRange) ([]Block, error)

FetchRanges requests ranges from the HTTP server.

type RangeFetcher

type RangeFetcher interface {
	FetchRanges([]ByteRange) ([]Block, error)
	ExpectedLength() (int64, error)
}

RangeFetcher is the interface that wraps the FetchBlocks method.

FetchBlocks fetches the specified block ranges and returns any errors encountered in doing so.

Length returns the length, in bytes, of the ranged-over source.

Initialize, called once and passed the Reader's block size, performs any necessary setup tasks for the RangeFetcher

type Reader

type Reader struct {
	// the range fetcher with which to download blocks
	Fetcher RangeFetcher

	// size of the blocks fetched from the source and cached; lower values translate to lower memory usage, but typically require more requests
	BlockSize int
	// contains filtered or unexported fields
}

Reader is an io.ReaderAt and io.ReadSeeker backed by a partial block store.

Example
url, _ := url.Parse(testServer.URL + "/b.zip")

reader, _ := NewReader(&HTTPRanger{URL: url})
length, _ := reader.Length()
zipreader, _ := zip.NewReader(reader, length)

for i, v := range zipreader.File {
	fmt.Printf("[%d]: %s (%d bytes)\n", i, v.Name, v.UncompressedSize64)
}

data := make([]byte, 16)

rc, _ := zipreader.File[0].Open()
defer rc.Close()

io.ReadFull(rc, data)

fmt.Printf("Data from f00: `%s`\n", string(data))
Output:

[0]: f00 (640 bytes)
[1]: f01 (640 bytes)
[2]: f02 (640 bytes)
[3]: f03 (640 bytes)
[4]: f04 (640 bytes)
[5]: f05 (640 bytes)
[6]: f06 (640 bytes)
[7]: f07 (640 bytes)
[8]: f08 (640 bytes)
[9]: f09 (640 bytes)
Data from f00: `f0000000000BEGIN`

func NewReader

func NewReader(fetcher RangeFetcher) (*Reader, error)

NewReader returns a newly-initialized Reader, which also initializes its provided RangeFetcher. It returns the new reader and an error, if any.

func (*Reader) Length

func (r *Reader) Length() (int64, error)

Length returns the length of the ranged-over source.

func (*Reader) Read

func (r *Reader) Read(p []byte) (int, error)

Read reads len(p) bytes from ranged-over source. It returns the number of bytes read and the error, if any. EOF is signaled by a zero count with err set to io.EOF.

func (*Reader) ReadAt

func (r *Reader) ReadAt(p []byte, off int64) (int, error)

ReadAt reads len(p) bytes from the ranged-over source. It returns the number of bytes read and the error, if any. ReadAt always returns a non-nil error when n < len(b). At end of file, that error is io.EOF.

func (*Reader) Seek

func (r *Reader) Seek(off int64, whence int) (int64, error)

Seek sets the offset for the next Read to offset, interpreted according to whence: 0 means relative to the origin of the file, 1 means relative to the current offset, and 2 means relative to the end. It returns the new offset and an error, if any.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL