got

package module
v0.7.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 1, 2022 License: MIT Imports: 15 Imported by: 17

README

Got.

Simple and fast concurrent downloader.

InstallationCLI UsageModule UsageLicense

Tests

Comparison

Comparison in cloud server:


[root@centos-nyc-12 ~]# time got -o /tmp/test -c 20 http://www.ovh.net/files/1Gio.dat
URL: http://www.ovh.net/files/1Gio.dat done!

real    0m8.832s
user    0m0.203s
sys 0m3.176s


[root@centos-nyc-12 ~]# time curl http://www.ovh.net/files/1Gio.dat --output /tmp/test1
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
								 Dload  Upload   Total   Spent    Left  Speed
100 1024M  100 1024M    0     0  35.6M      0  0:00:28  0:00:28 --:--:-- 34.4M

real    0m28.781s
user    0m0.379s
sys 0m1.970s

Installation

Download and install the latest release:
# go to tmp dir.
cd /tmp

# Download latest version.
curl -sfL https://git.io/getgot | sh

# Make the binary executable.
chmod +x /tmp/bin/got

# Move the binary to your PATH
sudo mv /tmp/bin/got /usr/bin/got
Or Go ahead compile it yourself:
go install github.com/melbahja/got/cmd/got@latest
Or from the AUR

Install got for the latest release version or got-git for the latest development version.

Note: these packages are not maintained by melbahja

Command Line Tool Usage

Simple usage:
got https://example.com/file.mp4
You can specify destination path:
got -o /path/to/save https://example.com/file.mp4
You can download multiple URLs and save them to directory:
got --dir /path/to/dir https://example.com/file.mp4 https://example.com/file2.mp4
You can download multiple URLs from a file:
got --dir /path/to/dir -f urls.txt

You can pipe multiple URLs:

cat urls.txt | got --dir /path/to/dir
Docs for available flags:
got help

Module Usage

You can use Got to download large files in your go code, the usage is simple as the CLI tool:

package main

import "github.com/melbahja/got"

func main() {

	g := got.New()

	err := g.Download("http://localhost/file.ext", "/path/to/save")

	if err != nil {
		// ..
	}
}

For more see PkgDocs.

How It Works?

Got takes advantage of the HTTP range requests support in servers RFC 7233, if the server supports partial content Got split the file into chunks, then starts downloading and merging the chunks into the destinaton file concurrently.

License

Got is provided under the MIT License © Mohammed El Bahja.

Documentation

Index

Examples

Constants

This section is empty.

Variables

View Source
var DefaultClient = &http.Client{
	Transport: &http.Transport{
		MaxIdleConns:        10,
		IdleConnTimeout:     30 * time.Second,
		TLSHandshakeTimeout: 5 * time.Second,
		Proxy:               http.ProxyFromEnvironment,
	},
}

DefaultClient is the default http client for got requests.

View Source
var DefaultFileName = "got.output"

DefaultFileName is the fallback name for GetFilename.

View Source
var ErrDownloadAborted = errors.New("Operation aborted")

ErrDownloadAborted - When download is aborted by the OS before it is completed, ErrDownloadAborted will be triggered

View Source
var UserAgent = "Got/1.0"

UserAgent is the default Got user agent to send http requests.

Functions

func GetFilename added in v0.1.2

func GetFilename(URL string) string

GetFilename it returns default file name from a URL.

func NewRequest added in v0.1.2

func NewRequest(ctx context.Context, method, URL string, header []GotHeader) (req *http.Request, err error)

NewRequest returns a new http.Request and error if any.

Types

type Chunk

type Chunk struct {
	Start, End uint64
}

Chunk represents the partial content range

type Download

type Download struct {
	Client *http.Client

	Concurrency uint

	URL, Dir, Dest string

	Interval, ChunkSize, MinChunkSize, MaxChunkSize uint64

	Header []GotHeader

	StopProgress bool
	// contains filtered or unexported fields
}

Download holds downloadable file config and infos.

Example
// Just for testing
destPath := createTemp()
defer clean(destPath)

ctx := context.Background()

dl := got.NewDownload(ctx, testUrl, destPath)

// Init
if err := dl.Init(); err != nil {
	fmt.Println(err)
}

// Start download
if err := dl.Start(); err != nil {
	fmt.Println(err)
}

fmt.Println("Done")
Output:

Done

func NewDownload added in v0.2.0

func NewDownload(ctx context.Context, URL, dest string) *Download

NewDownload returns new *Download with context.

func (*Download) AvgSpeed added in v0.2.0

func (d *Download) AvgSpeed() uint64

AvgSpeed returns average download speed.

func (*Download) Context added in v0.2.0

func (d *Download) Context() context.Context

Context returns download context.

func (*Download) DownloadChunk added in v0.2.0

func (d *Download) DownloadChunk(c *Chunk, dest io.Writer) error

DownloadChunk downloads a file chunk.

func (*Download) GetInfoOrDownload added in v0.6.0

func (d *Download) GetInfoOrDownload() (*Info, error)

Try downloading the first byte of the file using a range request. If the server supports range requests, then we'll extract the length info from content-range, Otherwise this just downloads the whole file in one go

func (*Download) Init

func (d *Download) Init() (err error)

Init set defaults and split file into chunks and gets Info, you should call Init before Start

func (*Download) IsRangeable added in v0.2.0

func (d *Download) IsRangeable() bool

IsRangeable returns file server partial content support state.

func (*Download) Path added in v0.6.0

func (d *Download) Path() string

Return constant path which will not change once the download starts

func (*Download) RunProgress added in v0.2.0

func (d *Download) RunProgress(fn ProgressFunc)

RunProgress runs ProgressFunc based on Interval and updates lastSize.

func (*Download) Size added in v0.2.0

func (d *Download) Size() uint64

Size returns downloaded size.

func (*Download) Speed added in v0.2.0

func (d *Download) Speed() uint64

Speed returns download speed.

func (*Download) Start

func (d *Download) Start() (err error)

Start downloads the file chunks, and merges them. Must be called only after init

func (*Download) TotalCost added in v0.2.0

func (d *Download) TotalCost() time.Duration

TotalCost returns download duration.

func (*Download) TotalSize added in v0.2.0

func (d *Download) TotalSize() uint64

TotalSize returns file total size (0 if unknown).

func (*Download) Write added in v0.2.0

func (d *Download) Write(b []byte) (int, error)

Write updates progress size.

type Got added in v0.2.0

type Got struct {
	ProgressFunc

	Client *http.Client
	// contains filtered or unexported fields
}

Got holds got download config.

Example
// Just for testing
destPath := createTemp()
defer clean(destPath)

g := got.New()

err := g.Download(testUrl, destPath)

if err != nil {
	log.Fatal(err)
	return
}

fmt.Println("done")
Output:

done
Example (WithContext)
// Just for testing
destPath := createTemp()
defer clean(destPath)

ctx := context.Background()

g := got.NewWithContext(ctx)

err := g.Download(testUrl, destPath)

if err != nil {
	log.Fatal(err)
	return
}

fmt.Println("done")
Output:

done

func New

func New() *Got

New returns new *Got with default context and client.

func NewWithContext added in v0.2.0

func NewWithContext(ctx context.Context) *Got

NewWithContext wants Context and returns *Got with default http client.

func (Got) Do added in v0.2.0

func (g Got) Do(dl *Download) error

Do inits and runs ProgressFunc if set and starts the Download.

func (Got) Download added in v0.2.0

func (g Got) Download(URL, dest string) error

Download creates *Download item and runs it.

type GotHeader added in v0.5.0

type GotHeader struct {
	Key   string
	Value string
}

type Info

type Info struct {
	Size      uint64
	Rangeable bool
}

Info holds downloadable file info.

type OffsetWriter added in v0.6.0

type OffsetWriter struct {
	io.WriterAt
	// contains filtered or unexported fields
}

func (*OffsetWriter) Write added in v0.6.0

func (dst *OffsetWriter) Write(b []byte) (n int, err error)

type ProgressFunc

type ProgressFunc func(d *Download)

ProgressFunc to show progress state, called by RunProgress based on interval.

Directories

Path Synopsis
cmd
got

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL