README
¶
cowtransfer
This is a client for https://cowtransfer.cn.
Most code is from https://github.com/Mikubill/cowtransfer-uploader. I have
removed dependency on github.com/cheggaaa/pb in favor of various progress
event hooks. The intention is to decouple the CLI from the library API.
Prepare your file
Download the file you want to your remote machine:
curl -Lo myfile.dat "http://www.example.com/myfile.dat"
Cowtransfer offers a password protection feature, but why trust 'em when you can encrypt locally? Of course, this step is optional.
openssl enc -aes-256-cbc -in ./myfile.dat -out myfilec.dat
Sometimes cowtransfer have problems with large files. If that happens, chop things up to smaller bits! The command below will create myfilec.aa, myfilec.ab, ..., each file being 512mb.
split --verbose -b 512M myfilec.dat myfilec.dat.
ls -lh myfilec.dat.*
Upload to Cowtransfer
Install the latest version of cowtransfer CLI client. Example for Linux shown below:
latest_ver=$(curl -fsSL "https://api.github.com/repos/imacks/cowtransfer/releases/latest" | grep tag_name | cut -d'"' -f4)
curl -Lo cowput "https://github.com/imacks/cowtransfer/releases/download/${latest_ver}/cowput_linux"
chmod +x cowput
./cowput -h
Time to upload:
files=$(ls -1 myfilec.dat.*)
./cowput $files
Lots of progress messages follows, but look out for the final download link. Here's an example:
link: https://cowtransfer.com/s/abab0000123456
Now you can use your local computer to visit the URL. You may simply choose to download what you want from the browser, but if there are a lot of files, read on to automate the download process too.
Download from Cowtransfer
You need to install cowtransfer CLI client on your local computer first. If your local PC is Linux too, installation steps are identical as for remote machine. I will show an example for Windows below:
$latestver = wget -UseBasicParsing "https://api.github.com/repos/imacks/cowtransfer/releases/latest" | select -expand Content | ConvertFrom-Json | select -expand tag_name
wget -OutFile cowtransfer.exe "https://github.com/imacks/cowtransfer/releases/download/${latestver}/cowput.exe"
cowput -h
You need the download link for the next step:
cowput https://cowtransfer.com/s/abab0000123456
This will get the actual direct download URLs for all the files. Download them using your favorite download tool.
On Windows, use the awesome 7-zip to open any of the downloaded files. 7-zip can handle decryption and split files.
A bit more work is necessary if using Linux:
To join split files:
cat myfilec.dat.* > myfilec.dat
To decrypt your file:
openssl enc -aes-256-cbc -d -in myfilec.dat > myfile.dat
Cleaning up
You may want to clear up your remote machine to save some disk space:
rm myfile.dat myfilec.dat myfilec.dat.*
Documentation
¶
Overview ¶
Package cowtransfer is a client API for cowtransfer.cn.
The file upload workflow involves opening up a session, upload one or more files, and finally closing the session. Files are not uploaded to Cowtransfer site directly, but to Qiniu object storage webservice.
Qiniu object storage webservice expects a file to be uploaded in multiple blocks. It offers APIs to signal the beginning of a file upload, upload of a block that will be associated with the file, and merging of blocks.
This package offers the ability to upload blocks with multi-threading by setting CowClient.MaxPushBlocks. This may not be faster than single threaded upload due to timeouts and retries.
Index ¶
Examples ¶
Constants ¶
This section is empty.
Variables ¶
var ( ErrInvalidResponse = errors.New("invalid response from endpoint") ErrBlockChecksum = errors.New("block has invalid checksum") ErrDownloadURL = errors.New("unsupported download URL") ErrDownloadNotFound = errors.New("download not found") ErrDownloadDeleted = errors.New("download is already deleted") ErrUploadInProgress = errors.New("upload in progress") )
Functions ¶
This section is empty.
Types ¶
type CowClient ¶
type CowClient struct {
// Timeout is the HTTP client timeout duration. Defaults to 10 seconds.
Timeout time.Duration
// MaxRetry is the maximum number of retries before giving up. Defaults to
// 3 tries.
MaxRetry int
// UserAgent overrides the default HTTP client useragent.
UserAgent string
// VerifyHash will use MD5 checksum to verify each block.
VerifyHash bool
// Password is an optional password that is used to protect content from
// downloads.
Password string
// Token overrides the default cookie.
Token string
// BlockSize is the size of each file part to download or upload. Defaults
// to 4096kb.
BlockSize int
// MaxPushBlocks is the maximum number of file parts to upload
// concurrently. This will be averaged amongst MaxPushFiles.
MaxPushBlocks int
// APIURL overrides the default Cowtransfer API endpoint.
APIURL string
// OSSURL overrides the default Qiniu OSS API endpoint.
OSSURL string
// contains filtered or unexported fields
}
CowClient is a client for CowTransfer.cn
func NewClient ¶ added in v1.1.0
func NewClient() *CowClient
NewClient creates a new CowClient instance with default values.
Example ¶
package main
import (
"fmt"
"github.com/imacks/cowtransfer"
)
func main() {
cc := cowtransfer.NewClient()
// default is transfer 1 block at a time.
// let's do 2 simultaneous block transfers
cc.MaxPushBlocks = 2
// you can add hooks to events
cc.OnStart(func(s *cowtransfer.UploadSession) {
fmt.Printf("session start\n")
})
cc.OnStop(func(s *cowtransfer.UploadSession) {
fmt.Printf("session stop\n")
})
cc.OnFileTransfer(func(fi *cowtransfer.FileTransfer) {
fmt.Printf("progressing\n")
})
// Upload method takes a slice of files. Here we will upload 1 file only.
dlURL, err := cc.Upload("./testdata/dummy.txt")
if err != nil {
panic(err)
}
fmt.Println(dlURL)
// to get the download URL for every file:
files, err := cc.Files(dlURL)
if err != nil {
panic(err)
}
for _, v := range files {
fmt.Printf("%v\n", v)
}
}
Output:
func (*CowClient) OnFileTransfer ¶ added in v1.1.0
func (cc *CowClient) OnFileTransfer(hook FileTransferFunc)
OnFileTransfer is a progress hook for file transfer progress.
func (*CowClient) OnStart ¶ added in v1.1.0
func (cc *CowClient) OnStart(hook SessionOpenCloseFunc)
OnStart is a progress hook for session start.
func (*CowClient) OnStop ¶ added in v1.1.0
func (cc *CowClient) OnStop(hook SessionOpenCloseFunc)
OnStart is a progress hook for session stop.
type FileInfo ¶ added in v1.1.0
type FileInfo struct {
FileName string `json:"fileName"`
Size int64 `json:"size"`
URL string `json:"url"`
Error error `json:"error"`
}
FileInfo represents information about a remote file.
type FileTransfer ¶ added in v1.1.0
type FileTransfer struct {
// Path to file on local filesystem.
Path string `json:"path"`
// Size of file.
Size int64 `json:"size"`
// State is current transfer state.
State TransferState `json:"state"`
// Blocks is total number of blocks.
Blocks int64 `json:"blocks"`
// DoneBlocks is the number of processed blocks.
DoneBlocks int64 `json:"done_blocks"`
// DoneSize is the total size of blocks processed.
DoneSize int64 `json:"done_size"`
// BlockNumber is block index (1-based) currently being processed.
BlockNumber int64 `json:"block_num"`
// BlockSize is the size of the block currently being processed.
BlockSize int `json:"block_size"`
// Retry is the current retry number.
Retry int `json:"retry"`
// RetriesLeft is the number of retries remaining before giving up.
RetriesLeft int `json:"retries_left"`
// Error is the error encountered in the last (retry) operation.
Error error `json:"error"`
}
File represents a file transfer operation.
func (*FileTransfer) MarshalJSON ¶ added in v1.1.0
func (f *FileTransfer) MarshalJSON() ([]byte, error)
type FileTransferFunc ¶ added in v1.1.0
type FileTransferFunc func(ft *FileTransfer)
FileTransferFunc is a file transfer progress hook.
type PushBlockErrorHandler ¶ added in v1.1.0
type PushBlockErrorHandler func(ft *FileTransfer) error
PushBlockErrorHandler is a handler for block upload failure.
type SessionOpenCloseFunc ¶ added in v1.1.0
type SessionOpenCloseFunc func(s *UploadSession)
SessionOpenCloseFunc is a session creation and close event hook.
type TransferState ¶ added in v1.1.0
type TransferState int
TransferState represents the state of a file transfer operation.
const ( InitTransfer TransferState = iota FinishTransfer RequestUpload ConfirmUpload Uploading Downloading DoBlock DoneBlock RetryBlock )
func (TransferState) String ¶ added in v1.1.0
func (ts TransferState) String() string
type UploadSession ¶ added in v1.1.0
type UploadSession struct {
UploadToken string `json:"uptoken"`
TransferGUID string `json:"transferguid"`
FileGUID string `json:"fileguid"`
UniqueURL string `json:"uniqueurl"`
Prefix string `json:"prefix"`
QRCode string `json:"qrcode"`
TempCode string `json:"temp_download_code"`
}
UploadSession is a file upload session. Multiple files can be uploaded in a single session.
Source Files
Directories