tusd

package module
v0.0.0-...-16f2b1d Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 24, 2016 License: MIT Imports: 12 Imported by: 0

README

tusd

tus is a protocol based on HTTP for resumable file uploads. Resumable means that an upload can be interrupted at any moment and can be resumed without re-uploading the previous data again. An interruption may happen willingly, if the user wants to pause, or by accident in case of an network issue or server outage.

tusd is the official reference implementation of the tus resumable upload protocol. The protocol specifies a flexible method to upload files to remote servers using HTTP. The special feature is the ability to pause and resume uploads at any moment allowing to continue seamlessly after e.g. network interruptions.

Protocol version: 1.0.0

Getting started

You can download ready-to-use packages including binaries for OS X, Linux and Windows in various formats of the latest release.

Compile from source

Requirements:

  • Go (1.4 or newer)

Running tusd from source:

Clone the git repository and cd into it.

git clone git@github.com:tus/tusd.git
cd tusd

Now you can run tusd:

go run cmd/tusd/main.go

Using tusd manually

Besides from running tusd using the provided binary, you can embed it into your own Go program:

package main

import (
	"github.com/nagae-memooff/tusd"
	"github.com/nagae-memooff/tusd/filestore"
	"net/http"
)

func main() {
	// Create a new FileStore instance which is responsible for
	// storing the uploaded file on disk in the specified directory.
	// If you want to save them on a different medium, for example
	// a remote FTP server, you can implement your own storage backend
	// by implementing the tusd.DataStore interface.
	store := filestore.FileStore{
		Path: "./uploads",
	}

	// A storage backend for tusd may consist of multiple different parts which
	// handle upload creation, locking, termination and so on. The composer is a
	// place where all those seperated pieces are joined together. In this example
	// we only use the file store but you may plug in multiple.
	composer := tusd.NewStoreComposer()
	store.UseIn(composer)

	// Create a new HTTP handler for the tusd server by providing a configuration.
	// The StoreComposer property must be set to allow the handler to function.
	handler, err := tusd.NewHandler(tusd.Config{
		BasePath:      "files/",
		StoreComposer: composer,
	})
	if err != nil {
		panic("Unable to create handler: %s", err)
	}

	// Right now, nothing has happened since we need to start the HTTP server on
	// our own. In the end, tusd will start listening on and accept request at
	// http://localhost:8080/files
	http.Handle("files/", http.StripPrefix("files/", handler))
	err = http.ListenAndServe(":8080", nil)
	if err != nil {
		panic("Unable to listen: %s", err)
	}
}

Please consult the online documentation for more details about tusd's APIs and its sub-packages.

Implementing own storages

The tusd server is built to be as flexible as possible and to allow the use of different upload storage mechanisms. By default the tusd binary includes filestore which will save every upload to a specific directory on disk.

If you have different requirements, you can build your own storage backend which will save the files to S3, a remote FTP server or similar. Doing so is as simple as implementing the tusd.DataStore interface and using the new struct in the configuration object. Please consult the documentation about detailed information about the required methods.

Packages

This repository does not only contain the HTTP server's code but also other useful tools:

  • s3store: A storage backend using AWS S3
  • filestore: A storage backend using the local file system
  • memorylocker: An in-memory locker for handling concurrent uploads
  • consullocker: A locker using the distributed Consul service
  • limitedstore: A storage wrapper limiting the total used space for uploads

Running the testsuite

Build Status Build status

go test -v ./...

License

This project is licensed under the MIT license, see LICENSE.txt.

Documentation

Overview

Package tusd provides ways to accept tus 1.0 calls using HTTP.

tus is a protocol based on HTTP for resumable file uploads. Resumable means that an upload can be interrupted at any moment and can be resumed without re-uploading the previous data again. An interruption may happen willingly, if the user wants to pause, or by accident in case of an network issue or server outage (http://tus.io).

The basics of tusd

tusd was designed in way which allows an flexible and customizable usage. We wanted to avoid binding this package to a specific storage system – particularly a proprietary third-party software. Therefore tusd is an abstract layer whose only job is to accept incoming HTTP requests, validate them according to the specification and finally passes them to the data store.

The data store is another important component in tusd's architecture whose purpose is to do the actual file handling. It has to write the incoming upload to a persistent storage system and retrieve information about an upload's current state. Therefore it is the only part of the system which communicates directly with the underlying storage system, whether it be the local disk, a remote FTP server or cloud providers such as AWS S3.

Using a store composer

The only hard requirements for a data store can be found in the DataStore interface. It contains methods for creating uploads (NewUpload), writing to them (WriteChunk) and retrieving their status (GetInfo). However, there are many more features which are not mandatory but may still be used. These are contained in their own interfaces which all share the *DataStore suffix. For example, GetReaderDataStore which enables downloading uploads or TerminaterDataStore which allows uploads to be terminated.

The store composer offers a way to combine the basic data store - the core - implementation and these additional extensions:

composer := tusd.NewStoreComposer()
composer.UseCore(dataStore) // Implements DataStore
composer.UseTerminater(terminater) // Implements TerminaterDataStore
composer.UseLocker(locker) // Implements LockerDataStore

The corresponding methods for adding an extension to the composer are prefixed with Use* followed by the name of the corresponding interface. However, most data store provide multiple extensions and adding all of them manually can be tedious and error-prone. Therefore, all data store distributed with tusd provide an UseIn() method which does this job automatically. For example, this is the S3 store in action (see S3Store.UseIn):

store := s3store.New(…)
locker := memorylocker.New()
composer := tusd.NewStoreComposer()
store.UseIn(composer)
locker.UseIn(composer)

Finally, once you are done with composing your data store, you can pass it inside the Config struct in order to create create a new tusd HTTP handler:

config := tusd.Config{
  StoreComposer: composer,
  BasePath: "/files/",
}
handler, err := tusd.NewHandler(config)

This handler can then be mounted to a specific path, e.g. /files:

http.Handle("/files/", http.StripPrefix("/files/", handler))

Index

Examples

Constants

This section is empty.

Variables

View Source
var (
	ErrUnsupportedVersion  = errors.New("unsupported version")
	ErrMaxSizeExceeded     = errors.New("maximum size exceeded")
	ErrInvalidContentType  = errors.New("missing or invalid Content-Type header")
	ErrInvalidUploadLength = errors.New("missing or invalid Upload-Length header")
	ErrInvalidOffset       = errors.New("missing or invalid Upload-Offset header")
	ErrNotFound            = errors.New("upload not found")
	ErrFileLocked          = errors.New("file currently locked")
	ErrMismatchOffset      = errors.New("mismatched offset")
	ErrSizeExceeded        = errors.New("resource's size exceeded")
	ErrNotImplemented      = errors.New("feature not implemented")
	ErrUploadNotFinished   = errors.New("one of the partial uploads is not finished")
	ErrInvalidConcat       = errors.New("invalid Upload-Concat header")
	ErrModifyFinal         = errors.New("modifying a final upload is not allowed")
	ErrNeedLogin           = errors.New("need to login")
)

HTTP status codes sent in the response when the specific error is returned.

Functions

func LogEvent

func LogEvent(logger *log.Logger, eventName string, details ...string)

Types

type ConcaterDataStore

type ConcaterDataStore interface {
	// ConcatUploads concatenations the content from the provided partial uploads
	// and write the result in the destination upload which is specified by its
	// ID. The caller (usually the handler) must and will ensure that this
	// destination upload has been created before with enough space to hold all
	// partial uploads. The order, in which the partial uploads are supplied,
	// must be respected during concatenation.
	ConcatUploads(destination string, partialUploads []string) error
}

ConcaterDataStore is the interface required to be implemented if the Concatenation extension should be enabled. Only in this case, the handler will parse and respect the Upload-Concat header.

type Config

type Config struct {
	// DataStore implementation used to store and retrieve the single uploads.
	// The usage of this field is deprecated and should be avoided in favor of
	// StoreComposer.
	DataStore DataStore
	// StoreComposer points to the store composer from which the core data store
	// and optional dependencies should be taken. May only be nil if DataStore is
	// set.
	StoreComposer *StoreComposer
	// MaxSize defines how many bytes may be stored in one single upload. If its
	// value is is 0 or smaller no limit will be enforced.
	MaxSize int64
	// BasePath defines the URL path used for handling uploads, e.g. "/files/".
	// If no trailing slash is presented it will be added. You may specify an
	// absolute URL containing a scheme, e.g. "http://tus.io"
	BasePath string

	// NotifyCompleteUploads indicates whether sending notifications about
	// completed uploads using the CompleteUploads channel should be enabled.
	NotifyCompleteUploads bool
	// NotifyTerminatedUploads indicates whether sending notifications about
	// terminated uploads using the TerminatedUploads channel should be enabled.
	NotifyTerminatedUploads bool
	// Logger is the logger to use internally, mostly for printing requests.
	Logger *log.Logger
	// Respect the X-Forwarded-Host, X-Forwarded-Proto and Forwarded headers
	// potentially set by proxies when generating an absolute URL in the
	// response to POST requests.
	RespectForwardedHeaders bool
	// contains filtered or unexported fields
}

Config provides a way to configure the Handler depending on your needs.

type DataStore

type DataStore interface {
	// Create a new upload using the size as the file's length. The method must
	// return an unique id which is used to identify the upload. If no backend
	// (e.g. Riak) specifes the id you may want to use the uid package to
	// generate one. The properties Size and MetaData will be filled.
	NewUpload(info FileInfo) (id string, err error)
	// Write the chunk read from src into the file specified by the id at the
	// given offset. The handler will take care of validating the offset and
	// limiting the size of the src to not overflow the file's size. It may
	// return an os.ErrNotExist which will be interpreted as a 404 Not Found.
	// It will also lock resources while they are written to ensure only one
	// write happens per time.
	// The function call must return the number of bytes written.
	WriteChunk(id string, offset int64, src io.Reader) (int64, error)
	// Read the fileinformation used to validate the offset and respond to HEAD
	// requests. It may return an os.ErrNotExist which will be interpreted as a
	// 404 Not Found.
	GetInfo(id string) (FileInfo, error)

	BinPath(id string) (bin_path string)

	InfoPath(id string) (info_path string)
}

type FileInfo

type FileInfo struct {
	ID string
	// Total file size in bytes specified in the NewUpload call
	Size int64
	// Offset in bytes (zero-based)
	Offset   int64
	MetaData MetaData
	// Indicates that this is a partial upload which will later be used to form
	// a final upload by concatenation. Partial uploads should not be processed
	// when they are finished since they are only incomplete chunks of files.
	IsPartial bool
	// Indicates that this is a final upload
	IsFinal bool
	// If the upload is a final one (see IsFinal) this will be a non-empty
	// ordered slice containing the ids of the uploads of which the final upload
	// will consist after concatenation.
	PartialUploads []string
}

type FinisherDataStore

type FinisherDataStore interface {
	// FinishUpload executes additional operations for the finished upload which
	// is specified by its ID.
	FinishUpload(id string) error
}

FinisherDataStore is the interface which can be implemented by DataStores which need to do additional operations once an entire upload has been completed. These tasks may include but are not limited to freeing unused resources or notifying other services. For example, S3Store uses this interface for removing a temporary object.

type GetReaderDataStore

type GetReaderDataStore interface {
	// GetReader returns a reader which allows iterating of the content of an
	// upload specified by its ID. It should attempt to provide a reader even if
	// the upload has not been finished yet but it's not required.
	// If the returned reader also implements the io.Closer interface, the
	// Close() method will be invoked once everything has been read.
	// If the given upload could not be found, the error tusd.ErrNotFound should
	// be returned.
	GetReader(id string) (io.Reader, error)
}

GetReaderDataStore is the interface which must be implemented if handler should expose and support the GET route. It will allow clients to download the content of an upload regardless whether it's finished or not. Please, be aware that this feature is not part of the official tus specification. Instead it's a custom mechanism by tusd.

type Handler

type Handler struct {
	*UnroutedHandler
	http.Handler
}

Handler is a ready to use handler with routing (using pat)

func NewHandler

func NewHandler(config Config) (*Handler, error)

NewHandler creates a routed tus protocol handler. This is the simplest way to use tusd but may not be as configurable as you require. If you are integrating this into an existing app you may like to use tusd.NewUnroutedHandler instead. Using tusd.NewUnroutedHandler allows the tus handlers to be combined into your existing router (aka mux) directly. It also allows the GET and DELETE endpoints to be customized. These are not part of the protocol so can be changed depending on your needs.

type LockerDataStore

type LockerDataStore interface {
	// LockUpload attempts to obtain an exclusive lock for the upload specified
	// by its id.
	// If this operation fails because the resource is already locked, the
	// tusd.ErrFileLocked must be returned. If no error is returned, the attempt
	// is consider to be successful and the upload to be locked until UnlockUpload
	// is invoked for the same upload.
	LockUpload(id string) error
	// UnlockUpload releases an existing lock for the given upload.
	UnlockUpload(id string) error
}

LockerDataStore is the interface required for custom lock persisting mechanisms. Common ways to store this information is in memory, on disk or using an external service, such as ZooKeeper. When multiple processes are attempting to access an upload, whether it be by reading or writing, a synchronization mechanism is required to prevent data corruption, especially to ensure correct offset values and the proper order of chunks inside a single upload.

type MetaData

type MetaData map[string]string

type Metrics

type Metrics struct {
	// RequestTotal counts the number of incoming requests per method
	RequestsTotal map[string]*uint64
	// ErrorsTotal counts the number of returned errors by their message
	ErrorsTotal       map[string]*uint64
	BytesReceived     *uint64
	UploadsFinished   *uint64
	UploadsCreated    *uint64
	UploadsTerminated *uint64
}

Metrics provides numbers about the usage of the tusd handler. Since these may be accessed from multiple goroutines, it is necessary to read and modify them atomically using the functions exposed in the sync/atomic package, such as atomic.LoadUint64. In addition the maps must not be modified to prevent data races.

type StoreComposer

type StoreComposer struct {
	Core DataStore

	UsesTerminater bool
	Terminater     TerminaterDataStore
	UsesFinisher   bool
	Finisher       FinisherDataStore
	UsesLocker     bool
	Locker         LockerDataStore
	UsesGetReader  bool
	GetReader      GetReaderDataStore
	UsesConcater   bool
	Concater       ConcaterDataStore
}

StoreComposer represents a composable data store. It consists of the core data store and optional extensions. Please consult the package's overview for a more detailed introduction in how to use this structure.

func NewStoreComposer

func NewStoreComposer() *StoreComposer

NewStoreComposer creates a new and empty store composer.

Example
package main

import (
	"github.com/nagae-memooff/tusd"
	"github.com/nagae-memooff/tusd/filestore"
	"github.com/nagae-memooff/tusd/limitedstore"
	"github.com/nagae-memooff/tusd/memorylocker"
)

func main() {
	composer := tusd.NewStoreComposer()

	fs := filestore.New("./data")
	fs.UseIn(composer)

	ml := memorylocker.New()
	ml.UseIn(composer)

	ls := limitedstore.New(1024*1024*1024, composer.Core, composer.Terminater)
	ls.UseIn(composer)

	config := tusd.Config{
		StoreComposer: composer,
	}

	_, _ = tusd.NewHandler(config)
}
Output:

func (*StoreComposer) Capabilities

func (store *StoreComposer) Capabilities() string

Capabilities returns a string representing the provided extensions in a human-readable format meant for debugging.

func (*StoreComposer) UseConcater

func (store *StoreComposer) UseConcater(ext ConcaterDataStore)

func (*StoreComposer) UseCore

func (store *StoreComposer) UseCore(core DataStore)

UseCore will set the used core data store. If the argument is nil, the property will be unset.

func (*StoreComposer) UseFinisher

func (store *StoreComposer) UseFinisher(ext FinisherDataStore)

func (*StoreComposer) UseGetReader

func (store *StoreComposer) UseGetReader(ext GetReaderDataStore)

func (*StoreComposer) UseLocker

func (store *StoreComposer) UseLocker(ext LockerDataStore)

func (*StoreComposer) UseTerminater

func (store *StoreComposer) UseTerminater(ext TerminaterDataStore)

type TerminaterDataStore

type TerminaterDataStore interface {
	// Terminate an upload so any further requests to the resource, both reading
	// and writing, must return os.ErrNotExist or similar.
	Terminate(id string) error
}

TerminaterDataStore is the interface which must be implemented by DataStores if they want to receive DELETE requests using the Handler. If this interface is not implemented, no request handler for this method is attached.

type UnroutedHandler

type UnroutedHandler struct {

	// CompleteUploads is used to send notifications whenever an upload is
	// completed by a user. The FileInfo will contain information about this
	// upload after it is completed. Sending to this channel will only
	// happen if the NotifyCompleteUploads field is set to true in the Config
	// structure. Notifications will also be sent for completions using the
	// Concatenation extension.
	CompleteUploads chan FileInfo
	// TerminatedUploads is used to send notifications whenever an upload is
	// terminated by a user. The FileInfo will contain information about this
	// upload gathered before the termination. Sending to this channel will only
	// happen if the NotifyTerminatedUploads field is set to true in the Config
	// structure.
	TerminatedUploads chan FileInfo
	// Metrics provides numbers of the usage for this handler.
	Metrics Metrics
	// contains filtered or unexported fields
}

UnroutedHandler exposes methods to handle requests as part of the tus protocol, such as PostFile, HeadFile, PatchFile and DelFile. In addition the GetFile method is provided which is, however, not part of the specification.

func NewUnroutedHandler

func NewUnroutedHandler(config Config) (*UnroutedHandler, error)

NewUnroutedHandler creates a new handler without routing using the given configuration. It exposes the http handlers which need to be combined with a router (aka mux) of your choice. If you are looking for preconfigured handler see NewHandler.

func (*UnroutedHandler) DelFile

func (handler *UnroutedHandler) DelFile(w http.ResponseWriter, r *http.Request)

DelFile terminates an upload permanently.

func (*UnroutedHandler) GetFile

func (handler *UnroutedHandler) GetFile(w http.ResponseWriter, r *http.Request)

GetFile handles requests to download a file using a GET request. This is not part of the specification.

func (*UnroutedHandler) HeadFile

func (handler *UnroutedHandler) HeadFile(w http.ResponseWriter, r *http.Request)

HeadFile returns the length and offset for the HEAD request

func (*UnroutedHandler) Middleware

func (handler *UnroutedHandler) Middleware(h http.Handler) http.Handler

Middleware checks various aspects of the request and ensures that it conforms with the spec. Also handles method overriding for clients which cannot make PATCH AND DELETE requests. If you are using the tusd handlers directly you will need to wrap at least the POST and PATCH endpoints in this middleware.

func (*UnroutedHandler) PatchFile

func (handler *UnroutedHandler) PatchFile(w http.ResponseWriter, r *http.Request)

PatchFile adds a chunk to an upload. Only allowed enough space is left.

func (*UnroutedHandler) PostFile

func (handler *UnroutedHandler) PostFile(w http.ResponseWriter, r *http.Request)

PostFile creates a new file upload using the datastore after validating the length and parsing the metadata.

Directories

Path Synopsis
cmd
Package consullocker provides a locking mechanism using a Consul server.
Package consullocker provides a locking mechanism using a Consul server.
Package filestore provide a storage backend based on the local file system.
Package filestore provide a storage backend based on the local file system.
Package limitedstore provides a storage with a limited space.
Package limitedstore provides a storage with a limited space.
Package memorylocker provides an in-memory locking mechanism.
Package memorylocker provides an in-memory locking mechanism.
package prometheuscollector allows to expose metrics for Prometheus.
package prometheuscollector allows to expose metrics for Prometheus.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL