README
¶
filestream 
A system for efficiently streaming bundles of files over a connection.
Features:
- Compression - supports gzip and lz4
- Chunked - can stream files without knowing the size in advance (e.g. generated files/downloads)
When would I use this?
This package was developed based on poor experiences with docker's usage of tar as a part of their API. The tar file format requires lengths to be written before the contents of the file. This created problems, as I had to buffer huge files in memory as they were collected over HTTP or generated on the fly.
I created this for the purpose of streaming files to test machines in a CI system. This is also potentially useful for data import/export.
Try it out on the command line
A command line interface has been created as a debugging tool and a demo. To install it, run:
go get github.com/jaddr2line/filestream/cmd/filestream/...
To encode files to stdout:
# Encode directories "example" and "test" to stdout
filestream example test | somecommand
# Encode to stdout with gzip compression
filestream -z gzip -l 9 example | somecommand
Decoding from stdin:
# Decode stream to current directory.
curl https://example.com/something | filestream -d
# Decode stream relative to directory example
curl https://example.com/something | filestream -d -C example
Listing files in stream:
curl https://example.com/something | filestream -t
The tool can also read/write the stream to different places:
# Store directory example to a file called "output.dat"
filestream example -s output.dat
# Decode from a file
filestream -d -s output.dat
# Decode data from an HTTP GET request
filestream -d -s https://example.com/something
Documentation
¶
Overview ¶
Package filestream implements a mechanism for streaming bundles of files over a single data stream. The data uses a chunked encoding, so data can be written without knowing the length of the data ahead of time.
Example (Minimal) ¶
package main
import (
"bytes"
"fmt"
"io/ioutil"
"log"
"github.com/jaddr2line/filestream"
)
func main() {
// Write some files to a stream.
var buf bytes.Buffer
w, err := filestream.NewWriter(&buf, filestream.StreamOptions{})
if err != nil {
log.Fatal(err)
}
var files = []struct {
Name, Body string
}{
{"hello.txt", "Hello World!"},
{"smile.txt", "☺"},
}
for _, file := range files {
fw, err := w.File(file.Name, filestream.FileOptions{})
if err != nil {
log.Fatal(err)
}
_, err = fw.Write([]byte(file.Body))
if err != nil {
log.Fatal(err)
}
err = fw.Close()
if err != nil {
log.Fatal(err)
}
}
if err := w.Close(); err != nil {
log.Fatal(err)
}
// Read files back out of stream.
r, err := filestream.NewReader(&buf)
if err != nil {
log.Fatal(err)
}
for r.Next() {
f := r.File()
dat, err := ioutil.ReadAll(f)
if err != nil {
log.Fatal(err)
}
fmt.Printf("File %q: %s\n", f.Path(), string(dat))
}
// check for error in streaming
if err = r.Err(); err != nil {
log.Fatal(err)
}
}
Output: File "hello.txt": Hello World! File "smile.txt": ☺
Index ¶
Examples ¶
Constants ¶
This section is empty.
Variables ¶
var ErrWriteInterrupted = errors.New("write interrupted")
ErrWriteInterrupted indicates that a close operation interrupted a file stream and may have resulted in a corrupted stream.
Functions ¶
func DecodeFiles ¶
func DecodeFiles(src *Reader, opts DecodeOptions) error
DecodeFiles decodes a filestream to the filesystem.
func EncodeFiles ¶
func EncodeFiles(dst *Writer, path string, opts EncodeOptions) error
EncodeFiles encodes files from a path into a stream.
Types ¶
type DecodeOptions ¶
type DecodeOptions struct {
// Base is the base directory from which relative paths will be resolved.
Base string
// PreservePermissions is whether or not to preserve the perimission codes from the stream.
PreservePermissions bool
// PreserveUser is whether or not to preserve the owning user info from the stream.
PreserveUser bool
// PreserveGroup is whether or not to preserve the owning group info from the stream.
PreserveGroup bool
// DefaultOpts are the default file options.
// If any given option is not being preserved, the corresponding default will be applied to everything.
// If any given option is being preserved, the corresponding default will be applied where not present in the stream.
// Defaults to 640, current user, current group.
DefaultOpts FileOptions
}
DecodeOptions is a set of options for decoding files from a stream into the filesystem.
type EncodeOptions ¶
type EncodeOptions struct {
// Base path which should be used for stream.
// Paths within this archive will be relative to this path.
// By default, it will be set to the top level path provided to EncodeFiles.
// Setting base to "/" will cause the encoding to use absolute paths.
Base string
// IncludePermissions is whether or not to include permission codes from the files.
// Setting this to true will cause permissions to be preserved in the stream.
// This does not control whether owning group/user is sent.
IncludePermissions bool
// IncludeUser is whether or not to include the owning username in the stream.
// Setting this to true will cause the system to look up the username of the owning user.
// Failed username lookups will result in errors.
// This is supported on Linux and Darwin, and may be a no-op on other systems.
IncludeUser bool
// IncludeGroup is whether or not to include the owning group name in the stream.
// Setting this to true will cause the system to look up the group name of the owning group.
// Failed group name lookups will result in errors.
// This is supported on Linux and Darwin, and may be a no-op on other systems.
IncludeGroup bool
}
EncodeOptions are a set of options for encoding files from the filesystem into a filestream. They may be used with EncodeFiles.
type FileOptions ¶
type FileOptions struct {
// Permissions are the unix permission code of the file.
// If the permissions component is 000, this will be converted to sane defaults.
// Optional.
Permissions os.FileMode
// User is the username of the owner.
// Optional.
User string
// Group is the groupname of the owning group.
// Optional.
Group string
}
FileOptions are the set of options which can be applied to a file stream.
type FileReader ¶
type FileReader struct {
// contains filtered or unexported fields
}
FileReader is a reader of a single file in a stream.
func (*FileReader) IsDir ¶
func (fr *FileReader) IsDir() bool
IsDir returns whether the entry is a directory.
func (*FileReader) Opts ¶
func (fr *FileReader) Opts() FileOptions
Opts are the options of the file.
type Reader ¶
type Reader struct {
// contains filtered or unexported fields
}
Reader is a filestream reader.
func (*Reader) Err ¶
Err returns the error from the last call to .Next(). If there is no error, nil will be returned.
func (*Reader) File ¶
func (r *Reader) File() *FileReader
File returns the currently selected file. File must be read completely before calling Next again. Directories do not need to be read, and have no body.
type StreamOptions ¶
type StreamOptions struct {
// Compression is the compression algorithm to use in transit.
// This package supports "gzip" and "lz4".
// Defaults to no compresion.
Compression string
// CompressionLevel is the level of compresion to use.
// Uses a sane default if omitted.
CompressionLevel int
}
StreamOptions are configuration options for a stream.
type Writer ¶
type Writer struct {
// contains filtered or unexported fields
}
Writer is an encoder for a filestream.
func NewWriter ¶
func NewWriter(dst io.Writer, opts StreamOptions) (*Writer, error)
NewWriter creates a new file stream writer.
func (*Writer) Close ¶
Close ends the stream. If a file stream is incomplete, generates a corrupted stream and returns ErrWriteInterrupted.
func (*Writer) Directory ¶
func (w *Writer) Directory(path string, opts FileOptions) error
Directory creates a directory in the stream with the given path.
func (*Writer) File ¶
func (w *Writer) File(path string, opts FileOptions) (io.WriteCloser, error)
File creates a new file stream at the given path. The file must be closed in order to be committed to the stream. Attempting to call File or Directory before closing a file may result in an error.
Source Files
Directories