tar-split: github.com/vbatts/tar-split/tar/storage Index | Files

package storage

import "github.com/vbatts/tar-split/tar/storage"

Package storage is for metadata of a tar archive.

Packing and unpacking the Entries of the stream. The types of streams are either segments of raw bytes (for the raw headers and various padding) and for an entry marking a file payload.

The raw bytes are stored precisely in the packed (marshalled) Entry, whereas the file payload marker include the name of the file, size, and crc64 checksum (for basic file integrity).


Package Files

doc.go entry.go getter.go packer.go


var CRCTable = crc64.MakeTable(crc64.ISO)

CRCTable is the default table used for crc64 sum calculations

var ErrDuplicatePath = errors.New("duplicates of file paths not supported")

ErrDuplicatePath occurs when a tar archive has more than one entry for the same file path

type Entries Uses

type Entries []Entry

Entries is for sorting by Position

func (Entries) Len Uses

func (e Entries) Len() int

func (Entries) Less Uses

func (e Entries) Less(i, j int) bool

func (Entries) Swap Uses

func (e Entries) Swap(i, j int)

type Entry Uses

type Entry struct {
    Type     Type   `json:"type"`
    Name     string `json:"name,omitempty"`
    NameRaw  []byte `json:"name_raw,omitempty"`
    Size     int64  `json:"size,omitempty"`
    Payload  []byte `json:"payload"` // SegmentType stores payload here; FileType stores crc64 checksum here;
    Position int    `json:"position"`

Entry is the structure for packing and unpacking the information read from the Tar archive.

FileType Payload checksum is using `hash/crc64` for basic file integrity, _not_ for cryptography. From http://www.backplane.com/matt/crc64.html, CRC32 has almost 40,000 collisions in a sample of 18.2 million, CRC64 had none.

func (*Entry) GetName Uses

func (e *Entry) GetName() string

GetName returns the string for the entry's name, regardless of the field stored in

func (*Entry) GetNameBytes Uses

func (e *Entry) GetNameBytes() []byte

GetNameBytes returns the bytes for the entry's name, regardless of the field stored in

func (*Entry) SetName Uses

func (e *Entry) SetName(name string)

SetName will check name for valid UTF-8 string, and set the appropriate field. See https://github.com/vbatts/tar-split/issues/17

func (*Entry) SetNameBytes Uses

func (e *Entry) SetNameBytes(name []byte)

SetNameBytes will check name for valid UTF-8 string, and set the appropriate field

type FileGetPutter Uses

type FileGetPutter interface {

FileGetPutter is the interface that groups both Getting and Putting file payloads.

func NewBufferFileGetPutter Uses

func NewBufferFileGetPutter() FileGetPutter

NewBufferFileGetPutter is a simple in-memory FileGetPutter

Implication is this is memory intensive... Probably best for testing or light weight cases.

type FileGetter Uses

type FileGetter interface {
    // Get returns a stream for the provided file path
    Get(filename string) (output io.ReadCloser, err error)

FileGetter is the interface for getting a stream of a file payload, addressed by name/filename. Presumably, the names will be scoped to relative file paths.

func NewPathFileGetter Uses

func NewPathFileGetter(relpath string) FileGetter

NewPathFileGetter returns a FileGetter that is for files relative to path relpath.

type FilePutter Uses

type FilePutter interface {
    // Put returns the size of the stream received, and the crc64 checksum for
    // the provided stream
    Put(filename string, input io.Reader) (size int64, checksum []byte, err error)

FilePutter is the interface for storing a stream of a file payload, addressed by name/filename.

func NewDiscardFilePutter Uses

func NewDiscardFilePutter() FilePutter

NewDiscardFilePutter is a bit bucket FilePutter

type Packer Uses

type Packer interface {
    // AddEntry packs the Entry and returns its position
    AddEntry(e Entry) (int, error)

Packer describes the methods to pack Entries to a storage destination

func NewJSONPacker Uses

func NewJSONPacker(w io.Writer) Packer

NewJSONPacker provides a Packer that writes each Entry (SegmentType and FileType) as a json document.

The Entries are delimited by new line.

type Type Uses

type Type int

Type of Entry

const (
    // FileType represents a file payload from the tar stream.
    // This will be used to map to relative paths on disk. Only Size > 0 will get
    // read into a resulting output stream (due to hardlinks).
    FileType Type = 1 + iota
    // SegmentType represents a raw bytes segment from the archive stream. These raw
    // byte segments consist of the raw headers and various padding.
    // Its payload is to be marshalled base64 encoded.

type Unpacker Uses

type Unpacker interface {
    // Next returns the next Entry being unpacked, or error, until io.EOF
    Next() (*Entry, error)

Unpacker describes the methods to read Entries from a source

func NewJSONUnpacker Uses

func NewJSONUnpacker(r io.Reader) Unpacker

NewJSONUnpacker provides an Unpacker that reads Entries (SegmentType and FileType) as a json document.

Each Entry read are expected to be delimited by new line.

Package storage imports 8 packages (graph) and is imported by 851 packages. Updated 2018-08-03. Refresh now. Tools for package owners.