siva

package module
v1.8.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 26, 2019 License: MIT Imports: 12 Imported by: 10

README

siva format GoDoc Build Status Build status codebeat badge

siva stand for seekable indexed verifiable archiver

siva is archive format very similar to tar or zip, focused on allowing: constant-time random file access, seekable access to the contained files and concatenable archive files

siva

The library implements a very similar API to the go tar package, allowing full control over and low level access to the contained files.

Installation

The recommended way to install siva

go get -u gopkg.in/src-d/go-siva.v1/...

Example

Creating a siva file:

// Create a buffer to write our archive to.
buf := new(bytes.Buffer)

// Create a new siva archive.
w := siva.NewWriter(buf)

// Add some files to the archive.
var files = []struct {
    Name, Body string
}{
    {"readme.txt", "This archive contains some text files."},
    {"gopher.txt", "Gopher names:\nGeorge\nGeoffrey\nGonzo"},
    {"todo.txt", "Get animal handling license."},
}
for _, file := range files {
    hdr := &siva.Header{
        Name:    file.Name,
        Mode:    0600,
        ModTime: time.Now(),
    }
    if err := w.WriteHeader(hdr); err != nil {
        log.Fatalln(err)
    }
    if _, err := w.Write([]byte(file.Body)); err != nil {
        log.Fatalln(err)
    }
}
// Make sure to check the error on Close.
if err := w.Close(); err != nil {
    log.Fatalln(err)
}

Reading from a siva file:

// Open the siva archive for reading.
file := bytes.NewReader(buf.Bytes())
r := siva.NewReader(file)

// Get all the files in the siva file.
i, err := r.Index()
if err != nil {
    log.Fatalln(err)
}

// Iterate through the files in the archive.
for _, e := range i {
    content, err := r.Get(e)
    if err != nil {
        log.Fatalln(err)
    }
    fmt.Printf("Contents of %s:\n", e.Name)
    if _, err := io.Copy(os.Stdout, content); err != nil {
        log.Fatalln(err)
    }
    fmt.Println()
}

Command-line interface

siva cli interface, is a convenient command that helps you to creates and manipulates siva files.

Output from: ./siva --help:

Usage:
  siva [OPTIONS] <command>

Help Options:
  -h, --help  Show this help message

Available commands:
  list     List the items contained on a file.
  pack     Create a new archive containing the specified items.
  unpack   Extract to disk from the archive.
  version  Show the version information.

Other comments

  • The Index Signature is specified as a sequence of 3 bytes. Go uses byte as an alias for uint8.
  • File Mode in an Index entry, see issue.
  • This implementation left in the client of the library side the task of check the integrity of the file contents. It just checks for the Index integrity.

License

MIT, see LICENSE

Documentation

Overview

Example 
// Create a buffer to write our archive to.
buf := new(bytes.Buffer)

// Create a new siva archive.
w := siva.NewWriter(buf)

// Add some files to the archive.
var files = []struct {
	Name, Body string
}{
	{"gopher.txt", "Gopher names:\nGeorge\nGeoffrey\nGonzo"},
	{"readme.txt", "This archive contains some text files."},
	{"todo.txt", "Get animal handling license."},
}
for _, file := range files {
	hdr := &siva.Header{
		Name:    file.Name,
		Mode:    0600,
		ModTime: time.Now(),
	}
	if err := w.WriteHeader(hdr); err != nil {
		log.Fatalln(err)
	}
	if _, err := w.Write([]byte(file.Body)); err != nil {
		log.Fatalln(err)
	}
}
// Make sure to check the error on Close.
if err := w.Close(); err != nil {
	log.Fatalln(err)
}

// Open the siva archive for reading.
file := bytes.NewReader(buf.Bytes())
r := siva.NewReader(file)

// Get all files in the siva file.
i, err := r.Index()
if err != nil {
	log.Fatalln(err)
}

// Iterate through the files in the archive.
for _, e := range i {
	content, err := r.Get(e)
	if err != nil {
		log.Fatalln(err)
	}
	fmt.Printf("Contents of %s:\n", e.Name)
	if _, err := io.Copy(os.Stdout, content); err != nil {
		log.Fatalln(err)
	}
	fmt.Println()
}

Output:

Contents of gopher.txt:
Gopher names:
George
Geoffrey
Gonzo
Contents of readme.txt:
This archive contains some text files.
Contents of todo.txt:
Get animal handling license.

Index

Examples

Constants

View Source 
const (
	IndexVersion uint8 = 1
)

Variables

View Source 
var (
	IndexSignature = []byte{'I', 'B', 'A'}

	ErrInvalidIndexEntry       = errors.New("invalid index entry")
	ErrInvalidSignature        = errors.New("invalid signature")
	ErrEmptyIndex              = errors.New("empty index")
	ErrUnsupportedIndexVersion = errors.New("unsupported index version")
	ErrCRC32Missmatch          = errors.New("crc32 mismatch")
)
View Source 
var (
	ErrPendingContent   = errors.New("entry wasn't fully read")
	ErrInvalidCheckshum = errors.New("invalid checksum")
	ErrInvalidReaderAt  = errors.New("reader provided dosen't implement ReaderAt interface")
)
View Source 
var (
	ErrMissingHeader = errors.New("WriteHeader was not called, or already flushed")
	ErrClosedWriter  = errors.New("Writer is closed")
)

Functions

func ToSafePath added in v1.2.0 

func ToSafePath(p string) string

ToSafePath transforms a filesystem path to one that is safe to use as a relative path on the native filesystem:

- Removes drive and network share on Windows. - Does regular clean up (removing `/./` parts). - Removes any leading `../`. - Removes leading `/`.

This is a convenience function to implement siva file extractors that are not vulnerable to zip slip and similar vulnerabilities. However, for Windows absolute paths (with drive or network share) it does not give consistent results across platforms.

If your application relies on using absolute paths, you should not use this and you are encouraged to do your own validation and normalization.

Types

type Flag 

type Flag uint32

Flag carries a bitwise OR of flag values affecting an archive entry. 

const (
	// FlagDeleted is set to indicate when a file is deleted.
	FlagDeleted Flag = 1 << iota
)
type Header struct {
	// Name is an arbitrary UTF-8 string identifying a file in the archive. Note
	// that this might or might not be a POSIX-compliant path.
	//
	// Security note: Users should be careful when using name as a file path
	// (e.g. to extract an archive) since it can contain relative paths and be
	// vulnerable to Zip Slip (https://snyk.io/research/zip-slip-vulnerability)
	// or other potentially dangerous values such as absolute paths, network
	// drive addresses, etc.
	Name    string
	ModTime time.Time
	Mode    os.FileMode
	Flags   Flag
}

Header contains the meta information from a file

type Index 

type Index []*IndexEntry

Index contains all the files on a siva file, including duplicate files or even does flagged as deleted.

func (*Index) Filter 

func (i *Index) Filter() Index

Filter returns a filtered version of the current Index removing duplicates keeping the latest versions and filtering all the deleted files

func (Index) Find 

func (i Index) Find(name string) *IndexEntry

Find returns the first IndexEntry with the given name, if any

func (Index) Glob added in v1.1.0 

func (i Index) Glob(pattern string) ([]*IndexEntry, error)

Glob returns all index entries whose name matches pattern or nil if there is no matching entry. The syntax of patterns is the same as in filepath.Match.

func (Index) Len 

func (s Index) Len() int

Len implements sort.Interface.

func (Index) Less 

func (s Index) Less(i, j int) bool

Less implements sort.Interface.

func (*Index) ReadFrom 

func (i *Index) ReadFrom(r io.ReadSeeker, endBlock uint64) error

ReadFrom reads an Index from a given reader, the position where the current block ends is required since we are reading the index from the end of the file

func (Index) Swap 

func (s Index) Swap(i, j int)

Swap implements sort.Interface.

func (*Index) ToSafePaths added in v1.2.0 

func (i *Index) ToSafePaths() Index

ToSafePaths creates a new index where all entry names are transformed to safe paths using the top-level `ToSafePath` function. If you are using siva to extract files to the file-system, you should either use this function or perform your own validation and normalization.

func (*Index) WriteTo 

func (i *Index) WriteTo(w io.Writer) error

WriteTo writes the Index to a io.Writer

type IndexEntry 

type IndexEntry struct {
	Header
	Start uint64
	Size  uint64
	CRC32 uint32
	// contains filtered or unexported fields
}

func (*IndexEntry) ReadFrom 

func (e *IndexEntry) ReadFrom(r io.Reader) error

ReadFrom reads a IndexEntry entry from an io.Reader

func (*IndexEntry) WriteTo 

func (e *IndexEntry) WriteTo(w io.Writer) error

WriteTo writes the IndexEntry to an io.Writer

type IndexFooter 

type IndexFooter struct {
	EntryCount uint32
	IndexSize  uint64
	BlockSize  uint64
	CRC32      uint32
}

func (*IndexFooter) ReadFrom 

func (f *IndexFooter) ReadFrom(r io.Reader) error

ReadFrom reads a IndexFooter entry from an io.Reader

func (*IndexFooter) WriteTo 

func (f *IndexFooter) WriteTo(w io.Writer) error

WriteTo writes the IndexFooter to an io.Writer

type IndexReadError added in v1.1.2 

type IndexReadError struct {
	Err error
}

func (*IndexReadError) Error added in v1.1.2 

func (e *IndexReadError) Error() string

type IndexWriteError added in v1.1.2 

type IndexWriteError struct {
	Err error
}

func (*IndexWriteError) Error added in v1.1.2 

func (e *IndexWriteError) Error() string

type OrderedIndex added in v1.3.0 

type OrderedIndex Index

OrderedIndex is a specialized index lexicographically ordered. It has methods to add or delete IndexEntries and maintain its order. Also has as faster Find method.

func (OrderedIndex) Add added in v1.3.0 

func (o OrderedIndex) Add(e *IndexEntry) OrderedIndex

Add returns an updated index with the new IndexEntry.

func (OrderedIndex) Delete added in v1.3.0 

func (o OrderedIndex) Delete(path string) OrderedIndex

Delete returns an updated index with the IndexEntry for the path deleted.

func (OrderedIndex) Find added in v1.3.0 

func (o OrderedIndex) Find(path string) *IndexEntry

Find returns the IndexEntry for a path or nil. This version is faster than Index.Find.

func (OrderedIndex) Glob added in v1.7.0 

func (o OrderedIndex) Glob(pattern string) ([]*IndexEntry, error)

Glob returns all index entries whose name matches pattern or nil if there is no matching entry. The syntax of patterns is the same as in filepath.Match.

func (OrderedIndex) Len added in v1.3.0 

func (s OrderedIndex) Len() int

Len implements sort.Interface.

func (OrderedIndex) Less added in v1.3.0 

func (s OrderedIndex) Less(i, j int) bool

Less implements sort.Interface.

func (OrderedIndex) Pos added in v1.3.0 

func (o OrderedIndex) Pos(path string) int

Pos gets the position of the file in the index or where it should be inserted if it's not already there.

func (OrderedIndex) Sort added in v1.3.0 

func (o OrderedIndex) Sort()

Sort orders the index lexicographically.

func (OrderedIndex) Swap added in v1.3.0 

func (s OrderedIndex) Swap(i, j int)

Swap implements sort.Interface.

func (OrderedIndex) Update added in v1.3.0 

func (o OrderedIndex) Update(e *IndexEntry) OrderedIndex

Update adds or deletes an IndexEntry to the index depending on the FlagDeleted value.

type ReadWriter added in v1.1.0 

type ReadWriter struct {
	// contains filtered or unexported fields
}

ReadWriter can read and write to the same siva file. It is not thread-safe.

func NewReaderWriter added in v1.1.0 

func NewReaderWriter(rw io.ReadWriteSeeker) (*ReadWriter, error)

func (ReadWriter) Close added in v1.1.0 

func (w ReadWriter) Close() error

Close closes the siva archive, writing the Index footer to the current writer.

func (ReadWriter) Flush added in v1.1.0 

func (w ReadWriter) Flush() error

Flush finishes writing the current file (optional)

func (ReadWriter) Get added in v1.1.0 

func (r ReadWriter) Get(e *IndexEntry) (*io.SectionReader, error)

Get returns a new io.SectionReader allowing concurrent read access to the content of the read

func (ReadWriter) Index added in v1.1.0 

func (r ReadWriter) Index() (Index, error)

Index reads the index of the siva file from the provided reader

func (ReadWriter) Read added in v1.1.0 

func (r ReadWriter) Read(p []byte) (n int, err error)

Read reads up to len(p) bytes, starting at the current position set by Seek and ending in the end of the content, retuning a io.EOF when its reached

func (ReadWriter) Seek added in v1.1.0 

func (r ReadWriter) Seek(e *IndexEntry) (int64, error)

Seek seek the internal reader to the starting position of the content for the given IndexEntry

func (ReadWriter) Write added in v1.1.0 

func (w ReadWriter) Write(b []byte) (int, error)

Write writes to the current entry in the siva archive, WriteHeader should called before, if not returns ErrMissingHeader

func (ReadWriter) WriteHeader added in v1.1.0 

func (w ReadWriter) WriteHeader(h *Header) error

WriteHeader writes hdr and prepares to accept the file's contents.

type Reader 

type Reader interface {
	io.Reader
	Seek(e *IndexEntry) (int64, error)
	Index() (Index, error)
	Get(e *IndexEntry) (*io.SectionReader, error)
}

A Reader provides random access to the contents of a siva archive.

func NewReader 

func NewReader(r io.ReadSeeker) Reader

NewReader creates a new Reader reading from r, reader requires be seekable and optionally should implement io.ReaderAt to make usage of the Get method

func NewReaderWithOffset added in v1.5.0 

func NewReaderWithOffset(r io.ReadSeeker, o uint64) Reader

NewReaderWithOffset creates a new Reader giving the position of the index. This is useful to open siva files that are being written or reading an old index.

type Writer 

type Writer interface {
	io.Writer
	io.Closer
	WriteHeader(h *Header) error
	Flush() error
}

A Writer provides sequential writing of a siva archive

func NewWriter 

func NewWriter(w io.Writer) Writer

NewWriter creates a new Writer writing to w.

Source Files

View all Source files

Directories

Path Synopsis
cmd
Package cmd, is a convinient command that helps you to creates and manipulate .siva files.
 Package cmd, is a convinient command that helps you to creates and manipulate .siva files.
siva
siva/impl

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.