pachyderm: github.com/pachyderm/pachyderm/src/server/pkg/storage/fileset Index | Files | Directories

package fileset

import "github.com/pachyderm/pachyderm/src/server/pkg/storage/fileset"

Index

Package Files

fileset.go merge.go option.go reader.go storage.go util.go writer.go

type FileSet Uses

type FileSet struct {
    // contains filtered or unexported fields
}

FileSet is a set of files. This may be a full filesystem or a subfilesystem (e.g. datum / datum set / shard).

func (*FileSet) Close Uses

func (f *FileSet) Close() error

Close closes the file set.

func (*FileSet) Delete Uses

func (f *FileSet) Delete(path string)

Delete deletes a file from the file set. (bryce) might need to delete ancestor directories in certain cases.

func (*FileSet) StartTag Uses

func (f *FileSet) StartTag(tag string)

StartTag starts a tag for the next set of files.

func (*FileSet) Write Uses

func (f *FileSet) Write(data []byte) (int, error)

Write writes to the current file in the tar stream.

func (*FileSet) WriteHeader Uses

func (f *FileSet) WriteHeader(hdr *tar.Header) error

WriteHeader writes a tar header and prepares to accept the file's contents. (bryce) should we prevent directories from being written here?

type Option Uses

type Option func(f *FileSet)

Option configures a file set.

func WithMemThreshold Uses

func WithMemThreshold(threshold int64) Option

WithMemThreshold sets the memory threshold of the file set.

func WithParent Uses

func WithParent(parentName string) Option

WithParent sets the parent of the file set.

func WithRoot Uses

func WithRoot(root string) Option

WithRoot sets the root path of the file set.

type Reader Uses

type Reader struct {
    // contains filtered or unexported fields
}

Reader reads the serialized format of a fileset.

func (*Reader) Close Uses

func (r *Reader) Close() error

Close closes the reader.

func (*Reader) Next Uses

func (r *Reader) Next() (*index.Header, error)

Next returns the next header, and prepares the file's content for reading.

func (*Reader) Peek Uses

func (r *Reader) Peek() (*index.Header, error)

Peek returns the next header without progressing the reader.

func (*Reader) PeekTag Uses

func (r *Reader) PeekTag() (*index.Tag, error)

PeekTag returns the next tag without progressing the reader.

func (*Reader) Read Uses

func (r *Reader) Read(data []byte) (int, error)

Read reads from the current file in the tar stream.

type Storage Uses

type Storage struct {
    // contains filtered or unexported fields
}

Storage is the abstraction that manages fileset storage.

func NewStorage Uses

func NewStorage(objC obj.Client, chunks *chunk.Storage) *Storage

NewStorage creates a new Storage.

func (*Storage) New Uses

func (s *Storage) New(ctx context.Context, name string, opts ...Option) *FileSet

New creates a new in-memory fileset.

func (*Storage) NewIndexReader Uses

func (s *Storage) NewIndexReader(ctx context.Context, fileSet, idxPrefix string) *index.Reader

NewIndexReader creates a new index.Reader.

func (*Storage) NewIndexWriter Uses

func (s *Storage) NewIndexWriter(ctx context.Context, fileSet string) *index.Writer

NewIndexWriter creates a new index.Writer.

func (*Storage) NewReader Uses

func (s *Storage) NewReader(ctx context.Context, fileSet, idxPrefix string) *Reader

NewReader creates a new Reader.

func (*Storage) NewWriter Uses

func (s *Storage) NewWriter(ctx context.Context, fileSet string) *Writer

NewWriter creates a new Writer.

type Writer Uses

type Writer struct {
    // contains filtered or unexported fields
}

Writer writes the serialized format of a fileset. The serialized format of a fileset consists of indexes and content which are both realized as compressed tar stream chunks.

func (*Writer) Close Uses

func (w *Writer) Close() error

Close closes the writer. (bryce) is not closing the tar writer the right choice here? We cannot include it in the last file's range because it will effect the hash. Our indexing is what will exit the reading of the tar stream, not the end of tar entry (two empty 512 bytes).

func (*Writer) CopyFiles Uses

func (w *Writer) CopyFiles(r *Reader, pathBound ...string) error

CopyFiles does a cheap copy of files from a reader to a writer. (bryce) need to handle delete operations.

func (*Writer) CopyTags Uses

func (w *Writer) CopyTags(r *Reader, tagBound ...string) error

CopyTags does a cheap copy of tagged file data from a reader to a writer.

func (*Writer) StartTag Uses

func (w *Writer) StartTag(id string)

StartTag starts a tag for the next set of bytes (used for the reverse index, mapping file output to datums).

func (*Writer) Write Uses

func (w *Writer) Write(data []byte) (int, error)

Write writes to the current file in the tar stream.

func (*Writer) WriteHeader Uses

func (w *Writer) WriteHeader(hdr *index.Header) error

WriteHeader writes a tar header and prepares to accept the file's contents.

Directories

PathSynopsis
index
tarPackage tar implements access to tar archives.

Package fileset imports 14 packages (graph). Updated 2019-10-16. Refresh now. Tools for package owners.