sophie: github.com/daviddengcn/sophie Index | Files | Directories

package sophie

import "github.com/daviddengcn/sophie"

Package sophie provides an raw mechanism for serializing data.

It aims at more efficiency than other serialization methods because of the following reasons:

* Maximum of reusing objects, allocation and GC are avoided * No reflections

Since the serialization is flexible, one can also make some trade-offs between efficiency and convinience. E.g., if the data structure may be changed in the future, in the ReadFrom/WriteTo, God codec can be used to provide future compatibility.

Sub packages:

mr  MapReduce library
kv  A file format storing key-value pairs.

Index

Package Files

fs.go io.go utils.go

Constants

const UNKNOWN_LEN = -1

The constants for unknown length. @see SophieReader.ReadFrom

Variables

var (
    // Returned if some Sophie file is found to have bad format.
    ErrBadFormat = errors.New("Bad Sophie format")
)
var NullCollectCloser = nullCollectCloser{}

A helper variable with a CollectCloser ignoring every thing collecting to it

func ReadStringSlice Uses

func ReadStringSlice(r Reader, sl *[]string) (err error)

A helper function that reads a slice String from a Reader. The data was serialized by WriteStringSlice function.

func WriteStringSlice Uses

func WriteStringSlice(w Writer, sl []string) error

A helper function that writes a slice of Strings to a Writer. Serialized data can be read by ReadStringSlice function.

type BufferedFileReader Uses

type BufferedFileReader struct {
    *bufio.Reader
    // contains filtered or unexported fields
}

BufferedFileReader is a sophie.ReadCloser with buffer

func (BufferedFileReader) Close Uses

func (b BufferedFileReader) Close() error

sophie.ReadCloser interface

func (BufferedFileReader) Skip Uses

func (b BufferedFileReader) Skip(n int64) (int64, error)

sophie.Reader interface

type BufferedFileWriter Uses

type BufferedFileWriter struct {
    *bufio.Writer
    // contains filtered or unexported fields
}

BufferedFileWriter is a sophie.WriteCloser with buffer.

func (BufferedFileWriter) Close Uses

func (b BufferedFileWriter) Close() error

sophie.WriteCloser interface

type ByteSlice Uses

type ByteSlice []byte

*ByteSlice implements Sophier interface.

func (*ByteSlice) ReadFrom Uses

func (ba *ByteSlice) ReadFrom(r Reader, l int) error

SophieReader interface

func (ByteSlice) WriteTo Uses

func (ba ByteSlice) WriteTo(w Writer) error

SophieWriter interface

type CollectCloser Uses

type CollectCloser interface {
    Collector
    io.Closer
}

CollectCloser is Collector + io.Closer

type CollectCloserStruct Uses

type CollectCloserStruct struct {
    CollectF func(SophieWriter, SophieWriter) error
    CloseF   func() error
}

CollectCloserStruct is a struct whose pointer implements CollectCloser interface

func (*CollectCloserStruct) Close Uses

func (c *CollectCloserStruct) Close() error

func (*CollectCloserStruct) Collect Uses

func (c *CollectCloserStruct) Collect(k, v SophieWriter) error

type Collector Uses

type Collector interface {
    Collect(key, val SophieWriter) error
}

Collector is an interface for collecting Sophie key-value pairs.

type CollectorF Uses

type CollectorF func(key, val SophieWriter) error

A func type implementing Collector interface. A nil func is a no-op Collector.

func (CollectorF) Collect Uses

func (c CollectorF) Collect(key, val SophieWriter) error

Collector interface.

type EmptyClose Uses

type EmptyClose struct{}

EmptyClose is a helper type defining an empty Close method

func (EmptyClose) Close Uses

func (EmptyClose) Close() error

io.Closer interface

type FileSystem Uses

type FileSystem interface {
    // Create creates a file of a specified name.
    Create(fn string) (WriteCloser, error)
    // Mkdir makes the directory and its parents if necessary of specifiy path
    // and perm.
    Mkdir(path string, perm os.FileMode) error
    // Open opens a ReadCloser for reading the file of a specified name.
    Open(fn string) (ReadCloser, error)
    // ReadDir reads the FileInfos of the files under a specified directory.
    ReadDir(dir string) ([]os.FileInfo, error)
    // Stat returns the FileInfo of a file/directory of a specified name.
    Stat(fn string) (os.FileInfo, error)
    // Remove deletes a file or a directory(and all its file/directories in it)
    Remove(fn string) error
}

An interface defining some actions an file-system should have.

var (
    // LocalFS is a FileSystem representing the local file-system.
    LocalFS FileSystem = localFileSystem{}
)

type FsPath Uses

type FsPath struct {
    // The FileSystem
    Fs  FileSystem
    // The path
    Path string
}

FsPath is a pair of FileSystem and a path

func LocalFsPath Uses

func LocalFsPath(path string) FsPath

LocalFsPath returns an FsPath of the LocalFS and the specified path

func TempDirPath Uses

func TempDirPath() FsPath

TempDirPath returns the OS temporary dir as the fs path.

func (FsPath) Create Uses

func (fp FsPath) Create() (WriteCloser, error)

Calls FileSystem.Create with the path

func (FsPath) Join Uses

func (fp FsPath) Join(sub string) FsPath

Join returns a new FsPath with the same FileSystem and the path joined with sub

func (FsPath) Mkdir Uses

func (fp FsPath) Mkdir(perm os.FileMode) error

Calls FileSystem.Mkdir with the path

func (FsPath) Open Uses

func (fp FsPath) Open() (ReadCloser, error)

Calls FileSystem.Open with the path

func (FsPath) ReadDir Uses

func (fp FsPath) ReadDir() ([]os.FileInfo, error)

Calls FileSystem.ReadDir with the path

func (FsPath) Remove Uses

func (fp FsPath) Remove() error

Calls FileSystem.Remove with the path

func (FsPath) Stat Uses

func (fp FsPath) Stat() (os.FileInfo, error)

Calls FileSystem.Stat with the path

type Int32 Uses

type Int32 int32

*Int32 implements Sophie interface

func (*Int32) ReadFrom Uses

func (i *Int32) ReadFrom(r Reader, l int) error

SophieReader interface

func (*Int32) Val Uses

func (i *Int32) Val() int32

func (Int32) WriteTo Uses

func (i Int32) WriteTo(w Writer) error

SophieWriter interface

type IterateCloser Uses

type IterateCloser interface {
    Iterator
    io.Closer
}

sohpie.IterateCloser is Iterator + io.Closer

type IterateCloserStruct Uses

type IterateCloserStruct struct {
    NextF   func(key, val SophieReader) error
    CloserF func() error
}

A struct implementing IterateCloser by funcs.

func (*IterateCloserStruct) Close Uses

func (ics *IterateCloserStruct) Close() error

io.Closer struct.

func (*IterateCloserStruct) Next Uses

func (ics *IterateCloserStruct) Next(key, val SophieReader) error

Iterator interface

type Iterator Uses

type Iterator interface {
    Next(key, val SophieReader) error
}

Iterator is an interface for iterating Sophier kv pairs.

type Null Uses

type Null struct{}

Null is an empty data structure implementing Sophie interface.

var NULL Null = Null{}

NULL is a variable of type Null.

func (Null) ReadFrom Uses

func (Null) ReadFrom(r Reader, l int) error

SophieReader interface

func (Null) WriteTo Uses

func (Null) WriteTo(w Writer) error

SophieWriter interface

type RawByteSlice Uses

type RawByteSlice []byte

*RawByteSlice implements Sophier interface. It encodes byte-slice assuming the length of buffer will be known when decoding.

func (*RawByteSlice) ReadFrom Uses

func (ba *RawByteSlice) ReadFrom(r Reader, sz int) error

SophieReader interface

func (RawByteSlice) WriteTo Uses

func (ba RawByteSlice) WriteTo(w Writer) error

SophieWriter interface

type RawString Uses

type RawString string

*RawString implements Sophie interface. It assumes the length to be known.

func (*RawString) ReadFrom Uses

func (s *RawString) ReadFrom(r Reader, l int) error

SophieReader interface

func (*RawString) String Uses

func (s *RawString) String() string

func (*RawString) Val Uses

func (s *RawString) Val() string

func (RawString) WriteTo Uses

func (s RawString) WriteTo(w Writer) error

SophieWriter interface.

type RawVInt Uses

type RawVInt int

*RawVInt implements Sophie interface and serializing as a vint. It assumes the length to be known.

func (*RawVInt) ReadFrom Uses

func (i *RawVInt) ReadFrom(r Reader, l int) error

SophieReader interface

func (*RawVInt) String Uses

func (i *RawVInt) String() string

func (*RawVInt) Val Uses

func (i *RawVInt) Val() int

func (RawVInt) WriteTo Uses

func (i RawVInt) WriteTo(w Writer) error

SophieWriter interface

type ReadCloser Uses

type ReadCloser interface {
    Reader
    io.Closer
}

sophie.ReadCloser is sohpie.Reader + io.Closer

type Reader Uses

type Reader interface {
    io.Reader
    io.ByteReader
    // Skip skips n bytes, returns the number of actually skipped
    Skip(n int64) (int64, error)
}

sophie.Reader is an interface extended from io.Reader + io.ByteReader

type SophieReader Uses

type SophieReader interface {
    // ReadFrom reads fields from a Reader. If l is not UNKNOWN_LEN, it can be
    // used to determine the border of the serialized data.
    // @param l  the number of bytes to read. UNKNOWN_LEN(-1) means unknown
    //           length
    ReadFrom(r Reader, l int) error
}

SophieReader is the interface for some data structure that can reads fields from a Reader. The data in the Reader could have known length. For all predefined Sophies with prefix Raw, the length must be specified.

type SophieWriter Uses

type SophieWriter interface {
    // WriteTo writes fields to the Writer
    WriteTo(w Writer) error
}

SophieWriter is the interface for some data structure that can write fields to a Writer.

type Sophier Uses

type Sophier interface {
    SophieReader
    SophieWriter
}

Sophier is a basic data structure for serialization. It is SophieReader + SophieWriter

func NewByteSlice Uses

func NewByteSlice() Sophier

Returns a new instance of *ByteSlice as a Sophier

func NewInt32 Uses

func NewInt32() Sophier

Returns a new instance of *Int32 as a Sophier

func NewRawByteSlice Uses

func NewRawByteSlice() Sophier

Returns a new instance of *RawByteSlice as a Sophier

func NewRawString Uses

func NewRawString() Sophier

Returns a new instance of *RawByteSlice as a Sophier

func NewRawVInt Uses

func NewRawVInt() Sophier

Returns a new instance of *RawVInt as a Sophier

func NewString Uses

func NewString() Sophier

Returns a new instance of *String as a Sophier

func NewTime Uses

func NewTime() Sophier

Returns a new instance of *Time as a Sophier

func NewVInt Uses

func NewVInt() Sophier

Returns a new instace of *VInt as a Sophier

func ReturnNULL Uses

func ReturnNULL() Sophier

type String Uses

type String string

*String implements Sophie interface

func ReadString Uses

func ReadString(r Reader) (String, error)

A helper function that reads a String from a Reader.

func (*String) ReadFrom Uses

func (s *String) ReadFrom(r Reader, l int) error

SophieReader interface

func (*String) String Uses

func (s *String) String() string

func (*String) Val Uses

func (s *String) Val() string

func (String) WriteTo Uses

func (s String) WriteTo(w Writer) error

SophieWriter interface

type Time Uses

type Time time.Time

Time is a time.Time, and *Time implements Sophie interface.

func (*Time) ReadFrom Uses

func (t *Time) ReadFrom(r Reader, l int) error

SophieReader interface

func (Time) WriteTo Uses

func (t Time) WriteTo(w Writer) error

SophieWriter interface

type VInt Uses

type VInt int

*VInt implements Sophie interface and serializing as a vint

func (*VInt) ReadFrom Uses

func (i *VInt) ReadFrom(r Reader, l int) error

SophieReader interface

func (*VInt) String Uses

func (i *VInt) String() string

func (*VInt) Val Uses

func (i *VInt) Val() int

func (VInt) WriteTo Uses

func (i VInt) WriteTo(w Writer) error

SophieWriter interface

type WriteCloser Uses

type WriteCloser interface {
    Writer
    io.Closer
}

sophie.WriteCloser is sophie.Writer + io.Closer

type Writer Uses

type Writer interface {
    io.Writer
    io.ByteWriter
}

sophie.Writer is io.Writer + io.ByteWriter

Directories

PathSynopsis
kvPackage kv supporting read and write of a simple file formating for Sophie, which stores key-value pairs.
mrPackage mr provides a local concurrent computing model(MapReduce) using Sophie serialization.

Package sophie imports 9 packages (graph) and is imported by 48 packages. Updated 2016-07-18. Refresh now. Tools for package owners.