Documentation
¶
Index ¶
- Constants
- Variables
- func FileName(i types.SegmentInfo) string
- type Filer
- func (f *Filer) Create(info types.SegmentInfo) (types.SegmentWriter, error)
- func (f *Filer) Delete(baseIndex uint64, ID uint64) error
- func (f *Filer) DumpLogs(after, before uint64, ...) error
- func (f *Filer) DumpSegment(baseIndex uint64, ID uint64, after, before uint64, ...) error
- func (f *Filer) List() (map[uint64]uint64, error)
- func (f *Filer) Open(info types.SegmentInfo) (types.SegmentReader, error)
- func (f *Filer) RecoverTail(info types.SegmentInfo) (types.SegmentWriter, error)
- type Reader
- type Writer
Constants ¶
const ( FrameInvalid uint8 = iota FrameEntry FrameIndex FrameCommit )
const ( // MaxEntrySize is the largest we allow any single raft log entry to be. This // is larger than our raft implementation ever allows so seems safe to encode // statically for now. We could make this configurable. It's main purpose it // to limit allocation when reading entries back if their lengths are // corrupted. MaxEntrySize = 64 * 1024 * 1024 // 64 MiB )
Variables ¶
var ( // ErrTooBig indicates that the caller tried to write a logEntry with a // payload that's larger than we are prepared to support. ErrTooBig = errors.New("entries larger than 64MiB are not supported") )
Functions ¶
func FileName ¶
func FileName(i types.SegmentInfo) string
FileName returns the formatted file name expected for this segment. SegmentFiler implementations could choose to ignore this but it's here to
Types ¶
type Filer ¶
type Filer struct {
// contains filtered or unexported fields
}
Filer implements the abstraction for managing a set of segment files in a directory. It uses a VFS to abstract actual file system operations for easier testing.
func (*Filer) Create ¶
func (f *Filer) Create(info types.SegmentInfo) (types.SegmentWriter, error)
Create adds a new segment with the given info and returns a writer or an error.
func (*Filer) Delete ¶
Delete removes the segment with given baseIndex and id if it exists. Note that baseIndex is technically redundant since ID is unique on it's own. But in practice we name files (or keys) with both so that they sort correctly. This interface allows a simpler implementation where we can just delete the file if it exists without having to scan the underlying storage for a.
func (*Filer) DumpLogs ¶
func (f *Filer) DumpLogs(after, before uint64, fn func(info types.SegmentInfo, e types.LogEntry) (bool, error)) error
DumpLogs attempts to read all log entries from segment files in the directory for debugging purposes. It does _not_ use the metadata and so may output log entries that are uncommitted or already truncated as far as the writing process is concerned. As such it should not be used for replication of data. It is useful though to debug the contents of the log even while the writing application is still running. After and before if non-zero specify exclusive bounds on the logs that should be returned which may allow the implementation to skip reading entire segment files that are not in the range.
func (*Filer) DumpSegment ¶
func (f *Filer) DumpSegment(baseIndex uint64, ID uint64, after, before uint64, fn func(info types.SegmentInfo, e types.LogEntry) (bool, error)) error
DumpSegment attempts to read the segment file specified by the baseIndex and ID. It's intended purpose is for debugging the contents of segment files and unlike the SegmentFiler interface, it doesn't assume the caller has access to the correct metadata. This allows dumping log segments in a WAL that is still being written to by another process. Without metadata we don't know if the file is sealed so always recover by reading through the whole file. If after or before are non-zero, the specify a exclusive lower or upper bound on which log entries should be emitted. No error checking is done on the read data. fn is called for each entry passing the raft info read from the file header (so that the caller knows which codec to use for example) the raft index of the entry and the raw bytes of the entry itself. The callback must return true to continue reading. The data slice is only valid for the lifetime of the call.
func (*Filer) List ¶
List returns the set of segment IDs currently stored. It's used by the WAL on recovery to find any segment files that need to be deleted following a unclean shutdown. The returned map is a map of ID -> BaseIndex. BaseIndex is returned to allow subsequent Delete calls to be made.
func (*Filer) Open ¶
func (f *Filer) Open(info types.SegmentInfo) (types.SegmentReader, error)
Open an already sealed segment for reading. Open may validate the file's header and return an error if it doesn't match the expected info.
func (*Filer) RecoverTail ¶
func (f *Filer) RecoverTail(info types.SegmentInfo) (types.SegmentWriter, error)
RecoverTail is called on an unsealed segment when re-opening the WAL it will attempt to recover from a possible crash. It will either return an error, or return a valid segmentWriter that is ready for further appends. If the expected tail segment doesn't exist it must return an error wrapping os.ErrNotExist.
type Reader ¶
type Reader struct {
// contains filtered or unexported fields
}
Reader allows reading logs from a segment file.
type Writer ¶
type Writer struct {
// contains filtered or unexported fields
}
Writer allows appending logs to a segment file as well as reading them back.
func (*Writer) Append ¶
Append adds one or more entries. It must not return until the entries are durably stored otherwise raft's guarantees will be compromised.
func (*Writer) LastIndex ¶
LastIndex returns the most recently persisted index in the log. It must respond without blocking on append since it's needed frequently by read paths that may call it concurrently. Typically this will be loaded from an atomic int. If the segment is empty lastIndex should return zero.
func (*Writer) OffsetForFrame ¶
OffsetForFrame implements tailWriter and allows readers to lookup entry frames in the tail's in-memory index.
func (*Writer) Sealed ¶
Sealed returns whether the segment is sealed or not. If it is it returns true and the file offset that it's index array starts at to be saved in meta data. WAL will call this after every append so it should be relatively cheap in the common case. This design allows the final Append to write out the index or any additional data needed at seal time in the same fsync.
Source Files