verifier

type ErrChecksumMismatch ¶

type ErrChecksumMismatch string

ErrChecksumMismatch is the error type returned in a VerificationReport where the log range's checksum didn't match.

func (e ErrChecksumMismatch) Error() string

type IsCheckpointFn ¶

type IsCheckpointFn func(*raft.Log) (bool, error)

IsCheckpointFn is a function that can decide whether the contents of a raft log's Data represents a checkpoint message. It is called on every append so it must be relatively fast in the common case. If it returns true for a log, the log's Extra field will be used to encode verification metadata and must be empty - if it's not empty the append will fail and force the leader to step down. If an error is returned the same will happen.

type LogRange ¶

type LogRange struct {
	Start uint64
	End   uint64
}

LogRange describes the set of logs in the range [Start, End). That is End is NOT inclusive.

func (r LogRange) String() string

type LogStore ¶

type LogStore struct {
	// contains filtered or unexported fields
}

LogStore is a raft.LogStore that acts as middleware around an underlying persistent store. It provides support for periodically verifying that ranges of logs read back from the LogStore match the values written, and the values read from the LogStores of other peers even though all peers will have different actual log ranges due to independent snapshotting and truncation.

Verification of the underlying log implementation may be performed as follows:

1. The application provides an implementation of `IsCheckpoint` that is able to identify whether the encoded data represents a checkpoint command.
2. The application's raft leader then may periodically append such a checkpoint log to be replicated out.
3. When the LogStore has a log appended for which IsCheckpoint returns true, it will write the current cumulative checksum over log entries since the last checkpoint into the Extra field. Since hashicorp/raft only replicates to peers _after_ a trip through the LogStore, this checksum will be replicated.
4. When a follower has a log appended for which IsCheckpoint returns true, but already has non-empty Extra metadata, it will trigger a background verification.
5. Verification happens in the background and reads all logs from the underlying store since the last checkpoint, calculating their checksums cumulatively before calling the configured Report func with a summary of what it found.

func NewLogStore(store raft.LogStore, checkpointFn IsCheckpointFn, reportFn ReportFn, mc metrics.Collector) *LogStore

func (s *LogStore) Close() error

func (s *LogStore) DeleteRange(min uint64, max uint64) error

func (s *LogStore) FirstIndex() (uint64, error)

func (s *LogStore) GetLog(index uint64, log *raft.Log) error

func (s *LogStore) IsMonotonic() bool

func (s *LogStore) LastIndex() (uint64, error)

func (s *LogStore) StoreLog(log *raft.Log) error

func (s *LogStore) StoreLogs(logs []*raft.Log) error

type ReportFn ¶

type ReportFn func(VerificationReport)

ReportFn is a function that will be called after every checkpoint has been verified. It will not be called concurrently. The VerificationReport may represent a failure to report so it's Err field should be checked. For example, if checkpoints are arriving faster than they can be calculated, some will be skipped and no report will be made for that range. The next report that is delivered will contain the range missed for logging. Note that ReportFn is called synchronously by the verifier so it should not block for long otherwise it may cause the verifier to miss later checkpoints.

type VerificationReport ¶

type VerificationReport struct {
	// Range is the range of raft indexes over which the leader calculated its
	// checksum. In steady state it typically starts with the index of the
	// previous checkpoint command, but after an election it could be an arbitrary
	// point in the log. If the range is no longer in the server's log (due to not
	// seeing one yet or it being truncated too soon) this will be reported as an
	// Err - a longer log retention (`raft.Config.TrailingLogs`) or shorter
	// interval between checkpoints should be chosen if this happens often.
	Range LogRange

	// ExpectedSum is a uint64 checksum over the logs in the range as calculated
	// by the leader before appending to disk.
	ExpectedSum uint64

	// WrittenSum is the uint64 checksum calculated over the logs in the range of
	// a follower as it wrote them to it's own LogStore. It might be zero to
	// indicate that the follower has not written all the logs in Range since
	// startup and so its written sum will be invalid. Risk of collision with
	// genuine zero sum is acceptable. If zero the verifier will have ignored it
	// and not raised an error if it didn't match expected.
	WrittenSum uint64

	// ReadSum is the uint64 checksum calculated over the logs in the range as
	// read from the underlying LogStore in the range [StartIndex, EndIndex).
	ReadSum uint64

	// Err indicates any error that prevented the report from being completed or
	// the result of the report. It will be set to ErrChecksumMismatch if the
	// report was conducted correctly, but the log data written or read checksum
	// did not match the leader's write checksum. The message in the error
	// describes the nature of the failure.
	Err error

	// SkippedRange indicates the ranges of logs covered by any checkpoints that
	// we skipped due to spending too much time verifying. If this is regularly
	// non-nil it likely indicates that the checkpoint frequency is too fast.
	SkippedRange *LogRange

	// Elapsed records how long it took to read the range and generate the report.
	Elapsed time.Duration
}

VerificationReport describes the result of attempting to verify the contents of all logs in a range compared with the input the leader delivered for that same range.