README
¶
This is a from scratch implementation of the reftable format for storing Git ref and reflog data.
The spec is at https://eclipse.googlesource.com/jgit/jgit/+/refs/heads/master/Documentation/technical/reftable.md
Based on the implementation, there is also a proposal for a v2 of the format.
Java
Reftable was originally implemented in JGit, and can be considered the reference implementation. The code has two parts:
-
The format itself: https://eclipse.googlesource.com/jgit/jgit/+/master/org.eclipse.jgit/src/org/eclipse/jgit/internal/storage/reftable
-
The layout within a local disk repository: https://gerrit.googlesource.com/jgit/+/master/org.eclipse.jgit/src/org/eclipse/jgit/internal/storage/file/FileReftableStack.java
GO
The Go implementation implements the spec completely. API doc.
C
An experimental implementation in C is under the directory c/ . It is functionally equivalent to the Go version. Build and test it using
bazel test c/...
API is documented in side the reftable.h header.
It includes a fragment of the zlib library to provide uncompress2(), which is a recent addition to the API. zlib is licensed as follows:
(C) 1995-2017 Jean-loup Gailly and Mark Adler
This software is provided 'as-is', without any express or implied
warranty. In no event will the authors be held liable for any damages
arising from the use of this software.
Permission is granted to anyone to use this software for any purpose,
including commercial applications, and to alter it and redistribute it
freely, subject to the following restrictions:
1. The origin of this software must not be misrepresented; you must not
claim that you wrote the original software. If you use this software
in a product, an acknowledgment in the product documentation would be
appreciated but is not required.
2. Altered source versions must be plainly marked as such, and must not be
misrepresented as being the original software.
3. This notice may not be removed or altered from any source distribution.
Jean-loup Gailly Mark Adler
jloup@gzip.org madler@alumni.caltech.edu
Background reading
-
Spec: https://github.com/eclipse/jgit/blob/master/Documentation/technical/reftable.md
-
Original discussion on JGit-dev: https://www.eclipse.org/lists/jgit-dev/msg03389.html
-
First design discussion on git@vger: https://public-inbox.org/git/CAJo=hJtyof=HRy=2sLP0ng0uZ4=S-DpZ5dR1aF+VHVETKG20OQ@mail.gmail.com/
-
Last design discussion on git@vger: https://public-inbox.org/git/CAJo=hJsZcAM9sipdVr7TMD-FD2V2W6_pvMQ791EGCDsDkQ033w@mail.gmail.com/
-
First attempt at implementation: https://public-inbox.org/git/CAP8UFD0PPZSjBnxCA7ez91vBuatcHKQ+JUWvTD1iHcXzPBjPBg@mail.gmail.com/
-
libgit2 support issue: https://github.com/libgit2/libgit2/issues
-
GitLab support issue: https://gitlab.com/gitlab-org/git/issues/6
-
go-git support issue: https://github.com/src-d/go-git/issues/1059
DISCLAIMER
This is not an official Google product
Documentation
¶
Index ¶
- Variables
- type Addition
- type BlockSource
- type BlockStats
- type ByteBlockSource
- type CompactionStats
- type Config
- type HashID
- type Iterator
- type LogExpirationConfig
- type LogRecord
- type Merged
- func (m *Merged) HashID() HashID
- func (m *Merged) MaxUpdateIndex() uint64
- func (m *Merged) MinUpdateIndex() uint64
- func (m *Merged) Name() string
- func (m *Merged) RefsFor(oid []byte) (*Iterator, error)
- func (m *Merged) SeekLog(refname string, updateIndex uint64) (*Iterator, error)
- func (m *Merged) SeekRef(name string) (*Iterator, error)
- type Reader
- func (r *Reader) Close()
- func (r *Reader) DebugData() string
- func (r *Reader) HashID() HashID
- func (r *Reader) MaxUpdateIndex() uint64
- func (r *Reader) MinUpdateIndex() uint64
- func (r *Reader) Name() string
- func (r *Reader) RefsFor(oid []byte) (*Iterator, error)
- func (r *Reader) SeekLog(name string, updateIndex uint64) (*Iterator, error)
- func (r *Reader) SeekRef(name string) (*Iterator, error)
- type RefRecord
- type Stack
- func (st *Stack) Add(write func(w *Writer) error) error
- func (st *Stack) AutoCompact() error
- func (st *Stack) Close()
- func (st *Stack) CompactAll(expiration *LogExpirationConfig) error
- func (st *Stack) Merged() *Merged
- func (st *Stack) NewAddition() (*Addition, error)
- func (st *Stack) NextUpdateIndex() uint64
- func (st *Stack) String() string
- func (st *Stack) UpToDate() (bool, error)
- type Stats
- type Table
- type Writer
Constants ¶
This section is empty.
Variables ¶
var ErrEmptyTable = errors.New("reftable: table is empty")
ErrEmptyTable indicates that a writer tried to create a table without blocks.
var ErrLockFailure = errors.New("reftable: lock failure")
ErrLockFailure is returned for failed writes. On a failed write, the stack is reloaded, so the transaction may be retried.
var NullHashID = HashID([4]byte{0, 0, 0, 0})
var SHA1ID = HashID([4]byte{'s', 'h', 'a', '1'})
var SHA256ID = HashID([4]byte{'s', '2', '5', '6'})
Functions ¶
This section is empty.
Types ¶
type Addition ¶
type Addition struct {
// contains filtered or unexported fields
}
Addition is a transaction that adds new tables to the top of the stack.
type BlockSource ¶
type BlockSource interface {
Size() uint64
ReadBlock(off uint64, size int) ([]byte, error)
Close() error
}
BlockSource is an interface for reading reftable bytes.
func NewFileBlockSource ¶
func NewFileBlockSource(name string) (BlockSource, error)
NewFileBlockSource opens a file on local disk as a BlockSource
type BlockStats ¶
type BlockStats struct {
Entries int
Restarts int
Blocks int
IndexBlocks int
MaxIndexLevel int
Offset uint64
IndexOffset uint64
}
BlockStats provides write statistics data of a certain block type.
type ByteBlockSource ¶
type ByteBlockSource struct {
Source []byte
}
ByteBlockSource is an in-memory block source.
func (*ByteBlockSource) Close ¶
func (s *ByteBlockSource) Close() error
func (*ByteBlockSource) ReadBlock ¶
func (s *ByteBlockSource) ReadBlock(off uint64, sz int) ([]byte, error)
func (*ByteBlockSource) Size ¶
func (s *ByteBlockSource) Size() uint64
type CompactionStats ¶
type CompactionStats struct {
Bytes uint64
// All entries written, including from failed compaction attempts.
EntriesWritten uint64
Attempts int
Failures int
}
CompactionStats holds some statistics of compaction over the lifetime of the stack.
type Config ¶
type Config struct {
// If set, do not pad blocks to blocksize.
Unaligned bool
// The block size, if not set 4096.
BlockSize uint32
SkipIndexObjects bool
RestartInterval int
// Hash identifier. Unset means sha1.
HashID HashID
// Allow dir/file conflicts and illegal refnames
SkipNameCheck bool
// If set, store reflog messages exactly. If unset, only allow
// a single line, and a trailing '\n' is added if it is missing.
ExactLogMessage bool
}
Options define write options for reftables.
type Iterator ¶
type Iterator struct {
// contains filtered or unexported fields
}
iterator is an iterator over reftable
type LogExpirationConfig ¶
type LogRecord ¶
type LogRecord struct {
RefName string
UpdateIndex uint64
New []byte
Old []byte
Name string
Email string
Time uint64
TZOffset int16
Message string
}
LogRecord is a Record from the reflog database.
func (*LogRecord) IsDeletion ¶
type Merged ¶
type Merged struct {
// contains filtered or unexported fields
}
Merged is a stack of reftables.
func (*Merged) MaxUpdateIndex ¶
MaxUpdateIndex implements the Table interface.
func (*Merged) MinUpdateIndex ¶
MinUpdateIndex implements the Table interface.
type Reader ¶
type Reader struct {
// contains filtered or unexported fields
}
Reader allows reading from a reftable.
func NewReader ¶
func NewReader(src BlockSource, name string) (*Reader, error)
NewReader creates a reader for a reftable file.
func (*Reader) MaxUpdateIndex ¶
func (*Reader) MinUpdateIndex ¶
type RefRecord ¶
type RefRecord struct {
RefName string
UpdateIndex uint64
Value []byte
TargetValue []byte
// is a 0-length target allowed?
Target string
}
RefRecord is a Record from the ref database.
func (*RefRecord) IsDeletion ¶
type Stack ¶
type Stack struct {
Stats CompactionStats
// contains filtered or unexported fields
}
Stack is an auto-compacting stack of reftables.
func (*Stack) AutoCompact ¶
AutoCompact runs a compaction if the stack looks imbalanced.
func (*Stack) Close ¶
func (st *Stack) Close()
Close releases file descriptors associated with this stack.
func (*Stack) CompactAll ¶
func (st *Stack) CompactAll(expiration *LogExpirationConfig) error
CompactAll compacts the entire stack. If expiration is given, expire log entries.
func (*Stack) Merged ¶
Returns the merged stack. The stack is only valid until the next write, as writes may trigger reloads
func (*Stack) NewAddition ¶
NewAddition returns an Addition instance. As a side effect, this takes a global filesystem lock on the ref database.
func (*Stack) NextUpdateIndex ¶
NextUpdateIndex returns the update index at which to write the next table.
type Stats ¶
type Stats struct {
ObjStats BlockStats
RefStats BlockStats
LogStats BlockStats
Blocks int
ObjectIDLen int
// contains filtered or unexported fields
}
Stats provides general write statistics
type Table ¶
type Table interface {
MaxUpdateIndex() uint64
MinUpdateIndex() uint64
HashID() HashID
SeekRef(refName string) (*Iterator, error)
SeekLog(refName string, updateIndex uint64) (*Iterator, error)
RefsFor(oid []byte) (*Iterator, error)
Name() string
// contains filtered or unexported methods
}
Table is a read interface for reftables, either file reftables or merged reftables.
type Writer ¶
type Writer struct {
Stats Stats
// contains filtered or unexported fields
}
Writer writes a single reftable.
func (*Writer) AddLog ¶
AddLog adds a LogRecord to the table. AddLog must be called in ascending order.
func (*Writer) AddRef ¶
AddRef adds a RefRecord to the table. AddRef must be called in ascending order. AddRef cannot be called after AddLog is called.
Source Files
Directories