go-git: github.com/go-git/go-git/plumbing/format/idxfile Index | Files

package idxfile

import "github.com/go-git/go-git/plumbing/format/idxfile"

Package idxfile implements encoding and decoding of packfile idx files.

== Original (version 1) pack-*.idx files have the following format:

  - The header consists of 256 4-byte network byte order
    integers.  N-th entry of this table records the number of
    objects in the corresponding pack, the first byte of whose
    object name is less than or equal to N.  This is called the
    'first-level fan-out' table.

  - The header is followed by sorted 24-byte entries, one entry
    per object in the pack.  Each entry is:

   4-byte network byte order integer, recording where the
   object is stored in the packfile as the offset from the

   20-byte object name.

 - The file is concluded with a trailer:

   A copy of the 20-byte SHA1 checksum at the end of
   corresponding packfile.

   20-byte SHA1-checksum of all of the above.

 Pack Idx file:

      --  +--------------------------------+
 fanout   | fanout[0] = 2 (for example)    |-.
 table    +--------------------------------+ |
          | fanout[1]                      | |
          +--------------------------------+ |
          | fanout[2]                      | |
          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
          | fanout[255] = total objects    |---.
      --  +--------------------------------+ | |
 main     | offset                         | | |
 index    | object name 00XXXXXXXXXXXXXXXX | | |
 tab      +--------------------------------+ | |
          | offset                         | | |
          | object name 00XXXXXXXXXXXXXXXX | | |
          +--------------------------------+<+ |
        .-| offset                         |   |
        | | object name 01XXXXXXXXXXXXXXXX |   |
        | +--------------------------------+   |
        | | offset                         |   |
        | | object name 01XXXXXXXXXXXXXXXX |   |
        | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~   |
        | | offset                         |   |
        | | object name FFXXXXXXXXXXXXXXXX |   |
      --| +--------------------------------+<--+
trailer | | packfile checksum              |
        | +--------------------------------+
        | | idxfile checksum               |
        | +--------------------------------+
Pack file entry: <+

   packed object header:
   1-byte size extension bit (MSB)
         type (next 3 bit)
         size0 (lower 4-bit)
       n-byte sizeN (as long as MSB is set, each 7-bit)
       size0..sizeN form 4+7+7+..+7 bit integer, size0
       is the least significant part, and sizeN is the
       most significant part.
   packed object data:
       If it is not DELTA, then deflated bytes (the size above
       is the size before compression).
   If it is REF_DELTA, then
     20-byte base object name SHA1 (the size above is the
       size of the delta data that follows).
         delta data, deflated.
   If it is OFS_DELTA, then
     n-byte offset (see below) interpreted as a negative
       offset from the type-byte of the header of the
       ofs-delta entry (the size above is the size of
       the delta data that follows).
     delta data, deflated.

   offset encoding:
     n bytes with MSB set in all but the last one.
     The offset is then the number constructed by
     concatenating the lower 7 bit of each byte, and
     for n >= 2 adding 2^7 + 2^14 + ... + 2^(7*(n-1))
     to the result.

 == Version 2 pack-*.idx files support packs larger than 4 GiB, and
    have some other reorganizations.  They have the format:

   - A 4-byte magic number '\377tOc' which is an unreasonable
     fanout[0] value.

   - A 4-byte version number (= 2)

   - A 256-entry fan-out table just like v1.

   - A table of sorted 20-byte SHA1 object names.  These are
     packed together without offset values to reduce the cache
     footprint of the binary search for a specific object name.

   - A table of 4-byte CRC32 values of the packed object data.
     This is new in v2 so compressed data can be copied directly
     from pack to pack during repacking without undetected
     data corruption.

   - A table of 4-byte offset values (in network byte order).
     These are usually 31-bit pack file offsets, but large
     offsets are encoded as an index into the next table with
     the msbit set.

   - A table of 8-byte offset entries (empty for pack files less
     than 2 GiB).  Pack files are organized with heavily used
     objects toward the front, so most object references should
     not need to refer to this table.

   - The same trailer as a v1 pack file:

     A copy of the 20-byte SHA1 checksum at the end of
     corresponding packfile.

     20-byte SHA1-checksum of all of the above.

Source: https://www.kernel.org/pub/software/scm/git/docs/v1.7.5/technical/pack-format.txt


Package Files

decoder.go doc.go encoder.go idxfile.go writer.go


const (
    // VersionSupported is the only idx version supported.
    VersionSupported = 2


var (
    // ErrUnsupportedVersion is returned by Decode when the idx file version
    // is not supported.
    ErrUnsupportedVersion = errors.New("Unsupported version")
    // ErrMalformedIdxFile is returned by Decode when the idx file is corrupted.
    ErrMalformedIdxFile = errors.New("Malformed IDX file")

type Decoder Uses

type Decoder struct {

Decoder reads and decodes idx files from an input stream.

func NewDecoder Uses

func NewDecoder(r io.Reader) *Decoder

NewDecoder builds a new idx stream decoder, that reads from r.

func (*Decoder) Decode Uses

func (d *Decoder) Decode(idx *MemoryIndex) error

Decode reads from the stream and decode the content into the MemoryIndex struct.

type Encoder Uses

type Encoder struct {
    // contains filtered or unexported fields

Encoder writes MemoryIndex structs to an output stream.

func NewEncoder Uses

func NewEncoder(w io.Writer) *Encoder

NewEncoder returns a new stream encoder that writes to w.

func (*Encoder) Encode Uses

func (e *Encoder) Encode(idx *MemoryIndex) (int, error)

Encode encodes an MemoryIndex to the encoder writer.

type Entry Uses

type Entry struct {
    Hash   plumbing.Hash
    CRC32  uint32
    Offset uint64

Entry is the in memory representation of an object entry in the idx file.

type EntryIter Uses

type EntryIter interface {
    // Next returns the next entry in the packfile index.
    Next() (*Entry, error)
    // Close closes the iterator.
    Close() error

EntryIter is an iterator that will return the entries in a packfile index.

type Index Uses

type Index interface {
    // Contains checks whether the given hash is in the index.
    Contains(h plumbing.Hash) (bool, error)
    // FindOffset finds the offset in the packfile for the object with
    // the given hash.
    FindOffset(h plumbing.Hash) (int64, error)
    // FindCRC32 finds the CRC32 of the object with the given hash.
    FindCRC32(h plumbing.Hash) (uint32, error)
    // FindHash finds the hash for the object with the given offset.
    FindHash(o int64) (plumbing.Hash, error)
    // Count returns the number of entries in the index.
    Count() (int64, error)
    // Entries returns an iterator to retrieve all index entries.
    Entries() (EntryIter, error)
    // EntriesByOffset returns an iterator to retrieve all index entries ordered
    // by offset.
    EntriesByOffset() (EntryIter, error)

Index represents an index of a packfile.

type MemoryIndex Uses

type MemoryIndex struct {
    Version uint32
    Fanout  [256]uint32
    // FanoutMapping maps the position in the fanout table to the position
    // in the Names, Offset32 and CRC32 slices. This improves the memory
    // usage by not needing an array with unnecessary empty slots.
    FanoutMapping    [256]int
    Names            [][]byte
    Offset32         [][]byte
    CRC32            [][]byte
    Offset64         []byte
    PackfileChecksum [20]byte
    IdxChecksum      [20]byte
    // contains filtered or unexported fields

MemoryIndex is the in memory representation of an idx file.

func NewMemoryIndex Uses

func NewMemoryIndex() *MemoryIndex

NewMemoryIndex returns an instance of a new MemoryIndex.

func (*MemoryIndex) Contains Uses

func (idx *MemoryIndex) Contains(h plumbing.Hash) (bool, error)

Contains implements the Index interface.

func (*MemoryIndex) Count Uses

func (idx *MemoryIndex) Count() (int64, error)

Count implements the Index interface.

func (*MemoryIndex) Entries Uses

func (idx *MemoryIndex) Entries() (EntryIter, error)

Entries implements the Index interface.

func (*MemoryIndex) EntriesByOffset Uses

func (idx *MemoryIndex) EntriesByOffset() (EntryIter, error)

EntriesByOffset implements the Index interface.

func (*MemoryIndex) FindCRC32 Uses

func (idx *MemoryIndex) FindCRC32(h plumbing.Hash) (uint32, error)

FindCRC32 implements the Index interface.

func (*MemoryIndex) FindHash Uses

func (idx *MemoryIndex) FindHash(o int64) (plumbing.Hash, error)

FindHash implements the Index interface.

func (*MemoryIndex) FindOffset Uses

func (idx *MemoryIndex) FindOffset(h plumbing.Hash) (int64, error)

FindOffset implements the Index interface.

type Writer Uses

type Writer struct {
    // contains filtered or unexported fields

Writer implements a packfile Observer interface and is used to generate indexes.

func (*Writer) Add Uses

func (w *Writer) Add(h plumbing.Hash, pos uint64, crc uint32)

Add appends new object data.

func (*Writer) Finished Uses

func (w *Writer) Finished() bool

func (*Writer) Index Uses

func (w *Writer) Index() (*MemoryIndex, error)

Index returns a previously created MemoryIndex or creates a new one if needed.

func (*Writer) OnFooter Uses

func (w *Writer) OnFooter(h plumbing.Hash) error

OnFooter implements packfile.Observer interface.

func (*Writer) OnHeader Uses

func (w *Writer) OnHeader(count uint32) error

OnHeader implements packfile.Observer interface.

func (*Writer) OnInflatedObjectContent Uses

func (w *Writer) OnInflatedObjectContent(h plumbing.Hash, pos int64, crc uint32, _ []byte) error

OnInflatedObjectContent implements packfile.Observer interface.

func (*Writer) OnInflatedObjectHeader Uses

func (w *Writer) OnInflatedObjectHeader(t plumbing.ObjectType, objSize int64, pos int64) error

OnInflatedObjectHeader implements packfile.Observer interface.

Package idxfile imports 13 packages (graph). Updated 2020-07-20. Refresh now. Tools for package owners.