ztoc

package
v0.6.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 11, 2024 License: Apache-2.0 Imports: 21 Imported by: 2

Documentation

Index

Constants

View Source 
const (
	// TarBlockSize is the size of a tar block
	TarBlockSize = 512
)

Variables

View Source 
var ErrInvalidTOCEntry = errors.New("invalid toc entry")

Functions

func AlignToTarBlock added in v0.5.0 

func AlignToTarBlock(o compression.Offset) compression.Offset

AlignToTarBlock aligns an offset to the next larger multiple of TarBlockSize

func Marshal 

func Marshal(ztoc *Ztoc) (io.Reader, ocispec.Descriptor, error)

Marshal serializes Ztoc to its flatbuffers schema and returns a reader along with the descriptor (digest and size only). If not successful, it will return an error.

func TarProviderGzip 

func TarProviderGzip(compressedReader *os.File) (io.Reader, error)

TarProviderGzip creates a tar reader from gzip reader.

func TarProviderTar added in v0.3.0 

func TarProviderTar(compressedReader *os.File) (io.Reader, error)

TarProviderTar return the tar file directly as the input to `tar.NewReader`.

func TarProviderZstd 

func TarProviderZstd(compressedReader *os.File) (io.Reader, error)

TarProviderZstd creates a tar reader from zstd reader.

func Xattrs added in v0.5.0 

func Xattrs(paxHeaders map[string]string) map[string]string

Xattrs converts a set of tar PAXRecords to a set of Xattrs. Specifically, it looks for PAXRecords where the key is prefixed by `SCHILY.xattr.` - the prefix for Xattrs used by go and GNU tar. Those keys are kept with the prefix stripped. Other keys are dropped.

Types

type BuildOption 

type BuildOption func(opt *buildConfig) error

BuildOption specifies a change to `buildConfig` when building a ztoc.

func WithCompression 

func WithCompression(algorithm string) BuildOption

WithCompression specifies which compression algorithm is used by the layer.

type Builder 

type Builder struct {
	// contains filtered or unexported fields
}

Builder holds a single `TocBuilder` that builds toc, and one `ZinfoBuilder` *per* compression algorithm that builds zinfo. `TocBuilder` is shared by different compression algorithms. Which `ZinfoBuilder` is used depends on the compression algorithm used by the layer.

func NewBuilder 

func NewBuilder(buildToolIdentifier string) *Builder

NewBuilder creates a `Builder` used to build ztocs. By default it supports gzip, user can register new compression algorithms by calling `RegisterCompressionAlgorithm`.

func (*Builder) BuildZtoc 

func (b *Builder) BuildZtoc(filename string, span int64, options ...BuildOption) (*Ztoc, error)

BuildZtoc builds a `Ztoc` given the filename of a layer blob. By default it assumes the layer is compressed using `gzip`, unless specified via `WithCompression`.

func (*Builder) CheckCompressionAlgorithm 

func (b *Builder) CheckCompressionAlgorithm(algorithm string) bool

CheckCompressionAlgorithm checks if a compression algorithm is supported.

The algorithm has to be supported by both (1) `tocBuilder` (straightforward, create a tar reader from the compressed io.reader in compressionFileReader) and (2) `zinfoBuilder` (require zinfo impl, see `compression/gzip_zinfo.go` as an example).

func (*Builder) RegisterCompressionAlgorithm 

func (b *Builder) RegisterCompressionAlgorithm(name string, tarProvider TarProvider, zinfoBuilder ZinfoBuilder)

RegisterCompressionAlgorithm supports a new compression algorithm in `ztoc.Builder`.

type CompressionInfo 

type CompressionInfo struct {
	MaxSpanID            compression.SpanID //The total number of spans in Ztoc - 1
	SpanDigests          []digest.Digest
	Checkpoints          []byte
	CompressionAlgorithm string
}

CompressionInfo is the "zinfo" part of ztoc including the `Checkpoints` data and other metadata such as all span digests.

type FileMetadata 

type FileMetadata struct {
	Name               string
	Type               string
	UncompressedOffset compression.Offset
	UncompressedSize   compression.Offset
	TarHeaderOffset    compression.Offset
	Linkname           string // Target name of link (valid for TypeLink or TypeSymlink)
	Mode               int64  // Permission and mode bits
	UID                int    // User ID of owner
	GID                int    // Group ID of owner
	Uname              string // User name of owner
	Gname              string // Group name of owner

	ModTime  time.Time // Modification time
	Devmajor int64     // Major device number (valid for TypeChar or TypeBlock)
	Devminor int64     // Minor device number (valid for TypeChar or TypeBlock)

	PAXHeaders map[string]string
}

FileMetadata contains metadata of a file in the compressed data.

func (FileMetadata) Equal added in v0.5.0 

func (src FileMetadata) Equal(o FileMetadata) bool

func (FileMetadata) FileMode added in v0.3.0 

func (src FileMetadata) FileMode() (m os.FileMode)

FileMode gets file mode for the file metadata

func (FileMetadata) Xattrs 

func (src FileMetadata) Xattrs() map[string]string

type MetadataEntry 

type MetadataEntry struct {
	UncompressedSize   compression.Offset
	UncompressedOffset compression.Offset
}

MetadataEntry is used to locate a file based on its metadata.

type TOC 

type TOC struct {
	FileMetadata []FileMetadata
}

TOC is the "ztoc" part of ztoc including metadata of all files in the compressed data (e.g., a gzip tar file).

func (TOC) GetMetadataEntry added in v0.3.0 

func (toc TOC) GetMetadataEntry(filename string) (MetadataEntry, error)

GetMetadataEntry gets MetadataEntry given a filename.

type TarProvider 

type TarProvider func(file *os.File) (io.Reader, error)

TarProvider creates a tar reader from a compressed file reader (e.g., a gzip file reader), which can be used by `TocBuilder` to create `TOC` from it.

type TocBuilder 

type TocBuilder struct {
	// contains filtered or unexported fields
}

TocBuilder builds the `TOC` part of a ztoc and works with different compression algorithms (e.g., gzip, zstd) with a registered `TarProvider`.

func NewTocBuilder 

func NewTocBuilder() TocBuilder

NewTocBuilder return a `TocBuilder` struct. Users need to call `RegisterTarProvider` to support a specific compression algorithm.

func (TocBuilder) CheckCompressionAlgorithm 

func (tb TocBuilder) CheckCompressionAlgorithm(algorithm string) bool

CheckCompressionAlgorithm checks if a compression algorithm is supported.

func (TocBuilder) RegisterTarProvider 

func (tb TocBuilder) RegisterTarProvider(algorithm string, provider TarProvider)

RegisterTarProvider adds a TarProvider for a compression algorithm.

func (TocBuilder) TocFromFile 

func (tb TocBuilder) TocFromFile(algorithm, filename string) (TOC, compression.Offset, error)

TocFromFile creates a `TOC` given a layer blob filename and the compression algorithm used by the layer.

type Version added in v0.3.0 

type Version string

Version defines the version of a Ztoc. 

const (
	Version09 Version = "0.9"
)

Ztoc versions available.

type ZinfoBuilder 

type ZinfoBuilder interface {
	// ZinfoFromFile builds zinfo given a compressed tar filename and span size, and calculate the size of the file.
	ZinfoFromFile(filename string, spanSize int64) (zinfo CompressionInfo, fs compression.Offset, err error)
}

ZinfoBuilder builds the `zinfo` part of a ztoc. This interface should be implemented for each compression algorithm we support.

type Ztoc 

type Ztoc struct {
	TOC
	CompressionInfo

	Version                 Version
	BuildToolIdentifier     string
	CompressedArchiveSize   compression.Offset
	UncompressedArchiveSize compression.Offset
}

Ztoc is a table of contents for compressed data which consists 2 parts:

(1). toc (`TOC`): a table of contents containing file metadata and its offset in the decompressed TAR archive. (2). zinfo (`CompressionInfo`): a collection of "checkpoints" of the state of the compression engine at various points in the layer.

func BuildZtocReader 

func BuildZtocReader(_ *testing.T, ents []testutil.TarEntry, compressionLevel int, spanSize int64, opts ...testutil.BuildTarOption) (*Ztoc, *io.SectionReader, error)

BuildZtocReader creates the tar gz file for tar entries. It returns ztoc and io.SectionReader of the file.

func Unmarshal 

func Unmarshal(serializedZtoc io.Reader) (*Ztoc, error)

Unmarshal takes the reader with flatbuffers byte stream and deserializes it ztoc. In case if there's any error situation during deserialization from flatbuffers, there will be an error returned.

func (Ztoc) ExtractFile added in v0.3.0 

func (zt Ztoc) ExtractFile(r *io.SectionReader, filename string) ([]byte, error)

ExtractFile extracts a file from compressed data (as a reader) and returns the byte data.

func (Ztoc) ExtractFromTarGz added in v0.3.0 

func (zt Ztoc) ExtractFromTarGz(gz string, filename string) (string, error)

ExtractFromTarGz extracts data given a gzip tar file (`gz`) and its `ztoc`.

func (Ztoc) Zinfo added in v0.3.0 

func (zt Ztoc) Zinfo() (compression.Zinfo, error)

Zinfo deserilizes and returns a Zinfo based on the zinfo bytes and compression algorithm in the ztoc.

Source Files

View all Source files

Directories

Path Synopsis
fbs/zinfo
fbs
ztoc

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.