Documentation
¶
Overview ¶
A manifest is a file containing a list of paths and their hash digests, canonically ordered by path in increasing lexicographical order. Manifests are encoded as:
<digest type>:<digest>[SP][SP]<path>[LF]
"shake256" is the only supported digest type. The digest is 64 bytes of hex encoded output of SHAKE256. See golang.org/x/crypto/sha3 and FIPS 202 for details on the SHAKE hash.
Manifest can read and write manifest files. Canonical form is produced when serialized (Manifest.MarshalText). Non-canonical form is a valid manifest and will not produce errors when deserializing.
Interacting with a manifest is typically by path (Manifest.Paths, Manifest.DigestFor) or by a Digest (Manifest.PathsFor).
Blob represents file content and its digest. BlobSet collects related blobs together into a set. NewMemoryBlob provides an in-memory implementation. A manifest, being a file, is also a blob (Manifest.Blob).
Blobs are anonymous files and a manifest gives names to anonymous files. It's possible to view a manifest and its associated blobs as a file system. NewBucket creates a storage bucket from a manifest and blob set. NewFromBucket does the inverse: the creation of a manifest and blob set from a storage bucket.
Example ¶
package main
import (
"context"
"fmt"
"github.com/apache/dubbo-kubernetes/pkg/bufman/pkg/manifest"
"github.com/apache/dubbo-kubernetes/pkg/bufman/pkg/storage/storagemem"
)
func main() {
ctx := context.Background()
bucket, _ := storagemem.NewReadBucket(
map[string][]byte{
"foo": []byte("bar"),
},
)
m, _, _ := manifest.NewFromBucket(ctx, bucket)
digest, _ := m.DigestFor("foo")
fmt.Printf("digest[:16]: %s\n", digest.Hex()[:16])
path, _ := m.PathsFor(digest.String())
fmt.Printf("path at digest: %s\n", path[0])
}
Output: digest[:16]: a15163728ed24e1c path at digest: foo
Index ¶
- func BlobEqual(ctx context.Context, a, b Blob) (_ bool, retErr error)
- func NewBucket(m Manifest, blobs BlobSet, opts ...BucketOption) (storage.ReadBucket, error)
- func NewFromBucket(ctx context.Context, bucket storage.ReadBucket) (*Manifest, *BlobSet, error)
- type Blob
- type BlobSet
- type BlobSetOption
- type BucketOption
- type Digest
- type DigestType
- type Digester
- type Manifest
- func (m *Manifest) AddEntry(path string, digest Digest) error
- func (m *Manifest) Blob() (Blob, error)
- func (m *Manifest) DigestFor(path string) (*Digest, bool)
- func (m *Manifest) Digests() []Digest
- func (m *Manifest) Empty() bool
- func (m *Manifest) MarshalText() ([]byte, error)
- func (m *Manifest) Paths() []string
- func (m *Manifest) PathsFor(digest string) ([]string, bool)
- func (m *Manifest) Range(f func(path string, digest Digest) error) error
- func (m *Manifest) UnmarshalText(text []byte) error
- type MemoryBlobOption
Examples ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func BlobEqual ¶
BlobEqual returns true if blob a is the same as blob b. The digest is checked for equality and the content bytes compared.
An error is returned if an unexpected I/O error occurred when opening, reading, or closing either blob.
func NewBucket ¶
func NewBucket(m Manifest, blobs BlobSet, opts ...BucketOption) (storage.ReadBucket, error)
NewBucket takes a manifest and a blob set and builds a readable storage bucket that contains the files in the manifest.
func NewFromBucket ¶
NewFromBucket creates a manifest and blob set from the bucket's files. Blobs in the blob set use the DigestTypeShake256 digest.
Types ¶
type Blob ¶
Blob is an anonymous file associated with a digest.
func NewMemoryBlob ¶
func NewMemoryBlob(digest Digest, content []byte, opts ...MemoryBlobOption) (Blob, error)
NewMemoryBlob takes a digest and a content, and turns it into an in-memory representation of a blob, which returns the digest and an io.ReadCloser for its content.
func NewMemoryBlobFromReader ¶
NewMemoryBlobFromReader creates a memory blob from content, which is read until completion. The returned blob contains all bytes read. If you are using this in a loop, you might better use NewMemoryBlobFromReaderWithDigester so you can reuse your digester.
func NewMemoryBlobFromReaderWithDigester ¶
NewMemoryBlobFromReaderWithDigester creates a memory blob from content with the passed digester. The content is read until completion. The returned blob contains all bytes read.
type BlobSet ¶
type BlobSet struct {
// contains filtered or unexported fields
}
BlobSet represents a set of deduplicated blobs by their digests.
func NewBlobSet ¶
NewBlobSet receives a slice of blobs, and de-duplicates them into a BlobSet.
type BlobSetOption ¶
type BlobSetOption func(*blobSetOptions)
BlobSetOption are options passed when creating a new blob set.
func BlobSetWithContentValidation ¶
func BlobSetWithContentValidation() BlobSetOption
BlobSetWithContentValidation turns on content validation for all the blobs when creating a new BlobSet. If this option is on, blobs with the same digest must have the same content (in case blobs with the same digest are sent). If this option is not passed, then the latest duplicated blob digest content will prevail in the set.
func BlobSetWithSkipNilBlobs ¶
func BlobSetWithSkipNilBlobs() BlobSetOption
BlobSetWithSkipNilBlobs allows passing nil blobs in the slice of blobs. The default behavior is that if you pass a nil blob in the slice, you'll get an error from the `NewBlobSet` constructor. If you pass this option, any nil blob will be skipped and the blob set will be built only from the non-nil ones.
type BucketOption ¶
type BucketOption func(*bucketOptions)
BucketOption are options passed when creating a new manifest bucket.
func BucketWithAllManifestBlobsValidation ¶
func BucketWithAllManifestBlobsValidation() BucketOption
BucketWithAllManifestBlobsValidation validates that all manifest digests have a corresponding blob in the blob set. If this option is not passed, then buckets with partial/incomplete blobs are allowed.
func BucketWithNoExtraBlobsValidation ¶
func BucketWithNoExtraBlobsValidation() BucketOption
BucketWithNoExtraBlobsValidation validates that the passed blob set has no additional blobs beyond the ones in the manifest.
type Digest ¶
type Digest struct {
// contains filtered or unexported fields
}
Digest represents a hash function's value.
func NewDigestFromBytes ¶
func NewDigestFromBytes(dtype DigestType, digest []byte) (*Digest, error)
NewDigestFromBytes builds a digest from a type and the digest bytes.
func NewDigestFromHex ¶
func NewDigestFromHex(dtype DigestType, hexstr string) (*Digest, error)
NewDigestFromHex builds a digest from a type and the hexadecimal string of the bytes. It returns an error if the received string is not a valid hex.
func NewDigestFromString ¶
NewDigestFromString build a digest from a string representation of it.
type DigestType ¶
type DigestType string
DigestType is the type for digests in this package.
const (
DigestTypeShake256 DigestType = "shake256"
)
type Digester ¶
Digester is something that can digest a content into a digest.
func NewDigester ¶
func NewDigester(dtype DigestType) (Digester, error)
NewDigester returns a digester of the requested type.
type Manifest ¶
type Manifest struct {
// contains filtered or unexported fields
}
Manifest represents a list of paths and their digests.
func NewFromReader ¶
NewFromReader builds a manifest from an encoded manifest, like one produced by Manifest.MarshalText.
func (*Manifest) AddEntry ¶
AddEntry adds an entry to the manifest with a path and its digest. It fails if the path already exists in the manifest with a different digest.
func (*Manifest) DigestFor ¶
DigestFor returns the matching digest for the given path. The path must be an exact match. Digest is nil and ok is false if no digest is found.
func (*Manifest) Digests ¶
Digests returns all unique digests in the manifest, order not guaranteed. If you want to iterate the paths and their digests, consider using `Range` instead.
func (*Manifest) MarshalText ¶
MarshalText encodes the manifest into its canonical form.
func (*Manifest) Paths ¶
Paths returns all unique paths in the manifest, order not guaranteed. If you want to iterate the paths and their digests, consider using `Range` instead.
func (*Manifest) PathsFor ¶
PathsFor returns one or more matching path for a given digest. The digest is expected to be a lower-case hex encoded value. Returned paths are unordered. Paths is nil and ok is false if no paths are found.
func (*Manifest) Range ¶
Range invokes a function for all the paths in the manifest, passing the path and its digest. The order in which the paths are iterated is not guaranteed. This func will stop iterating if an error is returned.
func (*Manifest) UnmarshalText ¶
UnmarshalText decodes a manifest from text.
See NewFromReader if your manifest is available in an io.Reader.
type MemoryBlobOption ¶
type MemoryBlobOption func(*memoryBlobOptions)
MemoryBlobOption are options passed when creating a new memory blob.
func MemoryBlobWithDigestValidation ¶
func MemoryBlobWithDigestValidation() MemoryBlobOption
MemoryBlobWithDigestValidation checks that the passed content and digest match.
Source Files