Documentation ¶
Overview ¶
This module is an implementation of the Oxford Common File Layout (OCFL) specification. The top-level package provides version-independent functionality. The ocflv1 package provides the bulk of implementation.
Index ¶
- Constants
- Variables
- func Copy(ctx context.Context, dstFS WriteFS, dst string, srcFS FS, src string) (err error)
- func DigestConcurrency() int
- func DigestFS(ctx context.Context, fsys FS, ...) error
- func Files(ctx context.Context, fsys FS, pth PathSelector, fn func(name string) error) error
- func ObjectRoots(ctx context.Context, fsys FS, sel PathSelector, fn func(*ObjectRoot) error) error
- func ParseVNum(v string, vn *VNum) error
- func ReadNamaste(ctx context.Context, fsys FS, name string) error
- func RegisterAlg(alg string, newDigester func() Digester)
- func RegisteredAlgs() []string
- func SetDigestConcurrency(i int)
- func SetXferConcurrency(i int)
- func WriteDeclaration(ctx context.Context, root WriteFS, dir string, d Namaste) error
- func WriteSpecFile(ctx context.Context, fsys WriteFS, dir string, n Spec) (string, error)
- func XferConcurrency() int
- type ContentSource
- type CopyFS
- type DigestErr
- type DigestMap
- func (m DigestMap) Clone() DigestMap
- func (m DigestMap) Digests() []string
- func (m DigestMap) EachPath(fn func(pth, digest string) bool) bool
- func (m DigestMap) Eq(other DigestMap) bool
- func (m DigestMap) GetDigest(p string) string
- func (m DigestMap) HasDigestCase() (hasLower bool, hasUpper bool)
- func (m1 DigestMap) Merge(m2 DigestMap, replace bool) (DigestMap, error)
- func (m DigestMap) Normalize() (norm DigestMap, err error)
- func (m DigestMap) NumPaths() int
- func (m DigestMap) PathMap() PathMap
- func (m DigestMap) PathMapValid() (PathMap, error)
- func (m DigestMap) Paths() []string
- func (m DigestMap) Remap(fns ...RemapFunc)
- func (m DigestMap) Valid() error
- type DigestSet
- type Digester
- type FS
- type FileIterator
- type FixitySource
- type InvType
- type MapDigestConflictErr
- type MapPathConflictErr
- type MapPathInvalidErr
- type MultiDigester
- type Namaste
- type ObjectRoot
- type ObjectRootFlag
- type ObjectRootIterator
- type PathMap
- type PathSelector
- type RemapFunc
- type Spec
- type Stage
- type User
- type VNum
- func (v VNum) AsHead() VNums
- func (v VNum) First() bool
- func (v VNum) IsZero() bool
- func (v VNum) MarshalText() ([]byte, error)
- func (v VNum) Next() (VNum, error)
- func (v VNum) Num() int
- func (v VNum) Padding() int
- func (v VNum) Prev() (VNum, error)
- func (v VNum) String() string
- func (v *VNum) UnmarshalText(text []byte) error
- func (v VNum) Valid() error
- type VNums
- type WriteFS
Constants ¶
const ( SHA512 = `sha512` SHA256 = `sha256` SHA1 = `sha1` MD5 = `md5` BLAKE2B = `blake2b-512` )
const ( NamasteTypeObject = "ocfl_object" // type string for OCFL Object declaration NamasteTypeStore = "ocfl" // type string for OCFL Storage Root declaration )
const ( // package version Version = "0.0.25" ExtensionsDir = "extensions" )
const ( Spec1_0 = Spec("1.0") Spec1_1 = Spec("1.1") )
Variables ¶
var ( ErrNoNamaste = fmt.Errorf("missing NAMASTE declaration: %w", fs.ErrNotExist) ErrNamasteInvalid = errors.New("invalid NAMASTE declaration contents") ErrNamasteMultiple = errors.New("multiple NAMASTE declarations found") )
var ( ErrObjectNotFound = fmt.Errorf("missing object declaration: %w", ErrNoNamaste) ErrObjectExists = fmt.Errorf("existing OCFL object declaration: %w", fs.ErrExist) )
var ( ErrSpecInvalid = errors.New("invalid OCFL spec version") ErrSpecNotFound = errors.New("OCFL spec file not found") )
var ( ErrVNumInvalid = errors.New(`invalid version`) ErrVNumPadding = errors.New(`inconsistent version padding in version sequence`) ErrVNumMissing = errors.New(`missing version in version sequence`) ErrVerEmpty = errors.New("no versions found") // Some functions in this package use the zero value VNum to indicate the // most recent, "head" version. Head = VNum{} )
var ErrMapMakerExists = errors.New("path and digest exist")
ErrMapMakerExists is returned when calling Add with a path and digest that are already present in the MapMaker
var ErrNotFile = errors.New("not a file")
var ErrUnknownAlg = errors.New("unknown digest algorithm")
Functions ¶
func Copy ¶ added in v0.0.23
Copy copies src in srcFS to dst in dstFS. If srcFS and dstFS are the same refererence and it implements CopyFS, then Copy uses the fs's Copy() method.
func DigestConcurrency ¶
func DigestConcurrency() int
DigestConcurrency is a global configuration for the number of files to digest concurrently.
func DigestFS ¶
func DigestFS(ctx context.Context, fsys FS, setupFunc func(addFile func(string, ...string) bool), resultFn func(string, DigestSet, error) error) error
DigestFS concurrently digests files in an FS. The setup function adds files to the work quue using the addFile function passed to it. addFile returns a bool indicating if the file was added to the queue. Results are passed back using the result function. If resultFn returns an error, not more results will be produced, and new calls to addFile will return false. DigestFS uses the value from DigestConcurrency() to determine to set the number of files that are digested concurrently.
func ObjectRoots ¶
func ObjectRoots(ctx context.Context, fsys FS, sel PathSelector, fn func(*ObjectRoot) error) error
ObjectRoots searches root and its subdirectories for OCFL object declarations and calls fn for each object root it finds. The *ObjectRoot passed to fn is confirmed to have an object declaration, but no other validation checks are made.
func ReadNamaste ¶ added in v0.0.24
ReadNamaste validates a namaste declaration
func RegisterAlg ¶ added in v0.0.21
RegisterAlg registers the Digester constructor for alg, so that alg.New() can be used.
func RegisteredAlgs ¶ added in v0.0.22
func RegisteredAlgs() []string
RegisteredAlgs returns a slice of all available digest algorithms
func SetDigestConcurrency ¶
func SetDigestConcurrency(i int)
SetDigestConcurrency sets the max number of files to digest concurrently.
func SetXferConcurrency ¶
func SetXferConcurrency(i int)
SetXferConcurrency sets the maximum number of files transferred concurrently during a commit operation.
func WriteDeclaration ¶
func WriteSpecFile ¶
func XferConcurrency ¶
func XferConcurrency() int
XferConcurrency is a global configuration for the maximum number of files transferred concurrently during a commit operation. It defaults to runtime.NumCPU().
Types ¶
type ContentSource ¶ added in v0.0.23
type ContentSource interface { // GetContent returns an FS and path to a file in FS for a file with the given digest. // If no content is associated with the digest, fsys is nil and path is an empty string. GetContent(digest string) (fsys FS, path string) }
ContentSource is used to access content with a given digest when creating and upadting objects.
type CopyFS ¶
type CopyFS interface { WriteFS // Copy creates or updates the file at dst with the contents of src. If dst // exists, it should be overwritten Copy(ctx context.Context, dst string, src string) error }
CopyFS is a storage backend that supports copying files.
type DigestErr ¶
type DigestErr struct { Name string // Content path Alg string // Digest algorithm Got string // Calculated digest Expected string // Expected digest }
DigestErr is returned when content's digest conflicts with an expected value
type DigestMap ¶
DigestMap maps digests to file paths.
func (DigestMap) Digests ¶
Digests returns a slice of the digest values in the DigestMap. Digest strings are not normalized; they may be uppercase, lowercase, or mixed.
func (DigestMap) EachPath ¶
EachPath calls fn for each path in the Map. If fn returns false, iteration stops and EachPath returns false.
func (DigestMap) Eq ¶
Eq returns true if m and the other have the same content: they have the same (normalized) digests corresponding to the same set of paths. If either map has a digest conflict (same digest appears twice with different case), Eq returns false.
func (DigestMap) GetDigest ¶
GetDigest returns the digest for path p or an empty string if the digest is not present.
func (DigestMap) HasDigestCase ¶ added in v0.0.23
HasDigestCase returns two booleans indicating whether m's digests use lowercase and uppercase characters.
func (DigestMap) Merge ¶
Merge returns a new DigestMap constructed by normalizing and merging m1 and m2. If a paths has different digests in m1 and m2, an error returned unless replace is true, in which case the value from m2 is used.
func (DigestMap) Normalize ¶
Normalize returns a normalized copy on m (with lowercase digests). An error is returned if m has a digest conflict.
func (DigestMap) PathMap ¶
PathMap returns the DigestMap's contents as a map with path names for keys and digests for values. PathMap doesn't check if the same path appears twice in the DigestMap.
func (DigestMap) PathMapValid ¶
PathMapValid is like PathMap, except it returns an error if it encounters invalid path names or if the same path appears multiple times.
type DigestSet ¶
Set is a set of digest results
func (DigestSet) ConflictWith ¶
ConflictWith returns keys in s with values that do not match the corresponding key in other.
type Digester ¶
type Digester interface { io.Writer // String() returns the digest value for the bytes written to the digester. String() string }
Digester is an interface used for generating digest values.
func NewDigester ¶ added in v0.0.22
New returns a new Digester for generated digest values. If a Digester constructor was not registered for a, nil is returne.
type FS ¶
type FS interface { OpenFile(ctx context.Context, name string) (fs.File, error) ReadDir(ctx context.Context, name string) ([]fs.DirEntry, error) }
FS is a minimal, read-only storage layer abstraction. It is similar to the standard library's io/fs.FS, except it uses contexts and OpenFile is not required to gracefully handle directories.
type FileIterator ¶
type FileIterator interface { FS // Files calls a function for each filename satisfying the path selector. // The function should only be called for "regular" files (never for // directories or symlinks). Files(context.Context, PathSelector, func(name string) error) error }
FileIterator is used to iterate over regular files in an FS
type FixitySource ¶ added in v0.0.23
type FixitySource interface { // GetFixity returns a DigestSet with alternate digests for the content with // the digest derrived using the stage's primary digest algorithm. GetFixity(digest string) DigestSet }
FixitySource is used to access alternate digests for content with a given digest (sha512 or sha256) when creating or updating objects.
type InvType ¶
type InvType struct {
Spec
}
InvType represents an inventory type string for example: https://ocfl.io/1.0/spec/#inventory
func (InvType) MarshalText ¶
func (*InvType) UnmarshalText ¶
type MapDigestConflictErr ¶
type MapDigestConflictErr struct {
Digest string
}
MapDigestConflictErr indicates same digest found multiple times in the digest map (i.e., with different cases)
func (*MapDigestConflictErr) Error ¶
func (d *MapDigestConflictErr) Error() string
type MapPathConflictErr ¶
type MapPathConflictErr struct {
Path string
}
MapPathConflictErr indicates a path appears more than once in the digest map. It's also used in cases where the path as used as a directory in one instance and a file in another.
func (*MapPathConflictErr) Error ¶
func (p *MapPathConflictErr) Error() string
type MapPathInvalidErr ¶
type MapPathInvalidErr struct {
Path string
}
MapPathInvalidErr indicates an invalid path in a Map.
func (*MapPathInvalidErr) Error ¶
func (p *MapPathInvalidErr) Error() string
type MultiDigester ¶
MultiDigester is used to generate digests for multiple digest algorithms at the same time.
func NewMultiDigester ¶
func NewMultiDigester(algs ...string) *MultiDigester
func (MultiDigester) Sum ¶ added in v0.0.22
func (md MultiDigester) Sum(alg string) string
func (MultiDigester) Sums ¶
func (md MultiDigester) Sums() DigestSet
Sums returns a DigestSet with all digest values for the MultiDigester
type Namaste ¶ added in v0.0.24
Namaste represents a NAMASTE declaration
func FindNamaste ¶ added in v0.0.24
FindNamaste returns the Namasted declaration from a fs.DirEntry slice. An error is returned if the number of declarations is not one.
func ParseNamaste ¶ added in v0.0.24
type ObjectRoot ¶
type ObjectRoot struct { FS FS // the FS where the object is stored Path string // object path in FS Spec Spec // the OCFL spec from the object's NAMASTE declaration VersionDirs VNums // versions directories found in the object directory SidecarAlg string // digest algorithm declared by the inventory sidecar NonConform []string // non-conforming entries found in the object root (max=8) Flags ObjectRootFlag }
ObjectRoot represents an existing OCFL object root directory. Instances are typically created with functions like GetObjectRoot().
func GetObjectRoot ¶
GetObjectRoot reads the contents of directory dir in fsys, confirms that an OCFL Object declaration is present, and returns a new ObjectRoot reference based on the directory contents. If the directory cannot be read or a declaration is not found, ErrObjectNotFound is returned. Note, the object declaration is not read or fully validated. The returned ObjectRoot will have the FoundNamaste flag set, but other flags expected for a complete object root may not be set (e.g., if the inventory is missing).
func NewObjectRoot ¶
func NewObjectRoot(fsys FS, dir string, entries []fs.DirEntry) *ObjectRoot
NewObjectRoot constructs an ObjectRoot for the directory dir in fsys using the given fs.DirEntry slice as dir's contents. The returned ObjectRoot may be invalid.
func (ObjectRoot) HasExtensions ¶ added in v0.0.24
func (obj ObjectRoot) HasExtensions() bool
HasExtensions returns true if the object's HasExtensions flag is set
func (ObjectRoot) HasInventory ¶
func (obj ObjectRoot) HasInventory() bool
HasInventory returns true if the object's FoundInventory flag is set
func (ObjectRoot) HasNamaste ¶ added in v0.0.24
func (obj ObjectRoot) HasNamaste() bool
HasNamaste returns true if the object's FoundDeclaration flag is set
func (ObjectRoot) HasSidecar ¶
func (obj ObjectRoot) HasSidecar() bool
HasSidecar returns true if the object's FoundSidecar flag is set
func (ObjectRoot) HasVersionDir ¶
func (obj ObjectRoot) HasVersionDir(dir VNum) bool
func (ObjectRoot) ValidateNamaste ¶ added in v0.0.24
func (obj ObjectRoot) ValidateNamaste(ctx context.Context) error
ValidateNamaste reads and validates the contents of the OCFL object declaration in the object root.
type ObjectRootFlag ¶
type ObjectRootFlag int
const ( // HasNamaste indicates that an ObjectRoot has been initialized // and an object declaration file is confirmed to exist in the object's root // directory HasNamaste ObjectRootFlag = 1 << iota // HasInventory indicates that an ObjectRoot includes an "inventory.json" // file HasInventory // HasSidecar indicates that an ObjectRoot includes an "inventory.json.*" // file (the inventory sidecar). HasSidecar // HasExtensions indicates that an ObjectRoot includes a directory // named "extensions" HasExtensions )
type ObjectRootIterator ¶
type ObjectRootIterator interface { // ObjectRoots searches root and its subdirectories for OCFL object declarations // and calls fn for each object root it finds. The *ObjectRoot passed to fn is // confirmed to have an object declaration, but no other validation checks are // made. ObjectRoots(ctx context.Context, sel PathSelector, fn func(obj *ObjectRoot) error) error }
ObjectRootIterator is used to iterate over object roots
type PathMap ¶ added in v0.0.23
PathMap maps filenames to digest strings.
func (PathMap) DigestMap ¶ added in v0.0.23
DigestMap returns a new DigestMap using the pathnames and digests in pm. The resulting DigestMap may be invalid if pm includes invalid paths or digests.
func (PathMap) DigestMapValid ¶ added in v0.0.23
DigestMap returns a new DigestMap using the pathnames and digests in pm. If the resulting DigestMap is invalid, an error is returned.
type PathSelector ¶
type PathSelector struct { // Dir is a parent directory for all paths that satisfy the selector. All // paths in the selection match Dir or have Dir as a parent (prefix). If Dir // is not a well-formed path (see fs.ValidPath), then no path names will // satisfy the path selector. There is one exception: The empty string is // converted to "." by consumers of the selector using Path(). Dir string // SkipDirFn is used to skip directories during an iteration process. If the // function returns true for a given path, the directory's contents will be // skipped. The string parameter is always a directory name relative to an // FS. SkipDirFn func(dir string) bool }
PathSelector is used to configure iterators that walk an FS. See FileIterator and ObjectRootIterator.
func Dir ¶
func Dir(name string) PathSelector
Dir is a convenient way to construct a PathSelector for a given directory.
func (PathSelector) Path ¶
func (ps PathSelector) Path() string
Path returns name as a valid path or an empty string if name is not a valid path
func (PathSelector) SkipDir ¶
func (ps PathSelector) SkipDir(name string) bool
SkipDir returns true if dir should be skipped during an interation process that uses the path selector
type Spec ¶
type Spec string
Spec represent an OCFL specification number
type Stage ¶
type Stage struct { // State is a DigestMap representing the new object version state. State DigestMap // DigestAlgorithm is the primary digest algorithm (sha512 or sha256) used by the stage // state. DigestAlgorithm string // ContentSource is used to access new content needed to construct // an object. It may be nil ContentSource // FixitySource is used to access fixity information for new // content. It may be nil FixitySource }
Stage is used to create/update objects.
func StageBytes ¶ added in v0.0.23
StageBytes builds a stage from a map of filenames to file contents
func StageDir ¶ added in v0.0.23
StageDir builds a stage based on the contents of the directory dir in
func (Stage) HasContent ¶ added in v0.0.23
HasContent returns true if the stage's content source provides an FS and path for the digest
type VNum ¶
type VNum struct {
// contains filtered or unexported fields
}
VNum represents an OCFL object version number (e.g., "v1", "v02"). A VNum has a sequence number (1,2,3...) and a padding number, which defaults to zero. The padding is the maximum number of numeric digits the version number can include (a padding of 0 is no maximum). The padding value constrains the maximum valid sequence number.
func MustParseVNum ¶
MustParseVNum parses str as a VNUm and returns a new VNum. It panics if str cannot be parsed as a VNum.
func V ¶
V returns a new Vnum. The first argument is a sequence number. An optional second argument can be used to set the padding. Additional arguments are ignored. Without any arguments, V() returns a zero value VNum.
func (VNum) MarshalText ¶
func (VNum) Next ¶
Next returns the next ocfl.VNum after v with the same padding. A non-nil error is returned if padding > 0 and next would overflow the padding
func (VNum) Prev ¶
Prev returns the previous version before v, with the same padding. An error is returned if v.Num() == 1
func (*VNum) UnmarshalText ¶
type VNums ¶
type VNums []VNum
VNums is a slice of VNum elements
type WriteFS ¶
type WriteFS interface { FS Write(ctx context.Context, name string, buffer io.Reader) (int64, error) // Remove the file with path name Remove(ctx context.Context, name string) error // Remove the directory with path name and all its contents. If the path // does not exist, return nil. RemoveAll(ctx context.Context, name string) error }
WriteFS is a storage backend that supports write and remove operations.
Source Files ¶
Directories ¶
Path | Synopsis |
---|---|
backend
|
|
examples
|
|
internal
|
|
pathtree
Package pathree provides Node[T] and generic functions used for storing arbitrary values in a hierarchical data structure following filesystem naming conventions.
|
Package pathree provides Node[T] and generic functions used for storing arbitrary values in a hierarchical data structure following filesystem naming conventions. |
Package [ocflv1] provides an implementation of OCFL v1.0 and v1.1.
|
Package [ocflv1] provides an implementation of OCFL v1.0 and v1.1. |