meta

package module
v0.0.0-...-75e6eb6 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 3, 2018 License: Apache-2.0 Imports: 20 Imported by: 0

README 

# Meta

The Meta package:

  - defines NSW State Archives approach to the construction of SIPS in the [OAIS model](https://www.oclc.org/research/publications/library/2000/lavoie-oais.html).
  - defines schemas for the metadata.json, manifest.json and json log files within these SIPS
  - is a software library that can be used by scripts in order to generate these SIPS
  - contains a set of generic metadata loaders and actions that can be used across projects
  - contains Meta cmd, a command line tool that demonstrates the generic loaders and can be used for very simple projects.

See the docs folder for more detail about the Digital State Archive storage and manifest file specifications.

Documentation

Overview

Package meta:

- defines NSW State Archives approach to the construction of SIPS in the [OAIS model](https://www.oclc.org/research/publications/library/2000/lavoie-oais.html). - defines schemas for the metadata.json, manifest.json and json log files within these SIPS - is a software library that can be used by scripts in order to generate these SIPS.

Index

Constants

View Source 
const (
	FBDTF      = "2006-01-02T15:04:05+0000"
	SlashDMY   = "2/01/2006"
	SlashYMD   = "2006/01/02"
	DashDMY    = "02-01-06"
	RelDTF     = SlashDMY + " 15:04" //relativity
	RelNoSlash = "20060102"
)

Add new date formats here, using the special Golang date value

View Source 
const (
	ModificationEvent = "http://id.loc.gov/vocabulary/preservation/eventType/mod"
	MigrationEvent    = "http://id.loc.gov/vocabulary/preservation/eventType/mig"
)

Variables

View Source 
var Cap int = 1000

Cap defines the capacity of the index slice. Edit for large jobs to an approximate number of objects

Functions

func IndexPath 

func IndexPath(m *Meta, index string) string

IndexPath is an example function that could be supplied to ManifestCopy IndexPath assumes that the index holds the full path to a file, so takes the dir of that path.

func MustInt 

func MustInt(s string) int

func NewDateTime 

func NewDateTime(t string) *time.Time

NewDate returns a reference to a time.Time from a W3C style datetime string. If the string provided is an invalid datetime, a nil reference is returned.

func ParseDateTime 

func ParseDateTime(t string) (time.Time, error)

ParseDateTime makes a time.Time from a W3C style datetime string

func ReadAll 

func ReadAll(path string, lq ...bool) ([][]string, error)

ReadAll is a helper function that opens a file at path and reads as a CSV. Provide an optional lazy quote bool if you'd like lazy quotes. Returns a slice of string slices and an error.

func ReferenceLog 

func ReferenceLog(i int) string

ReferenceLog makes a temporary reference to a log event. This reference is swapped for a UUID by the migrate tool.

func ReferenceMigration 

func ReferenceMigration(i int) string

ReferenceMigration makes a temporary reference to another migration (that doesn't yet have a UUID). This reference is swapped for a UUID by the migrate tool.

func ReferenceN 

func ReferenceN(refs ...Ref) string

ReferenceN constructs an internal reference string from the supplied Ref(s)

func ReferenceObject 

func ReferenceObject(i int) string

ReferenceObject makes a temporary reference to another object in the consignment. This reference is swapped for a UUID by the migrate tool.

func ReferenceVersion 

func ReferenceVersion(v int) string

func ToConsignment 

func ToConsignment(i int) string

ToConsignment turns a consignment number into an IRI @id

func ToID 

func ToID(i int, pat string) string

ToID is a helper function that turns an integer identififier into a string @id It is useful for places where we use integer IDs e.g. series numbers but want an IRI for the @id

func ToPUID 

func ToPUID(puid string) string

ToPUID is a helper function that turns a short PUID (e.g. fmt/1) into a fully qualified PUID (e.g. http://www.nationalarchives.gov.uk/pronom/fmt/1)

func ToRef 

func ToRef(i int, ref string) string

ToRef is a helper function that turns an integer identififier into a string ref:I reference. It is used for placedholder IDs like log:1 that get swapped out by the Migrate tool.

func ToSeries 

func ToSeries(i int) string

ToSeries turns a series number into an IRI @id

func ToSz 

func ToSz(s string) int64

ToSz is a helper function that turns a string int into an int64

Types

type AccessRule 

type AccessRule struct {
	ID           string  `json:"@id"`
	ExecuteDate  W3CDate `json:"executeDate"`
	Scope        string  `json:"scope"`
	Publish      bool    `json:"publish"`
	Basis        *Basis  `json:"basis,omitempty"`
	Patch        *int    `json:"metadataPatch,omitempty"`
	FullManifest *bool   `json:"fullManifest,omitempty"`
	Display      VarStr  `json:"displayTarget,omitempty"`
	Preview      VarStr  `json:"previewTarget,omitempty"`
	Text         VarStr  `json:"textTarget,omitempty"`
}

AccessRule represents accessRules

func AppendAR 

func AppendAR(rules []AccessRule,
	executeDate, scope string,
	publish bool,
	accessDirection int,
	accessDescription string,
	display, preview, text []FileTarget,
) ([]AccessRule, string, error)

AppendAR adds a new access rule to a set of rules (or create a new set of rules by appending to nil) Provide access rule fields in the supplied arguments. Because of their rarity, Patch and FullManifest fields aren't supplied as arguments but should be manipulated directly. Returns the list of access rules, the new access rule's @id, and an error.

func CopyARs 

func CopyARs(ar []AccessRule) []AccessRule

CopyARs copies a slice of AccessRules. This allows you to create a template access rule and copy it e.g. to change file references.

type Action 

type Action func(meta *Meta, target, index string) error

Action is a function (such as file copy) that is called when generating output

func Decompress 

func Decompress(sfpath string) Action

Decompress takes a path to a siegfried signature file and a pathfunc. The pathfunc returns the directory that will be joined to the filename out of the manifest. It returns an action that: - checks if a PUID (assuming a single version 0/ file 0) is a compressed type and recursively decompresses, - adding new files to manifest and copying them to output.

func ManifestCopy 

func ManifestCopy(pathfunc func(m *Meta, index string) string) Action

ManifestCopy copies files and versions as listed in the manifest Supply a pathfunc takes the Meta and index as parameters. The output of the pathfunc will be joined with the filename as listed in manifest.

func Progress 

func Progress(i int) Action

Progress prints progress message every n'th item processed

func SimpleManifest 

func SimpleManifest(fmtmap map[string][2]string, sfpath string) Action

SimpleManifest observes the files in the "versions" folder and builds a simple manifest based on that information. It takes a fmtmap argument and path to a siegfried file. The fmtmap links file extensions e.g. "pdf" to PUID + mimetype. It can be nil if you want siegfried identification only. The siegfried path can be an empty string if you don't want siegfried scanning.

type Agency 

type Agency struct {
	Name string
	ID   int
}

Agency loader. Applies a single agency creator to all digital objects

func (Agency) Load 

func (a Agency) Load(m *Meta) error

type Agent 

type Agent interface{}

Agent can be a string e.g. "Richard Lehane" or an object with @id/@type, or a slice of Agents

func AppendAgent 

func AppendAgent(a, b Agent) Agent

AppendAgent is a helper func to add agents to a slice of agents

func MakeAgency 

func MakeAgency(name string, id int) Agent

MakePerson creates an Agent that is of @type terms/Agency. Sets the @id to the supplied value.

func MakeAgent 

func MakeAgent(name, id, typ string) Agent

MakeAgent returns an Agent with the given name, @id and @type. If @id and @type aren't given, the Agent is a simple string e.g. "Richard Lehane"

func MakeOrganization 

func MakeOrganization(name string) Agent

MakePerson creates an Agent that is of @type schema.org/Organization.

func MakePerson 

func MakePerson(name string, id int) Agent

MakePerson creates an Agent that is of @type terms/Person. Sets the @id to the supplied value.

func MakeSDOPerson 

func MakeSDOPerson(name string) Agent

MakeSDOPerson creates an Agent that is of @type schema.org/Person. Does not set an @id.

func MakeSoftware 

func MakeSoftware(name, version string) Agent

MakeSoftware creates an Agent that is of @type https://schema.org/SoftwareApplication. Sets the version to the supplied value.

type Basis 

type Basis struct {
	AccessDirection   string `json:"accessDirection"`
	AccessDescription string `json:"accessDescription,omitempty"`
}

Basis is a json basis

type Business 

type Business struct {
	Typ                string   `json:"@type,omitempty"`
	LegalName          string   `json:"legalName,omitempty"`
	CommencedTrading   *W3CDate `json:"commencedTrading,omitempty"`
	CeasedTrading      *W3CDate `json:"ceasedTrading,omitempty"`
	RenewalDueDate     *W3CDate `json:"renewalDueDate,omitempty"`
	RegistrationNumber string   `json:"registrationNumber,omitempty"`
	ABN                string   `json:"abn,omitempty"`
	Proprietor         Agent    `json:"proprietor,omitempty"`
}

Business is a type of Thing. It is used for the BRS project

type Container 

type Container interface{}

Type container can be anything a metadata isPartOf

func Bundle 

func Bundle(title string) Container

func Email 

func Email(id string) Container

func Exhibit 

func Exhibit(title string) Container

func FileContainer 

func FileContainer(title string) Container

func MakeContainer 

func MakeContainer(title, id, typ string) Container

func MarkupSet 

func MarkupSet(title string) Container

type Context 

type Context map[string]Field

Context generates the @context field in json output

type Disposal 

type Disposal interface{}

Disposal can be a single DisposalRule{} or a slice of []DisposalRule{}

type DisposalRule 

type DisposalRule struct {
	Authority string `json:"authority"`
	Class     string `json:"class"`
}

func (DisposalRule) Load 

func (d DisposalRule) Load(m *Meta) error

DisposalRule loader. Applies a single disposal rule to all digital objects

type Field 

type Field interface{}

Fields are typically plain strings or objects with @id/@type

type File 

type File struct {
	ID             string     `json:"@id"`
	Name           string     `json:"name"`
	OriginalName   string     `json:"originalName,omitempty"`
	Size           int64      `json:"size"`
	Created        *time.Time `json:"fileCreated,omitempty"`
	Modified       *time.Time `json:"modified,omitempty"`
	MIME           string     `json:"mime,omitempty"`
	PUID           string     `json:"puid,omitempty"`
	Hash           *Hash      `json:"hash,omitempty"`
	HasAccessRules []string   `json:"hasAccessRules,omitempty"`
}

File represents files

type FileTarget 

type FileTarget [2]int

FileTarget is an internal reference for a file target

func (FileTarget) String 

func (ft FileTarget) String() string

String is a string representation of a FileTarget

type GlobalAccess 

type GlobalAccess struct {
	AccessDir    int
	AccessEffect string
	Execute      string
}

GlobalAccess loader. Applies a simple, global access rule to all digital objects

func (GlobalAccess) Load 

func (g GlobalAccess) Load(m *Meta) error

type Hash 

type Hash struct {
	Algorithm string `json:"hashAlgorithm,omitempty"`
	Value     string `json:"hashValue,omitempty"`
}

Hash is a json hash

type Loader 

type Loader interface {
	Load(m *Meta) error
}

Loader is anything that can load data into a Meta

type Log 

type Log struct {
	ID      string     `json:"@id"`
	Typ     string     `json:"@type"` // from http://id.loc.gov/vocabulary/preservation/eventType.html e.g. http://id.loc.gov/vocabulary/preservation/eventType/mig
	Start   *time.Time `json:"startTime,omitempty"`
	End     *time.Time `json:"endTime"`
	Detail  string     `json:"detail"`
	Agent   Agent      `json:"agent"`
	Context Context    `json:"@context"`
}

Log represents a preservation event e.g. format migration. The PROV and PREMIS ontologies are primarily used for this metadata.

func NewLog 

func NewLog(id int, typ string) *Log

NewLog creates a *Log

type Manifest 

type Manifest struct {
	Type        string       `json:"@type"`
	AccessRules []AccessRule `json:"accessRules,omitempty"`
	Versions    []Version    `json:"versions,omitempty"`
	Context     Context      `json:"@context"`
}

Manifest represents a manifest.json file

func NewManifest 

func NewManifest() *Manifest

NewManifest returns a reference to a manifest. It also sets the @type field.

func (*Manifest) AddAR 

func (m *Manifest) AddAR(
	executeDate, scope string,
	publish bool,
	accessDirection int,
	accessDescription string,
	display, preview, text []FileTarget,
) (arid string, err error)

AddAR adds a new access rule to a Manifest Provide access rule fields in the supplied arguments. Because of their rarity, Patch and FullManifest fields aren't supplied as arguments but should be manipulated directly. Returns the new access rule's @id and an error.

func (*Manifest) AddVersion 

func (m *Manifest) AddVersion(files []File) string

AddVersion adds a new version to a Manifest. It takes a slice of access rules, and a slice of files. It returns the version @id.

type Meta 

type Meta struct {
	SampleOff int
	SampleSz  int // sample size (-1 if doing a full run)
	Index     []string
	Metadata  map[string]*Metadata
	Manifest  map[string]*Manifest
	Logs      map[string][]*Log
	Store     map[string]interface{}
}

Meta is a set of metadata (metadata.json and manifest.json) The Index field provides ordering. The Store field can be used to store arbitrary data needed for particular projects.

func New 

func New(loaders ...Loader) (*Meta, error)

New creates a new meta from the supplied loaders

func NewSample 

func NewSample(offset, sample int, loaders ...Loader) (*Meta, error)

New Sample creates a new meta from the supplied loaders. For testing provide an offset where you'd like the sample to start e.g. 50 and a sample size e.g. 10. If a negative offset is provided then the offset will be calculated from the end. I.e. -10 will return the final 10.

func (*Meta) Output 

func (m *Meta) Output(target string, actions ...Action) error

Output generates metadata.json and manifest.json files for all of a Meta's metadata. Arbitrary actions based on that data can also be called by this function. Target is the target output directory.

type Metadata 

type Metadata struct {
	ID                string    `json:"@id"`
	Migration         string    `json:"migration"`
	Typ               VarStr    `json:"@type"`
	Title             string    `json:"title"`
	Description       string    `json:"description,omitempty"`
	Creator           Agent     `json:"creator,omitempty"`
	Created           *W3CDate  `json:"created,omitempty"`
	Modified          *W3CDate  `json:"modified,omitempty"`
	AgencyID          string    `json:"agencyIdentifier,omitempty"` // original @id used within agency e.g. TRANS.01.01
	Provenance        string    `json:"provenance,omitempty"`
	Source            VarStr    `json:"source,omitempty"`
	IsPartOf          Container `json:"isPartOf,omitempty"`
	DeliveryMethod    string    `json:"deliveryMethod,omitempty"`
	DocumentType      string    `json:"documentType,omitempty"` // document genre e.g. Research, Correspondence
	Series            string    `json:"series,omitempty"`
	Consignment       string    `json:"consignment,omitempty"`
	DisposalRule      Disposal  `json:"disposalRule,omitempty"`
	Duration          string    `json:"duration,omitempty"`
	Language          VarStr    `json:"language,omitempty"`
	Subtitles         VarStr    `json:"subtitles,omitempty"`
	Director          VarStr    `json:"director,omitempty"`
	Actor             VarStr    `json:"actor,omitempty"`
	ProductionCompany VarStr    `json:"productionCompany,omitempty"`
	About             Thing     `json:"about,omitempty"`
	Context           Context   `json:"@context"`
}

Metadata represents a metadata.json file

func NewMetadata 

func NewMetadata(id int, title string) *Metadata

NewMetadata returns a Metadata with the supplied title. It also sets the @type.

func (*Metadata) AddType 

func (m *Metadata) AddType(typ string)

Metadata can have multiple types e.g. both an DigitalArchive and a Movie

type Obj 

type Obj struct {
	ID              string `json:"@id,omitempty"`
	Typ             string `json:"@type,omitempty"`
	Container       string `json:"@container,omitempty"`
	Name            string `json:"name,omitempty"`
	Title           string `json:"title,omitempty"`
	SoftwareVersion string `json:"softwareVersion,omitempty"`
}

Obj is a json object. Used with @id/@type in @context. Can also be used to generate generic objects e.g. Agents and containers in metadata are Objs

type Ref 

type Ref struct {
	Prefix string
	N      int
}

Ref supports the construction of internal reference strings (blank node IDs) e.g. _:ar1

type Series 

type Series int

Series loader. Applies a single series to all digital objects

func (Series) Load 

func (s Series) Load(m *Meta) error

type Siegfried 

type Siegfried struct {
	Blacklist []string
	reader.Reader
}

Siegfried loader. Reads a siegfried file (or droid or fido) to generate generic digital objects

func NewSiegfried 

func NewSiegfried(rdr io.Reader, blacklist ...string) (*Siegfried, error)

NewSiegfried takes an io Reader for a siegfried/droid/fido results file and an optional blacklist. The blacklist is IDs you'd like to exclude e.g. to prune thumbs db files

func (*Siegfried) Load 

func (s *Siegfried) Load(m *Meta) error

type Thing 

type Thing interface{}

Thing can be anything that a metadata is "about"

func MakeBusiness 

func MakeBusiness(legalName, registrationNumber, abn string, commencedTrading, ceasedTrading, renewalDueDate time.Time, proprietors ...string) Thing

MakeBusiness returns a Thing of @type schema.org/Organization and sets the supplied fields. A variable number of proprietors can be supplied and these will be set as a slice of Organizations.

type TitleFn 

type TitleFn func(m *Meta, index string) string

Title func loader. Allows you to customise a title based on other metadata (e.g. manipulate the file name to make a title)

func (TitleFn) Load 

func (t TitleFn) Load(m *Meta) error

type VarStr 

type VarStr interface{}

VarStr represents a single string or a slice of strings

func ReferenceFiles 

func ReferenceFiles(ts []FileTarget) VarStr

func SetVarStr 

func SetVarStr(v VarStr) VarStr

type Version 

type Version struct {
	ID             string   `json:"@id"`
	Base           string   `json:"base,omitempty"`
	DerivedFrom    string   `json:"derivedFrom,omitempty"`
	GeneratedBy    string   `json:"generatedBy,omitempty"`
	HasAccessRules []string `json:"hasAccessRules,omitempty"`
	Files          []File   `json:"files"`
}

Version represents versions

type W3CDate 

type W3CDate struct {
	time.Time
	// contains filtered or unexported fields
}

W3CDate contains a time.Time but marshals to json in form yyyy-mm-dd

func NewDate 

func NewDate(d string) *W3CDate

NewDate returns a reference to W3CDate from a W3C style date string. If the string provided is an invalid date, a nil reference is returned.

func NewDateLayout 

func NewDateLayout(layout, d string) *W3CDate

NewDate returns a reference to W3CDate from a time layout string and a date in that layout string. If the string provided is an invalid date, a nil reference is returned.

func ParseDate 

func ParseDate(d string) (W3CDate, error)

ParseDate makes a W3CDate from a W3C style date string

func ParseDateLayout 

func ParseDateLayout(layout, d string) (*W3CDate, error)

ParseDateLayout returns a reference to W3CDate from a time layout string and a date in that layout string. If the string provided is an invalid date, an error is returned

func WrapDate 

func WrapDate(t time.Time) *W3CDate

WrapDate allows you to create a *W3CDate (with YMD precision) when you already have a *time.Time

func (W3CDate) MarshalJSON 

func (d W3CDate) MarshalJSON() ([]byte, error)

MarshalJSON makes W3CDate a json Marshaller with yyyy-mm-dd output

func (W3CDate) String 

func (d W3CDate) String() string

String writes date with a yyyy-mm-dd output, depeding on precision

Source Files

View all Source files

Directories

Path Synopsis
cmd
meta
Meta cmd provides a simple tool for creating SIPS from the command line.
 Meta cmd provides a simple tool for creating SIPS from the command line.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.