README
¶
dsv1
This module provides limited read support for v1 based dataset collections. It is intended to enable migration of data from a version 1 dataset collection to the current version of the dataset collection.
Notes
Version 1 dataset collections were exclusively pairtree based. The JSON objects are stored in the pairtree and the attached documents under that directory. The operational metadata as well as the general metadata is maintained in the collection.json file in the root collection folder. Frame data is also in the root folder. Frames do not need to migrate, they tend to be operationally empheral in practice. The JSON documents and their attachments do need to migrate.
If the version 1 collection is in good working order then the collections.json file can be used to remap the old collection to the new one. On the otherhand if it is not in good working order the pairtree itself can be used to derive the keys and for the objects in the collection. Leverage the pairtree itself is safeist way to proceed.
Documentation
¶
Overview ¶
dsv1 is a submodule of dataset package.
Authors R. S. Doiel, <rsdoiel@library.caltech.edu> and Tom Morrel, <tmorrell@library.caltech.edu>
Copyright (c) 2022, Caltech All rights not granted herein are expressly reserved by Caltech.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Package dataset includes the operations needed for processing collections of JSON documents and their attachments.
Authors R. S. Doiel, <rsdoiel@library.caltech.edu> and Tom Morrel, <tmorrell@library.caltech.edu>
Copyright (c) 2022, Caltech All rights not granted herein are expressly reserved by Caltech.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Index ¶
- Constants
- func Analyzer(collectionName string, verbose bool) error
- func IsCollection(p string) bool
- func SetupV1TestCollection(cName string, records map[string]map[string]interface{}) error
- type Collection
- func (c *Collection) Close() error
- func (c *Collection) DocPath(name string) (string, error)
- func (c *Collection) IsKeyNotFound(e error) bool
- func (c *Collection) KeyExists(key string) bool
- func (c *Collection) Keys() []string
- func (c *Collection) Length() int
- func (c *Collection) MetadataJSON() []byte
- func (c *Collection) Read(name string, data map[string]interface{}, cleanObject bool) error
- func (c *Collection) ReadJSON(name string) ([]byte, error)
- type PersonOrOrg
Constants ¶
const ( // Version 1.1.1-legacy Version = `1.1.1-legacy` // Asc is used to identify ascending sorts Asc = iota // Desc is used to identify descending sorts Desc = iota )
Variables ¶
This section is empty.
Functions ¶
func Analyzer ¶
Analyzer checks the collection version and analyzes current state of collection reporting on errors.
func IsCollection ¶
IsCollection checks to see if a given path contains a collection.json file
Types ¶
type Collection ¶
type Collection struct {
// DatasetVersion of the collection
DatasetVersion string `json:"dataset,omitempty"`
// Name (filename) of collection
Name string `json:"name"`
// KeyMap holds the document key to path in the collection
KeyMap map[string]string `json:"keymap,omitempty"`
// FrameMap is a list of frame names and with rel path to the frame defined in the collection
FrameMap map[string]string `json:"frames,omitempty"`
// Description describes what is in the collection.
Description string `json:"description,omitempty"`
// Created is the date/time the init command was run in
// RFC1123 format.
Created string `json:"created,omitempty"`
// Version of collection being stored in semvar notation
Version string `json:"version,omitempty"`
// Contact info
Contact string `json:"contact,omitempty"`
// Author holds a list of PersonOrOrg
Author []*PersonOrOrg `json:"author,omitempty"`
// Contributors holds a list of PersonOrOrg
Contributor []*PersonOrOrg `json:"contributor,omitempty"`
// Funder holds a list of PersonOrOrg
Funder []*PersonOrOrg `json:"funder,omitempty"`
// DOI holds the digital object identifier if defined.
DOI string `json:"doi,omitempty"`
// License holds a pointer to the license information for
// the collection. E.g. CC0 URL
License string `json:"license,omitempty"`
// Annotation is a map to any addition metadata associated with
// the Collection's metadata.
Annotation map[string]interface{} `json:"annotation,omitempty"`
// Who is the person(s)/organization(s) that created the collection
Who []string `json:"who,omitempty"`
// What - description of collection
What string `json:"what,omitempty"`
// When - date associated with collection (e.g. 2021,
// 2021-10, 2021-10-02), should map to an approx date like in
// archival work.
When string `json:"when,omitempty"`
// Where - location (e.g. URL, address) of collection
Where string `json:"where,omitempty"`
// contains filtered or unexported fields
}
Collection is the container holding a pairtree containing JSON docs
func Open ¶
func Open(name string) (*Collection, error)
Open reads in a collection's metadata and returns and new collection structure or error.
```
var (
c *Collection
err error
)
c, err = dataset.Open("collection.ds")
if err != nil {
// ... handle error
}
defer c.Close()
```
func (*Collection) Close ¶
func (c *Collection) Close() error
Close closes a collection, writing the updated keys to disc Close removes the "lock.pid" file in the collection root. Close is often called in conjunction with "defer" keyword.
```
c, err := dataset.Open("my_collection.ds")
if err != nil { // .. handle error ...
}
// do some stuff with the collection
if err := c.Close(); err != nil {
// ... handle closing error ...
}
```
func (*Collection) DocPath ¶
func (c *Collection) DocPath(name string) (string, error)
DocPath returns a full path to a key or an error if not found
```
c, err := dataset.Open("my_collection.ds")
if err != nil {
// ... handle error ...
}
defer c.Close()
key := "my-object-key"
docPath := c.DocPath(key)
```
func (*Collection) IsKeyNotFound ¶
func (c *Collection) IsKeyNotFound(e error) bool
IsKeyNotFound checks an error message and returns true if it is a key not found error.
func (*Collection) KeyExists ¶
func (c *Collection) KeyExists(key string) bool
KeyExists returns true if key is in collection's KeyMap, false otherwise
var (
c *dataset.Collection
)
// ... collection previously opened and assigned to "c" ...
key := "object-1"
if c.KeyExists(key) == true {
// ... do something with the key ...
}
```
func (*Collection) Keys ¶
func (c *Collection) Keys() []string
Keys returns a list of keys in a collection
var (
c *dataset.Collection
keys []string
)
// ... collection previously opened and assigned to "c" ...
keys := c.Keys()
for _, key := range keys {
// ... do something with the list of keys ...
}
```
func (*Collection) Length ¶
func (c *Collection) Length() int
Length returns the number of keys in a collection
var ( c *dataset.Collection ) // ... collection previously opened and assigned to "c" ... l := c.Length() // ... do something with the number of itemsin the collection ...
```
func (*Collection) MetadataJSON ¶
func (c *Collection) MetadataJSON() []byte
MetadataJSON() returns a collection's metadata fields as a JSON encoded byte array.
func (*Collection) Read ¶
func (c *Collection) Read(name string, data map[string]interface{}, cleanObject bool) error
Read finds the record in a collection, updates the data interface provide and if problem returns an error name must exist or an error is returned
```
var (
c *dataset.Collection
)
// ... collection previously opened and assigned to "c" ...
key := "object-2"
obj, err := c.Read(key)
if err != nil { // ... handle error ...
}
```
func (*Collection) ReadJSON ¶
func (c *Collection) ReadJSON(name string) ([]byte, error)
ReadJSON finds a the record in the collection and returns the JSON source or an error.
```
var (
c *Collection
)
// ... collection previously opened and assigned to "c" ...
key := "object-1"
src, err := c.ReadJSON(key)
if err != nil {
// ... handle error ...
}
// ... do something with the JSON encoded "src" value ...
```
type PersonOrOrg ¶
type PersonOrOrg struct {
// Type is either "Person" or "Organization"
Type string `json:"@type,omitempty"`
// ID is either an ORCID or ROR
ID string `json:"@id,omitempty"`
// Name of an organization, empty if person
Name string `json:"name,omitempty"`
// Given name for a person, empty of organization
GivenName string `json:"givenName,omitempty"`
// Family name for a person, empty of organization
FamilyName string `json:"familyName,omitempty"`
// Affiliation holds the intitution affiliation of a person.
Affiliation []*PersonOrOrg `json:"affiliation,omitempty"`
// Annotation holds custom fields, e.g. a grant number of a funder
Annotation map[string]interface{} `json:"annotation,omitempty"`
}
PersonOrOrg holds a the description of a person or organizaion associated with the dataset collection. e.g. author, contributor or funder.
Source Files
Directories