dsfs

package
v0.1.4 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 4, 2019 License: MIT Imports: 21 Imported by: 0

Documentation

Overview

Package dsfs glues datsets to cafs (content-addressed-file-system)

Index

Constants

This section is empty.

Variables

View Source
var ErrNoTransform = fmt.Errorf("this dataset has no transform component")

ErrNoTransform is the error for asking a dataset without a tranform component for viz info

View Source
var ErrNoViz = fmt.Errorf("this dataset has no viz component")

ErrNoViz is the error for asking a dataset without a viz component for viz info

View Source
var Timestamp = func() time.Time {
	return time.Now().UTC()
}

Timestamp is an function for getting commit timestamps timestamps MUST be stored in UTC time zone

Functions

func CreateDataset

func CreateDataset(store cafs.Filestore, ds, dsPrev *dataset.Dataset, pk crypto.PrivKey, pin, force, shouldRender bool) (path string, err error)

CreateDataset places a new dataset in the store. Admittedly, this isn't a simple process. Store is where we're going to Dataset to be saved Pin the dataset if the underlying store supports the pinning interface All streaming files (Body, Transform Script, Viz Script) Must be Resolved before calling if data their data is to be saved

func DerefDataset

func DerefDataset(store cafs.Filestore, ds *dataset.Dataset) error

DerefDataset attempts to fully dereference a dataset

func DerefDatasetCommit

func DerefDatasetCommit(store cafs.Filestore, ds *dataset.Dataset) error

DerefDatasetCommit derferences a dataset's Commit element if required should be a no-op if ds.Structure is nil or isn't a reference

func DerefDatasetMeta

func DerefDatasetMeta(store cafs.Filestore, ds *dataset.Dataset) error

DerefDatasetMeta derferences a dataset's transform element if required should be a no-op if ds.Structure is nil or isn't a reference

func DerefDatasetStructure

func DerefDatasetStructure(store cafs.Filestore, ds *dataset.Dataset) error

DerefDatasetStructure derferences a dataset's structure element if required should be a no-op if ds.Structure is nil or isn't a reference

func DerefDatasetTransform

func DerefDatasetTransform(store cafs.Filestore, ds *dataset.Dataset) error

DerefDatasetTransform derferences a dataset's transform element if required should be a no-op if ds.Structure is nil or isn't a reference

func DerefDatasetViz

func DerefDatasetViz(store cafs.Filestore, ds *dataset.Dataset) error

DerefDatasetViz dereferences a dataset's Viz element if required should be a no-op if ds.Viz is nil or isn't a reference

func GetHashBase

func GetHashBase(in, network string) string

GetHashBase strips paths to return just the hash

func JSONFile

func JSONFile(name string, m json.Marshaler) (qfs.File, error)

JSONFile is a convenenience method for creating a file from a json.Marshaller

func LoadBody

func LoadBody(store cafs.Filestore, ds *dataset.Dataset) (qfs.File, error)

LoadBody loads the data this dataset points to from the store

func LoadCommit

func LoadCommit(store cafs.Filestore, path string) (st *dataset.Commit, err error)

LoadCommit loads a commit from a given path in a store

func LoadDataset

func LoadDataset(store cafs.Filestore, path string) (*dataset.Dataset, error)

LoadDataset reads a dataset from a cafs and dereferences structure, transform, and commitMsg if they exist, returning a fully-hydrated dataset

func LoadDatasetRefs

func LoadDatasetRefs(store cafs.Filestore, path string) (*dataset.Dataset, error)

LoadDatasetRefs reads a dataset from a content addressed filesystem without dereferencing it's components

func LoadMeta

func LoadMeta(store cafs.Filestore, path string) (md *dataset.Meta, err error)

LoadMeta loads a metadata from a given path in a store

func LoadRows

func LoadRows(store cafs.Filestore, ds *dataset.Dataset, limit, offset int) ([]byte, error)

LoadRows loads a slice of raw bytes inside a limit/offset row range

func LoadStructure

func LoadStructure(store cafs.Filestore, path string) (st *dataset.Structure, err error)

LoadStructure loads a structure from a given path in a store

func LoadTransform

func LoadTransform(store cafs.Filestore, path string) (q *dataset.Transform, err error)

LoadTransform loads a transform from a given path in a store

func LoadTransformScript

func LoadTransformScript(store cafs.Filestore, dspath string) (qfs.File, error)

LoadTransformScript loads transform script data from a dataset path if the given dataset has a transform script specified the returned qfs.File will be the value of dataset.Transform.ScriptPath TODO - this is broken, assumes file is JSON. fix & test or depricate

func LoadViz

func LoadViz(store cafs.Filestore, path string) (st *dataset.Viz, err error)

LoadViz loads a viz from a given path in a store

func LoadVizScript

func LoadVizScript(store cafs.Filestore, dspath string) (qfs.File, error)

LoadVizScript loads script data from a dataset path if the given dataset has a viz script is specified the returned qfs.File will be the value of dataset.Viz.ScriptPath

func PackageFilepath

func PackageFilepath(store cafs.Filestore, path string, pf PackageFile) string

PackageFilepath returns the path to a package file for a given base path It relies relies on package storage conventions and cafs.Filestore path prefixes If you supply a path that does not match the filestore's naming conventions will return an invalid path

func SaveCommit

func SaveCommit(store cafs.Filestore, s *dataset.Commit, pin bool) (path string, err error)

SaveCommit writes a commit message to a cafs

func SaveMeta

func SaveMeta(store cafs.Filestore, s *dataset.Meta, pin bool) (path string, err error)

SaveMeta saves a query's metadata to a given store

func SaveStructure

func SaveStructure(store cafs.Filestore, s *dataset.Structure, pin bool) (path string, err error)

SaveStructure saves a query's structure to a given store

func SaveTransform

func SaveTransform(store cafs.Filestore, q *dataset.Transform, pin bool) (path string, err error)

SaveTransform writes a transform to a cafs

func SaveViz

func SaveViz(store cafs.Filestore, v *dataset.Viz, pin bool) (path string, err error)

SaveViz saves a query's viz to a given store

func WriteDataset

func WriteDataset(store cafs.Filestore, ds *dataset.Dataset, pin bool) (string, error)

WriteDataset writes a dataset to a cafs, replacing subcomponents of a dataset with path references during the write process. Directory structure is according to PackageFile naming conventions. This method is currently exported, but 99% of use cases should use CreateDataset instead of this lower-level function

Types

type PackageFile

type PackageFile int

PackageFile specifies the different types of files that are stored in a package

const (
	// PackageFileUnknown is the default package file, which
	// should be erroneous, as there is no sensible default
	// for PackageFile
	PackageFileUnknown PackageFile = iota
	// PackageFileDataset is the main dataset.json file
	// that contains all dataset metadata, and is the only
	// required file to constitute a dataset
	PackageFileDataset
	// PackageFileStructure isolates this dataset's structure
	// in it's own file
	PackageFileStructure
	// PackageFileAbstract is the abstract verion of
	// structure
	PackageFileAbstract
	// PackageFileResources lists the resource datasets
	// that went into creating a dataset
	// TODO - I think this can be removed now that Transform exists
	PackageFileResources
	// PackageFileCommit isolates the user-entered
	// documentation of the changes to this dataset's history
	PackageFileCommit
	// PackageFileTransform isloates the concrete transform that
	// generated this dataset
	PackageFileTransform
	// PackageFileAbstractTransform is the abstract version of
	// the operation performed to create this dataset
	PackageFileAbstractTransform
	// PackageFileMeta encapsulates human-readable metadata
	PackageFileMeta
	// PackageFileViz isolates the data related to representing a dataset as a visualization
	PackageFileViz
	// PackageFileRenderedViz is the rendered visualization of the dataset
	PackageFileRenderedViz
)

func (PackageFile) Filename

func (p PackageFile) Filename() string

Filename gives the canonical filename for a PackageFile

func (PackageFile) String

func (p PackageFile) String() string

String implements the io.Stringer interface for PackageFile

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL