qri: github.com/qri-io/qri/base Index | Files | Directories

package base

import "github.com/qri-io/qri/base"

Package base defines business that operates on local data. it's main job is to composing APIs from the lower half of our tech stack, providing uniform functions for higher up packages, mainly p2p and lib. p2p and lib use base as the only way of operate on the local repo Here's some ascii art to clarify the stack:

┌───────────────┐ ┌───────────────┐
│      cmd      │ │     api       │
└───────────────┘ └───────────────┘
│               lib               │
│          p2p          │
│              base               │  <-- you are here
┌──────┐ ┌──────┐ ┌──────┐ ┌──────┐
│ repo │ │ dsfs │ │ qfs  │ │ ...  │
└──────┘ └──────┘ └──────┘ └──────┘

There are packages omitted from this diagram, but these are the vitals. base functions mainly work with repo.Repo instances, using repo interface methods and related packages to do their work. This is part of a larger pattern of having lib rely on lower level interfaces wherever possible to enhance configurability


Package Files

base.go body.go dataset.go dataset_prepare.go hidden_file.go log.go ref.go render.go render_readme.go revisions.go save.go select.go validate.go


var DefaultTemplate = "" /* 251 byte string literal not displayed */

DefaultTemplate is the template that render will fall back to should no template be available

func ApplyPath Uses

func ApplyPath(ds *dataset.Dataset, path string) (interface{}, error)

ApplyPath gets a dataset value by applying a case.Sensitve.dot.separated.path ApplyPath cannot select file fields

func CloseDataset Uses

func CloseDataset(ds *dataset.Dataset) (err error)

CloseDataset ensures all open dataset files are closed

func ConvertBodyFile Uses

func ConvertBodyFile(file qfs.File, in, out *dataset.Structure, limit, offset int, all bool) (data []byte, err error)

ConvertBodyFile takes an input file & structure, and converts a specified selection to the structure specified by out

func ConvertBodyFormat Uses

func ConvertBodyFormat(bodyFile qfs.File, fromSt, toSt *dataset.Structure) (qfs.File, error)

ConvertBodyFormat rewrites a body from a source format to a destination format. TODO (b5): Combine this with ConvertBodyFile, update callers.

func CreateDataset Uses

func CreateDataset(ctx context.Context, r repo.Repo, streams ioes.IOStreams, ds, dsPrev *dataset.Dataset, dryRun, pin, force, shouldRender bool) (ref repo.DatasetRef, err error)

CreateDataset uses dsfs to add a dataset to a repo's store, updating all references within the repo if successful

func DatasetBodyFile Uses

func DatasetBodyFile(ctx context.Context, store cafs.Filestore, ds *dataset.Dataset) (qfs.File, error)

DatasetBodyFile creates a streaming data file from a Dataset using the following precedence: * ds.BodyBytes not being nil (requires ds.Structure.Format be set to know data format) * ds.BodyPath being a url * ds.BodyPath being a path on the local filesystem

func DatasetLogFromHistory Uses

func DatasetLogFromHistory(ctx context.Context, r repo.Repo, ref repo.DatasetRef, offset, limit int, loadDatasets bool) (rlog []repo.DatasetRef, err error)

DatasetLogFromHistory fetches the history of changes to a dataset by walking backwards through dataset commits. if loadDatasets is true, dataset information will be populated

func FetchDataset Uses

func FetchDataset(ctx context.Context, r repo.Repo, ref *repo.DatasetRef, pin, load bool) (err error)

FetchDataset grabs a dataset from a remote source

func GenerateAvailableName Uses

func GenerateAvailableName(r repo.Repo, peername, prefix string) string

GenerateAvailableName creates a name for the dataset that is not currently in use

func InLocalNamespace Uses

func InLocalNamespace(r repo.Repo, ref *repo.DatasetRef) bool

InLocalNamespace checks if a dataset ref is local, assumes the reference is already canonicalized

func InferValues Uses

func InferValues(pro *profile.Profile, ds *dataset.Dataset) error

InferValues populates any missing fields that must exist to create a snapshot

func InlineJSONBody Uses

func InlineJSONBody(ds *dataset.Dataset) error

InlineJSONBody reads the contents dataset.BodyFile() into a json.RawMessage, assigning the result to dataset.Body

func ListDatasets Uses

func ListDatasets(ctx context.Context, r repo.Repo, term string, limit, offset int, RPC, publishedOnly, showVersions bool) (res []repo.DatasetRef, err error)

ListDatasets lists datasets from a repo

func LoadRevs Uses

func LoadRevs(ctx context.Context, r repo.Repo, ref repo.DatasetRef, revs []*dsref.Rev) (res *dataset.Dataset, err error)

LoadRevs grabs a component of a dataset that exists <n>th generation ancestor of the referenced version, where presence of a component in a previous snapshot constitutes ancestry

func MaybeAddDefaultViz Uses

func MaybeAddDefaultViz(ds *dataset.Dataset)

MaybeAddDefaultViz sets a dataset viz component and template if none exists TODO(dlong): This is deprecated and should be removed.

func MaybeInferName Uses

func MaybeInferName(ds *dataset.Dataset) bool

MaybeInferName infer a name for the dataset if none is set

func ModifyDatasetRef Uses

func ModifyDatasetRef(ctx context.Context, r repo.Repo, current, new *repo.DatasetRef, isRename bool) (err error)

ModifyDatasetRef alters a reference by changing what dataset it refers to

func ModifyRepoUsername Uses

func ModifyRepoUsername(ctx context.Context, r repo.Repo, book *logbook.Book, from, to string) error

ModifyRepoUsername performs all tasks necessary to switch a username (formerly: peername) for a local repo, and must be called when a username is changed TODO (b5) - make this transactional

func OpenDataset Uses

func OpenDataset(ctx context.Context, fsys qfs.Filesystem, ds *dataset.Dataset) (err error)

OpenDataset prepares a dataset for use, checking each component for populated Path or Byte suffixed fields, consuming those fields to set File handlers that are ready for reading

func PinDataset Uses

func PinDataset(ctx context.Context, r repo.Repo, ref repo.DatasetRef) error

PinDataset marks a dataset for retention in a store

func PrepareDatasetSave Uses

func PrepareDatasetSave(ctx context.Context, r repo.Repo, peername, name string) (prev, mutable *dataset.Dataset, prevPath string, err error)

PrepareDatasetSave prepares a set of changes for submission to SaveDataset prev is the previous dataset, if it exists body is the previous dataset body, if it exists mutable is the previous dataset, but without the commit and transform, making it sutable for mutation/combination with any potential changes requested by the user we do not error if the dataset is not found in the repo, instead we return all empty values TODO (b5): input parameters here assume the store can properly resolve the previous dataset path through canonicalization (looking the name up in the repo). The value given by the input dataset document may differ, and we should probably respect that value if it does

func RawDatasetRefs Uses

func RawDatasetRefs(ctx context.Context, r repo.Repo) (string, error)

RawDatasetRefs converts the dataset refs to a string

func ReadBody Uses

func ReadBody(ds *dataset.Dataset, format dataset.DataFormat, fcfg dataset.FormatConfig, limit, offset int, all bool) (data []byte, err error)

ReadBody grabs some or all of a dataset's body, writing an output in the desired format

func ReadDataset Uses

func ReadDataset(ctx context.Context, r repo.Repo, ref *repo.DatasetRef) (err error)

ReadDataset grabs a dataset from the store

func ReadDatasetPath Uses

func ReadDatasetPath(ctx context.Context, r repo.Repo, path string) (ds *dataset.Dataset, err error)

ReadDatasetPath takes a path string, parses, canonicalizes, loads a dataset pointer, and opens the file The medium-term goal here is to obfuscate use of repo.DatasetRef, which we're hoping to deprecate

func ReadEntries Uses

func ReadEntries(reader dsio.EntryReader) (interface{}, error)

ReadEntries reads entries and returns them as a native go array or map

func Recall Uses

func Recall(ctx context.Context, r repo.Repo, str string, ref repo.DatasetRef) (*dataset.Dataset, error)

Recall loads revisions of a dataset from history

func RemoveNVersionsFromStore Uses

func RemoveNVersionsFromStore(ctx context.Context, r repo.Repo, ref *repo.DatasetRef, n int) (*repo.DatasetRef, error)

RemoveNVersionsFromStore removes n versions of a dataset from the store starting with the most recent version when n == -1, remove all versions does not remove the dataset reference

func Render Uses

func Render(ctx context.Context, r repo.Repo, ref repo.DatasetRef, tmplData []byte) ([]byte, error)

Render executes a template for a dataset, returning a slice of HTML Render uses go's html/template package to generate html documents from an input dataset. It's API has been adjusted to use lowerCamelCase instead of UpperCamelCase naming conventions

func RenderReadme Uses

func RenderReadme(ctx context.Context, file qfs.File) (string, error)

RenderReadme converts the markdown from the file into html.

func ReplaceRefIfMoreRecent Uses

func ReplaceRefIfMoreRecent(r repo.Repo, prev, curr *repo.DatasetRef) error

ReplaceRefIfMoreRecent replaces the given ref in the ref store, if it is more recent then the ref currently in the refstore

func SaveDataset Uses

func SaveDataset(ctx context.Context, r repo.Repo, str ioes.IOStreams, changes *dataset.Dataset, secrets map[string]string, scriptOut io.Writer, sw SaveDatasetSwitches) (ref repo.DatasetRef, err error)

SaveDataset initializes a dataset from a dataset pointer and data file

func Select Uses

func Select(ctx context.Context, r repo.Repo, ref repo.DatasetRef, path string) (interface{}, error)

Select loads a dataset value specified by case.Sensitve.dot.separated.paths

func SetFileHidden Uses

func SetFileHidden(path string) error

SetFileHidden ensures the filename begins with a dot. Other OSes may do more

func SetPublishStatus Uses

func SetPublishStatus(r repo.Repo, ref *repo.DatasetRef, published bool) error

SetPublishStatus updates the Published field of a dataset ref

func ToDatasetRef Uses

func ToDatasetRef(path string, r repo.Repo, allowFSI bool) (*repo.DatasetRef, error)

ToDatasetRef parses the dataset ref and returns it, allowing datasets with no history only if FSI is enabled.

func UnpinDataset Uses

func UnpinDataset(ctx context.Context, r repo.Repo, ref repo.DatasetRef) error

UnpinDataset unmarks a dataset for retention in a store

func Validate Uses

func Validate(ctx context.Context, r repo.Repo, body qfs.File, st *dataset.Structure) ([]jsonschema.ValError, error)

Validate checks a dataset body for errors based on the structure's schema

func ValidateDataset Uses

func ValidateDataset(ds *dataset.Dataset) (err error)

ValidateDataset checks that a dataset is semantically valid

func WriteHiddenFile Uses

func WriteHiddenFile(path, content string) error

WriteHiddenFile ensures the filename begins with a dot, and writes content to it. Other Operating Systems may do more

type DatasetLogItem Uses

type DatasetLogItem struct {
    Ref dsref.Ref `json:"ref,omitempty"`
    // Creation timestamp
    Timestamp time.Time `json:"timestamp,omitempty"`
    // Title field from dataset.commit component
    CommitTitle string `json:"commitTitle,omitempty"`
    // Message field from dataset.commit component
    CommitMessage string `json:"commitMessage,omitempty"`
    // Published indicates if this version has been published
    Published bool `json:"published,omitempty"`
    // Size of dataset in bytes
    Size int64 `json:"size,omitempty"`
    // Local indicates the connected filesystem has this version available
    Local bool `json:"local,omitempty"`

DatasetLogItem is a line item in a dataset response

func DatasetLog Uses

func DatasetLog(ctx context.Context, r repo.Repo, ref repo.DatasetRef, limit, offset int, loadDatasets bool) (items []DatasetLogItem, err error)

DatasetLog fetches the change version history of a dataset

type SaveDatasetSwitches Uses

type SaveDatasetSwitches struct {
    Replace             bool
    DryRun              bool
    Pin                 bool
    ConvertFormatToPrev bool
    Force               bool
    ShouldRender        bool
    NewName             bool

SaveDatasetSwitches provides toggleable flags to SaveDataset that control save behaviour


dsfsPackage dsfs glues datsets to cafs (content-addressed-file-system)
dsfs/dstestPackage dstest defines an interface for reading test cases from static files leveraging directories of test dataset input files & expected output files
dsfs/dsutilPackage dsutil includes dataset util funcs, placed here to avoid dataset package bloat TODO - consider merging this package with the dsfs package, as most of the functions in here rely on a Filestore argument
dsfs/subsetPackage subset provides methods for extracting defined abbreviations of a dataset document.
fillPackage fill assigns arbitrary values to struct fields using reflection.

Package base imports 33 packages (graph) and is imported by 4 packages. Updated 2020-01-24. Refresh now. Tools for package owners.