dagstore

NOTE: THIS PROJECT IS NOT READY FOR USE AT THIS TIME.

DOCUMENTATION BELOW IS A PLACEHOLDER.

About The Project

DAGStore is a storage system for large-scale branched versioning of data types including key-value pairs, a log, and N-dimensional chunks. It distills lessons learned over a decade developing and using DVID, a dataservice supporting branched versioning of reconstruction data in the field of Connectomics at HHMI Janelia. Domain-specific applications (like DVID) can build higher-level data operations on top of DAGStore by using the mix of supported data types. As described below, DAGStore's architecture will simplify devops issues that became evident through real-world use of branched versioning at terabyte scale, and allow a number of novel features like "chained" immutable stores.

The initial prototype will focus on the ordered key-value store and a Go language implementation (since DVID was written in Go). If DAGStore proves useful, the hope is to have implementations in other languages, at least to serve up immutable versions already stored in various formats.

Aside from scientific work that requires checkpointed versions of large-scale data, the general community might find DAGStore useful for their own work. To that end, a FUSE-based filesystem and branched versioning tool was written as a simple application layer on top of DAGStore. This tool can be tested by installing the dags command-line tool and running dags fs.

DAGStore Architecture in Nutshell

DAGStore explicitly separates mutable data from immutable data through a manual “commit” operation similar to how commits work in the git version control system. This explicit separation provides a number of advantages including limiting the use of any suitable off-the-shelf database for newly written data while maintaining a common, transferable data format for the committed, immutable data.

DAGStore physically separates immutable data into versions, which allows easy transfer of versions and “chained” stores where remote DAGStores provide immutable data for ancestors of versions. Typically only changes are stored in a version though a published version could be a flattened snapshot of the entire dataset. For Open Science, this physical separation can leverage published open data by delegating storage and access costs to large scientific institutions and repositories like Google Public Datasets and AWS OpenData. In the case of Connectomics, each published dataset is now tens of terabytes with the expected sizes to grow substantially over the next decade. DAGStore will allow users to modify public datasets yet limit their data handling burden to just their alterations which can be stored locally, combining their local changes with reads from unchanged data in ancestor versions.

(back to top)

Getting Started

This package can be used either as a Go library, providing an embedded datastore, or a standalone server by launching it using the "dags" command-line tool. Both cases are described below.

Install for use as a Go library

1. Install Go
2. Get the package from GitHub.

go get github.com/dagstore/dagstore-go

Install as command-line tool

The simplest approach is to get the single binary executable for your platform and use the dags command-line tool.

TODO -- Provide single binary executables for popular platforms via github Releases.

Via source-code:

1. Install Go
2. Clone the package from GitHub and build the dags cli tool:

git clone https://github.com/dagstore/dagstore-go
cd dagstore-go/dags
go build  # this creates a "dags" executable in that directory
./dags fs mount /path/to/DAGStore/dir /path/to/mount/dir # demo tool using FUSE filesystem interface

(back to top)

Usage

This space will be fleshed in later. For now, please consult the automatically-generated GoDocs.

In the future, more exampples will be in the project's website section for Documentation

(back to top)

Roadmap

• Framework for branched versioning metadata
• Framework for human-friendly names
• Command-line use
  • Scaffolding via Cobra
  • FUSE interface for demo and testing
  • Server for using DAGStore standalone instead of a library
• Mutable store using Badger
• Immutable store
  • Index using Badger
  • Cloud-friendly immutable data storage by version
• Key-value support
  • Local-only
  • Support chained immutable stores.
  • Add rpc (probably gRPC) to support cli from even embedded library use.
• Branched versioning for logs.
• Branched versioning for N-Dimensional chunks.
  • Support static neuroglancer precomputed versions.
  • Support static zarr versions.
  • Support static N5 versions.
• Mechanism for pull requests by transfer of immutable version data.
  • Ingestion of transferred versions
  • Nice workflow allowing manual inspect, perhaps using Github API

See the open issues for a full list of proposed features (and known issues).

(back to top)

Contributing

Contributing to Go implementation

Contributions are appreciated. Please fork the repo and create a pull request. You can also open an issue with the tag "enhancement".

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

Creating an implementation in another language

We hope to have implementations in other languages, at least to serve up immutable versions already stored in various formats. This would allow reuse of libraries focused on particular kinds of immutable data. For example, a C++ implementation could more easily build on top of tensorstore and leverage that system to efficiently read a variety of data formats.

Please open an issue with the tag "implementations" with a description of your intended system. This might help build momentum for that implementation. We could also host other language implentations under the "DAGStore" github organization.

(back to top)

License

Distributed by HHMI Janelia under a BSD 3-Clause License. See LICENSE.txt for more information.

(back to top)

Acknowledgments

• Thanks to dgraph and the creators of the Badger Go key-value database
• This system wouldn't be possible without the fantastic open source community. A look at the go.mod file will show the first-level of work we've built upon. Those not in that Go source code list include the Go language team, Best-README-Template, and others.

(back to top)