README
¶
nodedb
NodeDB - a simple yet powerful persistency layer.
Why NodeDB?
NodeDB was created to serve my need to have a simple yet powerful persistency layer. After decades of writing software and using ORMs, I grew tired of designing data base schemas, object hierarchies and doing all the plumbing and wiring up to overcome the object relational-impedance-mismatch-problem, again and again, ...
After many years of working with relational data bases, I got interested in exploring different data base concepts, for example key-value stores, document databases and in particular graph databases. They all have their strengths and draw backs. Wouldn't it be great to combine all strengths and avoid the weaknesses? Somehow?
PostgreSQL, Oracle, MySQL, MongoDB, DynamoDB, Couchbase, Neo4j, Cassandra, ... these databases are all big and heavy players and I wanted something really small and simple, light-weight but capable to help me getting my job done.
As with all software, data needs to be written into file(s) eventually and I didn't want to write the io-code to do so myself. So I picked a relational data base as the lowest layer of abstraction. In principal it could be any SQL data base but it should provide Common Table Expressions. My pick is SQLite! It is fast, reliable, modern and extremely easy to use/administer. And it should be fairly easy to move to a "bigger" database like PostgreSQL, should the need arise later.
In order to overcome the object-relational gap, there are simplications and compromises to be made on both sides:
Relational side
There is only on data schema: the "Node". This node is the unit of data exchange, all data is expressed as nodes, the data base excepts only nodes and delivers only nodes.
Schema of a node:
| Field | Type | Description |
|---|---|---|
| id | Integer, Primary Key | unique, automatically assigned |
| typeid | Integer, Indexed | schema node axis |
| ownerid | Integer, Indexed | owner axis |
| up | Integer, Indexed | structural axis: tree, list, composite, master-detail, ... |
| left | Integer, Indexed | edge axis to the left - uni- or bidirectional links between nodes |
| right | Integer, Indexed | edge axis to the right - uni- or bidirectional links between nodes |
| flags | Integer | top 31 bits reserved, low 32 bits for custom use |
| text | String, Indexed | any text, node name, etc. could be used with like operator |
| created | DateTime | creation date/time of this node, automatically set to current date/time |
| start | DateTime | custom date/time, initialized with created date/time |
| end | DateTime | custom date/time, initialized with deleted date/time |
| deleted | DateTime | deletion data/time initialized to a date far into the future and set to the current date/time on deletion |
| content | Blob/[]Byte | payload of this node, can be anything, i.e. an image, a document, a serialized object, ... |
This simple schema can provide for all my needs in designing data structures, lists, documents, trees, composite, graphs, ... more on that later.
With only one schema in the data base there is only one set of SQL-Statements needed to handle all data updates and queries. And everything could be done with only one table. Using only one table felt like a loss of design "ressources". So I came up with the idea to "call" tables collections (of nodes). The structur Database - Tables - Records becomes Connection - Collections - Nodes. The node-ID is unique only within its collection, copying or moving nodes between collections requires additional adjustment work of source and target ids.
API:
-
Connection:
- Open(filename), Close(), Collections(), Collection(name), RemoveCollection(name), CloneCollection(source_name, target_name), Ping()
-
Collection:
- NewNode(), NewNodes(number), LoadNode(id), LoadNodes([]id), UpdateNode(node), UpdateNodes([]node), DeleteNode(node), DeleteNodes([]node)
-
Node:
- Public Fields + IsContentLoaded(), SetContentLoaded(bool), GetFlag(), SetFlag(), ...
-
Query:
- QueryAxes recursively, QueryList, QueryNeighbour, ... work in progress
-
Graph-Algorithms:
- Shortest Path, Minimum Spanning Tree, ... future work
-
...
The Object-Oriented side
There is no help from an ORM! Put up with the work of serializing your objects into a single unit of data exchange. The objects don't get filled with data automagically, but the process of storing and loading is fairly straightforward (the implemention language is Go, this makes it possible to directly embed a node as an anonymous property -> easy access to its data fields and content).
The current work represents only the lowest possible layer of abstractions and even this is still a long way away from completion.
The current design principles are:
- keep it as simple as possible, but not simpler, keep as many future options open as possible.
- use every part to its best performance and do the work where it fits best, i.e. let the underlying database do as much work as possible.
- provide the means to explore what can be done with a single unit of data exchange.
- abstract away the database layer, stay in the realm of writing code for the most of time.
- create and persist arbitrary object structures
- deletion should not be scary and be undone easily
- every information has a lifecycle: a creation and deletion interval and a custom start-end interval to i.e. indicate when an information becomes valid or invalid or a duration for a task, project, etc.
- there are three hierarchical axes: up, type and owner, to build type and ownership hierarchies and any tree-like structures with the up axis.
- the is a left-right edge axis to build i.e double-linked lists or relationships between nodes like links in graph-databases.
- put anything into the content field: object properties, json-schema, json-data, images, text documents, html, markdown, ...
- create type nodes that provide schema information and use their id as the typeid in the corresponding data nodes.
- it is easy to create queries and all kinds of algorithms on top of NodeDB, it is just SQL and the underlying database is readily available.
Intended uses:
A quick and simple persistency layer for a toy project or prototype. Simple Data-Management for a web site, backend for ... see JAMStack (Javascript - API - Markup) A Data-Layer for multiple client web apps, where each client gets their own single collection, this should provide a good data storage solution and great separation between clients.
Ideas:
- Put a GraphQL-Server-Layer on top of NodeDB, that would be really cool!
- Use JSON-Schema to provide Schema, Validation (and UI-Information for the frontend) in schema-nodes. Data nodes have corresponding JSON-data in their content field.
- Create Composite Objects, i.e. Person with Adress, Email, Phone- Information embedded. Adress, Email and Phone-Information are individual nodes themselves, or projects with master-detail-task information.
This is just the start of my journey, I want to create a second-brain-personal-information-project-management-system on top of NodeDB! Care to join?
Cheers, Thomas
Documentation
¶
Index ¶
- Constants
- type Collection
- func (c *Collection) DeleteNode(update *Node) error
- func (c *Collection) DeleteNodes(updates []*Node) error
- func (c *Collection) LoadNode(nodeID int64, withContent bool) (*Node, error)
- func (c *Collection) LoadNodes(nodeIDs []int64, withContent bool) ([]*Node, error)
- func (c *Collection) LoadNodesInSequence(nodeIDs []int64, withContent bool) ([]*Node, error)
- func (c *Collection) NewNode() (*Node, error)
- func (c *Collection) NewNodes(count int) ([]*Node, error)
- func (c *Collection) UpdateNode(update *Node) error
- func (c *Collection) UpdateNodes(updates []*Node) error
- type Connection
- func (c *Connection) CloneCollection(sourceName string, cloneName string) error
- func (c *Connection) Close() error
- func (c *Connection) Collection(name string) (*Collection, error)
- func (c *Connection) Collections() (collections []string, err error)
- func (c *Connection) LoggingEnabled() bool
- func (c *Connection) Ping() error
- func (c *Connection) RemoveCollection(name string) error
- func (c *Connection) SetLogging(logging bool)
- type Flag
- type GenericData
- type GenericDataSlice
- type Node
- type SQLString
Constants ¶
const SQLCloneCollection = "insert into {{.targetCollection}} select * from {{.sourceCollection}};"
SQLCloneCollection perform a record by record from source table into (empty) target table
const SQLCollection = `select tbl_name, type from sqlite_master where tbl_name = ?;`
SQLCollection find a single table name
const SQLCollections = `select tbl_name from sqlite_master where type = "table" and tbl_name != "sqlite_sequence" order by tbl_name;`
SQLCollections lists all tables in SQLite database
const SQLDropCollection = `DROP TABLE IF EXISTS "{{.collection}}";`
SQLDropCollection deletes a table permanently - no undoing this
const SQLInsertNode = `` /* 139-byte string literal not displayed */
SQLInsertNode insert a new node into the collection
const SQLListQuery = `` /* 152-byte string literal not displayed */
SQLListQuery read all nodes under a provided up-ID and optionally narrow down certain type-IDs
const SQLLoadNode = `` /* 137-byte string literal not displayed */
SQLLoadNode read a single record by id from selected table (collection)
const SQLLoadNodes = `` /* 153-byte string literal not displayed */
SQLLoadNodes read a set of record by ids from selected table (collection)
const SQLNewCollection = `` /* 919-byte string literal not displayed */
SQLNewCollection creates the scheme and the indexes for the node
const SQLRecursiveAxis = `` /* 711-byte string literal not displayed */
SQLRecursiveAxis reads records recursively to find all descendents of a single node under a specified axis. It keeps a level info. This query can be stopped at a maximum level and/or at a limit of returned records count
const SQLUpdateNode = `` /* 176-byte string literal not displayed */
SQLUpdateNode update a single node in the collection
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Collection ¶
type Collection struct {
Conn *Connection
Name string
}
func (*Collection) DeleteNode ¶
func (c *Collection) DeleteNode(update *Node) error
func (*Collection) DeleteNodes ¶
func (c *Collection) DeleteNodes(updates []*Node) error
func (*Collection) LoadNode ¶
func (c *Collection) LoadNode(nodeID int64, withContent bool) (*Node, error)
LoadNode loads a node by its ID, provide withContent = true, if the payload (content) should be loaded with the node.
func (*Collection) LoadNodes ¶
func (c *Collection) LoadNodes(nodeIDs []int64, withContent bool) ([]*Node, error)
func (*Collection) LoadNodesInSequence ¶
func (c *Collection) LoadNodesInSequence(nodeIDs []int64, withContent bool) ([]*Node, error)
func (*Collection) NewNode ¶
func (c *Collection) NewNode() (*Node, error)
NewNode creates and inserts a new node into the collection (runs SQL insert command), this way a node has allways a valid ID The ID may never be zero. A node that you use in your code must allways have a valid representation in the database.
func (*Collection) UpdateNode ¶
func (c *Collection) UpdateNode(update *Node) error
func (*Collection) UpdateNodes ¶
func (c *Collection) UpdateNodes(updates []*Node) error
type Connection ¶
type Connection struct {
DB *sql.DB
RootID int64
Created time.Time
Start time.Time
End time.Time
Deleted time.Time
// contains filtered or unexported fields
}
Connection provide access to the database
func Open ¶
func Open(filename string) (*Connection, error)
Open creates a new Connection Object and opens the underlying sqlite database
func (*Connection) CloneCollection ¶
func (c *Connection) CloneCollection(sourceName string, cloneName string) error
CloneCollection makes a full copy of source collection in a new clone collection
func (*Connection) Close ¶
func (c *Connection) Close() error
Close closes the underlying sqlite database
func (*Connection) Collection ¶
func (c *Connection) Collection(name string) (*Collection, error)
Collection opens (and creates) collections
func (*Connection) Collections ¶
func (c *Connection) Collections() (collections []string, err error)
Collections returs a string of collection names
func (*Connection) LoggingEnabled ¶
func (c *Connection) LoggingEnabled() bool
LoggingEnabled returns true if logging is enabled, false otherwise
func (*Connection) RemoveCollection ¶
func (c *Connection) RemoveCollection(name string) error
RemoveCollection removes collections, it is destructive, no undo possible!
func (*Connection) SetLogging ¶
func (c *Connection) SetLogging(logging bool)
SetLogging to true or false, default is false
type Flag ¶
type Flag int64
const ( First Flag = 1 << (62 - iota) // first node in a sequence of nodes (the head of a double-linked-list) Bidirectional // in relational node this points in both directions, not the usual left->right Content // node has a Content, this needs to be loaded separately, needs to be set if one of the following is set )
func (Flag) IsBidirectional ¶
type GenericData ¶
type GenericData map[string]interface{}
func GenericDataFromBytes ¶
func GenericDataFromBytes(bytes []byte) (GenericData, error)
func (GenericData) Bytes ¶
func (g GenericData) Bytes() ([]byte, error)
type GenericDataSlice ¶
type GenericDataSlice []map[string]interface{}
func GenericDataSliceFromBytes ¶
func GenericDataSliceFromBytes(bytes []byte) (GenericDataSlice, error)
func (GenericDataSlice) Bytes ¶
func (gs GenericDataSlice) Bytes() ([]byte, error)
type Node ¶
type Node struct {
ID int64
TypeID int64
OwnerID int64
Flags Flag
Created time.Time
Start time.Time
End time.Time
Deleted time.Time
Up int64
Left int64
Right int64
Text string // should be short < 2 KB
Content []byte
// contains filtered or unexported fields
}
func QueryList ¶
func QueryList(collection *Collection, upID int64, withContent bool, typeids ...int64) ([]*Node, error)
QueryList finds all Nodes with provided parentID (up) and and optional list of typeids. It first queries the database and then builds a sequence ordered by the left-right-axis. There may be more then one list in the result set, check for start nodes with left==0 or end nodes with right == 0 It reports an error if the number of sequenced nodes differs from the number of nodes return by the data base query. The result set may still be usable.
func (*Node) IsContentLoaded ¶
func (*Node) SetContentLoaded ¶
type SQLString ¶
type SQLString string
SQLString provide changed text replacements to form a valid sql string
func (SQLString) Collection ¶
helper: replaces collection name template
func (SQLString) WithContent ¶
helper: replaces commaContent, handles with or without content variations
Source Files
Directories