README
¶
This repository houses the core server and orchestration elements which are at the heart of the Datum cloud service. We have no plans to ever gate / silo elements of the code that may fall under our "enterprise licensing" (or any other commercial license we offer) and intend to keep the code Apache 2.0 licensed and free for use, forever. Given that, if you find value in anything we're doing here, our cloud services, or use this software yourself (for any purpose) - don't be afraid to become a contributor! If you have any questions please reach out to contribute@datum.net.
What is Datum?
datum: a fixed starting point of a scale or operation
We’re on a mission to build a better, more sustainable world by providing digital leaders with open source solutions which help them innovate faster and remove massive toil. We believe that in order to change the world for good, we’ll benefit from having thousands of companies working at scale, not just a handful (the “hyperscalers”). Check out our documentation or reach out to get involved!
Features
At it's core, Datum is a collection of services built on top of an entity framework which allows us to:
- Model database schemas as graph structures
- Define schemas as programmatic go code
- Execute complex database queries and graph traversals easily
- Extend and customize using templates and code generation utilities
- Type-safe resolvers and GraphQL schema stitching
- Code generated audit / history tables for defined schemas
On top of this powerful core we also have an incredible amount of pluggable, extensible services:
- Authentication: we today support password, OAuth2 / Social login providers (Github, Google), Passkeys as well as standard OIDC Discovery flows
- Multi-factor: built-in 2FA mechanisms, TOTP
- Authorization: extensible and flexible permissions constructs via openFGA based on Google Zanzibar
- Session Management: built-in session manaagement with JWKS key validation, encrypted cookies and sessions
- Robust Middleware: cache control, CORS, Rate Limiting, transaction rollbacks, and more
- Queuing and Scheduling: Task management and scheduling with Marionette
- External Storage Providers: store data in AWS S3, Google GCS, or locally
- External Database Providers: Leverage Turso, or other PostgreSQL / SQLite compatible vendors and libraries
- Data Isolation and Management: Heirarchal organizations and granular permissions controls
Development
Developing against this repo involves a few mandatory tools; please read up on these and familiarize yourself if you're interested in making additions or changes!
- ent - insane entity mapping tool, definitely not an ORM but kind of an ORM (handles our relational data storage, mappings, codegen processes)
- atlas - Schema generation and migrations (can be disabled in lieu of migrations on disk)
- goose - Secondary database migration utility we also use for seeding data
- gqlgen - Code generation + GraphQL server building from from
entschema definitions - gqlgenc - Client building utilities with GraphQL
- openfga - Flexible authorization/permission engine inspired by Google Zanzibar
- echo - High performance, extensible, minimalist Go web framework
- koanf - Configuration management library which parses Go structs + creates our main configuration files
- viper - command line flags / management
We also leverage many secondary technologies in use, including (but not limited to!):
- taskfile - So much better than Make zomg
- redis - in-memory datastore used for sessions, caching
- sqlite - currently planned database system but also offer additional support for PostgreSQL
- golangci-lint - an annoyingly opinionated linter
- buildkite - our CI system of choice (with github actions providing some intermediary support)
- sonar - used for code scanning, vulnerability scanning
Lastly we're already ourselves using (and plan to support our customers usage in our cloud service) these third party integrations:
- turso/libsql - Turso is an edge-hosted, distributed database that's based on libSQL , an open-source and open-contribution fork of SQLite
- posthog - Product analytics
- sendgrid - Transactional email send provider
All of these components are bundled into our respective Docker images; for additional information / instructions, see the contributing guide in this repository. We're constantly adding and changing things, but have tried to list all the great open source tools and projects we rely on; if you see your project (or one you use) in here and wish to list it, feel free to open a PR!
Dependencies
The vast majority of behaviors of the system can be turned on or off by updating the configuration parameters found in config; in some instances, we've made features or integrations with third party systems which are "always on", but we're happy to receive PR's wrapping those dependencies if you are interested in running the software without them!
Installing Dependencies
Setup Taskfile by following the instructions and using one of the various convenient package managers or installation scripts. After installation, you can then simply run task install to load the associated dependencies. Nearly everything in this repository assumes you already have a local golang environment setup so this is not included. Please see the associated documentation.
Updating Configuration Settings
See the README in the config directory.
Deploying
The only "supported" method of deploying today is locally, but we have a WIP Helm chart which can be found here
Contributing
Please read the contributing guide as well as the Developer Certificate of Origin. You will be required to sign all commits to the Datum project, so if you're unfamiliar with how to set that up, see github's documentation.
Licensing
This repository contains datum which is open source software under Apache 2.0. Datum is a product produced from this open source software exclusively by Datum Technology, Inc. This product is produced under our published commercial terms (which are subject to change), and any logos or trademarks in this repository or the broader datumforge organization are not covered under the Apache License.
Others are allowed to make their own distribution of this software or include this software in other commercial offerings, but cannot use any of the Datum logos, trademarks, cloud services, etc.
Security
We take the security of our software products and services seriously, including all of the open source code repositories managed through our Github Organizations, such as datumforge. If you believe you have found a security vulnerability in any of our repositories, please report it to us through coordinated disclosure.
Please do NOT report security vulnerabilities through public github issues, discussions, or pull requests!
Instead, please send an email to security@datum.net with as much information as possible to best help us understand and resolve the issues. See the security policy attached to this repository for more details.
Questions?
You can email us at info@datum.net, open a github issue in this repository, or reach out to matoszz directly.
Documentation
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package cmd is our cobra/viper cli implementation
|
Package cmd is our cobra/viper cli implementation |
|
cli/cmd
Package datum is our cobra/viper cli implementation
|
Package datum is our cobra/viper cli implementation |
|
cli/cmd/apitokens
Package datumapitokens is our cobra/viper cli for api token endpoints
|
Package datumapitokens is our cobra/viper cli for api token endpoints |
|
cli/cmd/group
Package datumgroup is our cobra/viper cli for group endpoints
|
Package datumgroup is our cobra/viper cli for group endpoints |
|
cli/cmd/groupmembers
Package datumgroupmembers is our cobra/viper cli for group member endpoints
|
Package datumgroupmembers is our cobra/viper cli for group member endpoints |
|
cli/cmd/groupsetting
Package datumgroupsetting provides commands for managing group settings
|
Package datumgroupsetting provides commands for managing group settings |
|
cli/cmd/invite
Package datuminvite creates invitation emails + tokens for external users to join a Datum organization
|
Package datuminvite creates invitation emails + tokens for external users to join a Datum organization |
|
cli/cmd/login
Package datumlogin is our cobra/viper cli for authentication endpoints
|
Package datumlogin is our cobra/viper cli for authentication endpoints |
|
cli/cmd/org
Package datumorg is our cobra/viper cli for organization endpoints
|
Package datumorg is our cobra/viper cli for organization endpoints |
|
cli/cmd/orgmembers
Package datumorgmembers is our cobra/viper cli for org member endpoints
|
Package datumorgmembers is our cobra/viper cli for org member endpoints |
|
cli/cmd/orgsetting
Package datumorgsetting provides commands for managing organization settings
|
Package datumorgsetting provides commands for managing organization settings |
|
cli/cmd/personalaccesstokens
Package datumtokens is our cobra/viper cli for token endpoints
|
Package datumtokens is our cobra/viper cli for token endpoints |
|
cli/cmd/register
Package register allows user registration
|
Package register allows user registration |
|
cli/cmd/reset
Package reset allows user password reset
|
Package reset allows user password reset |
|
cli/cmd/subscriber
Package datumsubscribers is our cobra/viper cli for subscriber endpoints
|
Package datumsubscribers is our cobra/viper cli for subscriber endpoints |
|
cli/cmd/switch
Package datumswitch provides a basic interface to switch between oragnization contexts
|
Package datumswitch provides a basic interface to switch between oragnization contexts |
|
cli/cmd/user
Package datumuser is our cobra/viper cli for user endpoints
|
Package datumuser is our cobra/viper cli for user endpoints |
|
cli/cmd/usersetting
Package datumusersetting is our cobra/viper cli for user setting endpoints
|
Package datumusersetting is our cobra/viper cli for user setting endpoints |
|
cli/cmd/version
Package version contains the version information for the CLI
|
Package version contains the version information for the CLI |
|
Package config holds configuration stuff to configure the things
|
Package config holds configuration stuff to configure the things |
|
Package db provides an embedded filesystem containing all the database migrations
|
Package db provides an embedded filesystem containing all the database migrations |
|
internal
|
|
|
constants
Package constants contains constants used throughout the application
|
Package constants contains constants used throughout the application |
|
ent/customtypes
Package customtypes is a custom type definition for ent and the associated interfaces
|
Package customtypes is a custom type definition for ent and the associated interfaces |
|
ent/enums
Package enums has enums
|
Package enums has enums |
|
ent/generated
Package generated is the ent generated package
|
Package generated is the ent generated package |
|
ent/hooks
Package hooks is middleware to alter the graphql mutation
|
Package hooks is middleware to alter the graphql mutation |
|
ent/interceptors
Package interceptors is middleware to alter the graphql query
|
Package interceptors is middleware to alter the graphql query |
|
ent/mixin
Package mixin contains the mixin package
|
Package mixin contains the mixin package |
|
ent/privacy/rule
Package rule contains policy rules
|
Package rule contains policy rules |
|
ent/privacy/token
Package token provides policy wrappers / getters for queries and mutations during password reset, signup
|
Package token provides policy wrappers / getters for queries and mutations during password reset, signup |
|
ent/privacy/utils
Package utils is a helper for working with ent policies
|
Package utils is a helper for working with ent policies |
|
ent/privacy/viewer
Package viewer contains role definitions for policies
|
Package viewer contains role definitions for policies |
|
ent/schema
Package schema contains the ent schema
|
Package schema contains the ent schema |
|
entdb
Package entdb extends the ent db library and satisfies matt's needs for consistency
|
Package entdb extends the ent db library and satisfies matt's needs for consistency |
|
graphapi
Package graphapi is the graph api package
|
Package graphapi is the graph api package |
|
httpserve/config
Package config holds the echo server configuration utilities
|
Package config holds the echo server configuration utilities |
|
httpserve/handlers
Package handlers contains custom handler functions
|
Package handlers contains custom handler functions |
|
httpserve/route
Package route will hold the routes and route groups
|
Package route will hold the routes and route groups |
|
httpserve/server
Package server contains the server functions
|
Package server contains the server functions |
|
httpserve/serveropts
Package serveropts contains an echo server options wrapper
|
Package serveropts contains an echo server options wrapper |
|
envparse
Package envparse provides a way to parse environment variables from a struct
|
Package envparse provides a way to parse environment variables from a struct |
|
pkg
|
|
|
analytics
Package analytics offers datum system analytics
|
Package analytics offers datum system analytics |
|
analytics/machine
Package machine provides a basic wrapper for getting machine ID
|
Package machine provides a basic wrapper for getting machine ID |
|
analytics/posthog
Package posthog contains posthog helper functions and utilities
|
Package posthog contains posthog helper functions and utilities |
|
auth
Package auth handles auth
|
Package auth handles auth |
|
cache
Package cache holds the library for interacting with redis
|
Package cache holds the library for interacting with redis |
|
datumclient
Package datumclient contains the client to interact with the datum server
|
Package datumclient contains the client to interact with the datum server |
|
events
Package events does event stuff
|
Package events does event stuff |
|
events/kafka/config
Package config provides configuration for the Kafka event source
|
Package config provides configuration for the Kafka event source |
|
events/kafka/kafkaconfig
Package kafkaconfig provides configuration for the Kafka event source
|
Package kafkaconfig provides configuration for the Kafka event source |
|
events/kafka/publisher
Package publisher provides a basic kafka event publisher
|
Package publisher provides a basic kafka event publisher |
|
keygen
Package keygen provides functionality for generating API client IDs and secrets
|
Package keygen provides functionality for generating API client IDs and secrets |
|
middleware
Package middleware provides middleware for http Handlers.
|
Package middleware provides middleware for http Handlers. |
|
middleware/auth
Package auth handles auth
|
Package auth handles auth |
|
middleware/authtest
Package authtest is a mini server and token generator with a mock test to confirm keygen works correctly; not intended to test the whole api but rather just the token issuance and authentication
|
Package authtest is a mini server and token generator with a mock test to confirm keygen works correctly; not intended to test the whole api but rather just the token issuance and authentication |
|
middleware/cachecontrol
Package cachecontrol creates a cache control echo middleware
|
Package cachecontrol creates a cache control echo middleware |
|
middleware/cors
Package cors does cors stuff
|
Package cors does cors stuff |
|
middleware/debug
Package debug contains utility helper functions for debugging when working with echo server
|
Package debug contains utility helper functions for debugging when working with echo server |
|
middleware/echocontext
Package echocontext adds the echo context to the parent context
|
Package echocontext adds the echo context to the parent context |
|
middleware/mime
Package mime does mime stuff
|
Package mime does mime stuff |
|
middleware/ratelimit
Package ratelimit implements a rate limiting middleware
|
Package ratelimit implements a rate limiting middleware |
|
middleware/redirect
Package redirect does redirect stuff
|
Package redirect does redirect stuff |
|
middleware/secure
Package secure is middleware that provides protection against cross-site scripting (XSS) attack, content type sniffing, clickjacking, insecure connection and other code injection attacks
|
Package secure is middleware that provides protection against cross-site scripting (XSS) attack, content type sniffing, clickjacking, insecure connection and other code injection attacks |
|
middleware/sentry
Package sentry is a middleware package for sentry integration
|
Package sentry is a middleware package for sentry integration |
|
middleware/transaction
Package transaction implements a transaction middleware for REST endpoints using the ent db client
|
Package transaction implements a transaction middleware for REST endpoints using the ent db client |
|
otelx
Package otelx provides configuration options for various OpenTelemetry integrations.
|
Package otelx provides configuration options for various OpenTelemetry integrations. |
|
passwd
Package passwd provides fancy crypto shit for passwords
|
Package passwd provides fancy crypto shit for passwords |
|
providers/github
Package github provides GitHub OAuth2 login and callback handlers.
|
Package github provides GitHub OAuth2 login and callback handlers. |
|
providers/google
Package google provides Google OAuth2 login and callback handlers.
|
Package google provides Google OAuth2 login and callback handlers. |
|
providers/oauth2
Package oauth2 provides handlers for OAuth2 login and callback requests.
|
Package oauth2 provides handlers for OAuth2 login and callback requests. |
|
providers/oidc
Package oidc provides oidc authentication helpers and flow
|
Package oidc provides oidc authentication helpers and flow |
|
providers/webauthn
Package webauthn is a provider package offering Passkey login functionality
|
Package webauthn is a provider package offering Passkey login functionality |
|
rout
Package rout is a semi-centralized method of handling and surfacing user facing errors
|
Package rout is a semi-centralized method of handling and surfacing user facing errors |
|
sessions
Package sessions manages users sessions
|
Package sessions manages users sessions |
|
testutils
Package testutils provides utilities for use in tests
|
Package testutils provides utilities for use in tests |
|
tokens
Package tokens creates tokenmanager, responsible for signing, issuing, and validating tokens
|
Package tokens creates tokenmanager, responsible for signing, issuing, and validating tokens |
|
utils/cli/rows
Package rows makes CLI output pretty
|
Package rows makes CLI output pretty |
|
utils/cli/tables
Package tables is a collection of functions that generate tables for the CLI
|
Package tables is a collection of functions that generate tables for the CLI |
|
utils/cli/useragent
Package useragent is a utility package for generating the user agent string for the CLI
|
Package useragent is a utility package for generating the user agent string for the CLI |
|
utils/dumper
Package dumper is a utility package for dumping http request contents
|
Package dumper is a utility package for dumping http request contents |
|
utils/emails
Package emails contains all of the logic and templating for sending emails within the datum services - it is built to allow for other email providers other than sendgrid if required but is mostly specific to sendgrid as that's our email provider of choice.
|
Package emails contains all of the logic and templating for sending emails within the datum services - it is built to allow for other email providers other than sendgrid if required but is mostly specific to sendgrid as that's our email provider of choice. |
|
utils/emails/mock
Package mock provides intermediary data structures to assist in testing and debugging sending an email without actually sending the email - testing the creation and sending of email typically requires some kind of tracking of what was sent or not and metadata about the message which is what is setup here
|
Package mock provides intermediary data structures to assist in testing and debugging sending an email without actually sending the email - testing the creation and sending of email typically requires some kind of tracking of what was sent or not and metadata about the message which is what is setup here |
|
utils/gravatar
Package gravatar generates random avatar images for users or organizations on sign up which they can override with their own avatars
|
Package gravatar generates random avatar images for users or organizations on sign up which they can override with their own avatars |
|
utils/marionette
Package marionette is a golang based task manager with scheduling, backoff, future scheduling built in.
|
Package marionette is a golang based task manager with scheduling, backoff, future scheduling built in. |
|
utils/pdf/invoice
nolint: gomnd
|
nolint: gomnd |
|
utils/sendgrid
Package sendgrid is a utiliuty package for interfacing with Sendgrid, a third party transactional email provider
|
Package sendgrid is a utiliuty package for interfacing with Sendgrid, a third party transactional email provider |
|
utils/sentry
Package sentry is a helper package for sentry integration
|
Package sentry is a helper package for sentry integration |
|
utils/slack
Package slack holds slack integration clients and message definitions
|
Package slack holds slack integration clients and message definitions |
|
utils/slice
Package sliceutil contains utilities for working with slices in Go
|
Package sliceutil contains utilities for working with slices in Go |
|
utils/sqlite
Package sqlite implements a connect hook around the sqlite3 driver so that the underlying connection can be fetched from the driver for more advanced operations such as backups.
|
Package sqlite implements a connect hook around the sqlite3 driver so that the underlying connection can be fetched from the driver for more advanced operations such as backups. |
|
utils/storage
Package storage provides a methods and interfaces to access various storage systems and objects
|
Package storage provides a methods and interfaces to access various storage systems and objects |
|
utils/storage/fs
Package fs provides a storage implementation for the local filesystem
|
Package fs provides a storage implementation for the local filesystem |
|
utils/storage/gcs
Package gcs provides a storage implementation for GCS
|
Package gcs provides a storage implementation for GCS |
|
utils/storage/s3
Package s3 provides a storage implementation for S3
|
Package s3 provides a storage implementation for S3 |
|
utils/totp
Package totp provides code generation for TOTP (RFC 6238) and HOTP (RFC 4226)
|
Package totp provides code generation for TOTP (RFC 6238) and HOTP (RFC 4226) |
|
utils/ulids
Package ulids is a lightweight wrapper around the github.com/oklog/ulid package that provides some common functionality (like checking if a ULID is null or is zero) as well as a process-global, cryptographically random, monotonic, and thread-safe ulid generation mechanism that can be used from external packages
|
Package ulids is a lightweight wrapper around the github.com/oklog/ulid package that provides some common functionality (like checking if a ULID is null or is zero) as well as a process-global, cryptographically random, monotonic, and thread-safe ulid generation mechanism that can be used from external packages |
|
utils/viperconfig
Package viperconfig is a fancy interface allowing you to bind viper commands in your individual packages without having to make some insane monolithic file Credit goes to the stacklok folks: https://github.com/stacklok/minder/blob/main/internal/util/configutils.go TO DO: convert allthethings over to using utils.BindConfigFlag()
|
Package viperconfig is a fancy interface allowing you to bind viper commands in your individual packages without having to make some insane monolithic file Credit goes to the stacklok folks: https://github.com/stacklok/minder/blob/main/internal/util/configutils.go TO DO: convert allthethings over to using utils.BindConfigFlag() |
Click to show internal directories.
Click to hide internal directories.