thema

package module
v0.0.0-...-067284b Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 26, 2023 License: Apache-2.0 Imports: 23 Imported by: 28

README

Thema

Thema is a system for writing schemas. Much like JSON Schema or OpenAPI, it is general-purpose and its most obvious application is as an IDL. However, those systems treat changing schemas as out of scope: a single version of a schema for some object is the atomic unit, and versioning is left to opaque strings in external systems like git or HTTP. Thema, by contrast, makes schema change a first-class system property: the atomic unit is the set of schema for some object, iteratively appended to over time as requirements evolve.

Thema's approach is novel, so an analogy to the familiar may help. "Branching by abstraction" suggests that you refactor large applications not with long-running VCS branches and big-bang merges, but by letting old and new code live side-by-side on main, and choosing between them with logical gates, like feature flags. Thema is "schema versioning by abstraction": all versions of a schema live side-by-side on main, within logical structures Thema defines.

This holistic view allows Thema to act like a typechecker, but for change-safety between schema versions: either schema versions must be backwards compatible, or there must exist logic to translate a valid instance of schema from one schema version to the next. CUE, the language in which Thema schemas are written, allows Thema to mechanically verify these properties.

These capabilities make Thema a general framework for decoupling the evolution of communicating systems. This can be outward-facing: Thema's guardrails allow anyone to create APIs with Stripe's renowned backwards compatibility guarantees. Or it can be inward-facing: or to change the messages passed in a mesh of microservices without intricately orchestrating deployment.

Learn more in our docs, or in this overview video! (Some things have been renamed since that video, but the logic is unchanged.)

Usage

Thema defines the way schemas are written, organizing each object's history into a "lineage." Once authored, Thema also provides tools for working with lineages via a few basic operations. There are a few different usage patterns, all largely equivalent in capability:

  • CLI: a CLI command that provides access to Thema's basic operations, one lineage per invocation. Use it for fast exploration and testing of schemas, or as a tool in CI.
  • Server: An HTTP server that provides access to Thema's basic operations for a configurable set of lineages. Run it as a stateless sidecar in your infrastructure or microservice mesh.
  • Library: a library, importable in your application code, that provides a convenient interface to Thema's basic operations, as well as helpers for common usage patterns. Naturally the most flexible, and the recommended approach for creating new helpers, such as code generators, API generators, or a whole Kubernetes operator framework. (Currently only for Go[^evaluator])

The CLI and server modes are bundled together in the thema command. To install:

go install github.com/grafana/thema/cmd/thema@latest

Maturity

Thema is a young project. The goals are large, but bounded: we will know when the core system is complete. And it mostly is, now - though some breaking changes to how schemas are written are planned before reaching stability.

It is not yet recommended to replace established, stable systems with Thema, but experimenting with doing so is reasonable (and appreciated!). For newer projects, Thema may be a good choice today; the decision is likely to come down to whether the long-term benefit of a simpler architecture for authoring, composing and evolving schema will offset the short-term cost of some incomplete functionality and breaking changes.

A number of systems partially overlap with Thema - for some data, rolling together a set of schema with the relations between those schema.

  • Project Cambria - Thema's closest analogue. Limited in verifiability by (intentionally) being without a notion of linear schema ordering and versioning, and because schema and translations are written in a Turing complete language (Typescript).
  • Kubernetes resources and webhook conversions - Similar goals: multiple versions of resources (schema) and convertibility between them. Limited in verifiability by relying on convention for grouping schemas, and by expressing translation in a Turing complete language (Go).
  • Stripe's HTTP API - exhibits the backwards compatibility properties an API can have that arise from a schema system with translatability.

[^evaluator]: Using Thema as a library in a language depends on a CUE evaluator for that language. Currently, the only CUE evaluator is written in Go.

Documentation

Index

Constants

This section is empty.

Variables

View Source
var CueFS embed.FS

CueFS contains the raw .cue files that comprise the core thema system, making them available directly in Go.

This virtual filesystem is relied on by the Go functions exported by this library, effectively co-versioning the Go logic with the CUE logic. It is exported such that other Go packages have the unfettered capability to create their own thema-based systems.

View Source
var CueJointFS embed.FS

CueJointFS contains the raw thema .cue files, as well as the cue.mod directory.

View Source
var ErrPointerDepth = errors.New("assignability does not support more than one level of pointer indirection")

ErrPointerDepth indicates that a Go type having pointer indirection depth greater than 1, such as

**struct{ V: string })

was provided to a Thema func that checks assignability, such as BindType.

Functions

func AssignableTo

func AssignableTo(sch Schema, T any) error

AssignableTo indicates whether all valid instances of the provided Thema schema can be assigned to the provided Go type.

If the provided T is a pointer, it will be dereferenced before verification. Double pointers (or any n-pointer > 1) are not allowed.

The provided T must be struct-kinded, as it is a requirement that all Thema schemas are of base type struct.

type MyType struct {
	MyField string `json:"myfield"`
}

AssignableTo(sch, &MyType{})

Assignability rules are specified here: https://github.com/grafana/thema/blob/main/docs/invariants.md#go-assignability

func IsAppendOnly

func IsAppendOnly(oldLineage Lineage, newLineage Lineage) error

IsAppendOnly returns nil if the new lineage only contains new schemas compared to the old one. It returns an error if old schemas are updated or deleted.

Types

type Assignee

type Assignee any

Assignee is a type constraint used by Thema generics for type parameters where there exists a particular Schema that is AssignableTo the type.

This property is not representable in Go's static type system, as Thema types are dynamic, and AssignableTo() is a runtime check. Thus, the only actual type constraint Go's type system can be made aware of is any.

Instead, Thema's implementation guarantees that it is only possible to instantiate a generic type with an Assignee type parameter if the relevant AssignableTo() relation has already been verified, and there is an unambiguous relationship between the generic type and the relevant Schema.

For example: for TypedSchema[T Assignee], it is the related Schema. With TypedInstance[T Assignee], the related schema is returned from its TypedSchema() method.

As this type constraint is simply any, it exists solely as a signal to the human reader that the relation to a Schema exists, and that the relation has been verified in any properly instantiated type carrying this generic type constraint. (Improperly instantiated generic Thema types panic upon calls to any of their methods)

type BindOption

type BindOption bindOption

A BindOption defines options that may be specified only at initial construction of a Lineage via BindLineage.

func ImperativeLenses

func ImperativeLenses(lenses ...ImperativeLens) BindOption

ImperativeLenses takes a slice of ImperativeLens. These lenses will be executed on calls to Instance.Translate.

Currently, the entire lens set must be provided in either Go or CUE. This restriction may be relaxed in the future to allow a mix of Go and CUE lenses, or to allow Go funcs to supersede CUE lenses as a performance optimization.

When providing lenses in this way, BindLineage will fail unless exactly the set of expected lenses is provided. The correctness of the function bodies cannot be pre-verified in this way, as Go is Turing-complete, but it is enforced at runtime that lenses return an Instance of the schema version they claim to in [ImperativeLens.To].

Writing lenses in Go means that pure native CUE is no longer sufficient to produce a valid lineage. As a result, lineages are no longer portable outside of Go programs with compile-time access to the Go-defined lenses.

func SkipBuggyChecks

func SkipBuggyChecks() BindOption

SkipBuggyChecks indicates that BindLineage should skip validation checks which have known bugs (e.g. panics) for certain should-be-valid CUE inputs.

By default, BindLineage performs these checks anyway, as otherwise the default behavior of BindLineage is to not provide the guarantees it's supposed to provide.

As Thema and CUE move towards maturity and the set of validations that are both a) necessary and b) buggy empties out, this will naturally become a no-op. At that point, this function will be marked deprecated.

Ratcheting up verification checks in this way does mean that any code relying on this to bypass verification in BindLineage may begin failing in future versions of Thema if the underlying lineage being verified doesn't comply with a planned invariant.

type CUEWrapper

type CUEWrapper interface {
	// Underlying returns the underlying cue.Value wrapped by the object.
	Underlying() cue.Value
}

A CUEWrapper wraps a cue.Value, and can return that value for inspection.

type ConvergentLineage

type ConvergentLineage[T Assignee] interface {
	Lineage

	TypedSchema() TypedSchema[T]
}

ConvergentLineage is a lineage where exactly one of its contained schemas is associated with a Go type - a TypedSchema[Assignee], as returned from BindType.

This variant of lineage is intended to directly support the primary anticipated use pattern for Thema within a Go program: accepting all historical forms of an object's schema as input to the program, but writing the program against just one version.

This process is known as version multiplexing. See github.com/grafana/thema/vmux.

type ConvergentLineageFactory deprecated

type ConvergentLineageFactory[T Assignee] func(*Runtime, ...BindOption) (ConvergentLineage[T], error)

A ConvergentLineageFactory is the same as a LineageFactory, but for a ConvergentLineage.

There is no reason to provide both a ConvergentLineageFactory and a LineageFactory, as the latter is always reachable from the former. As such, idiomatic naming conventions are unchanged.

Deprecated: having an explicit type for this adds little value.

type FieldRef

type FieldRef struct {
	Path  string      `json:"path"`
	Value interface{} `json:"value"`
}

FieldRef identifies a path/field and the value in it within a Lacuna.

type ImperativeLens

type ImperativeLens struct {
	To, From SyntacticVersion
	Mapper   func(inst *Instance, to Schema) (*Instance, error)
}

ImperativeLens is a lens transformation defined as a Go function, rather than in native CUE alongside the lineage.

See ImperativeLenses for more information.

type Instance

type Instance struct {
	// contains filtered or unexported fields
}

Instance represents data that is a valid instance of a Thema Schema.

It is not possible to create a valid Instance directly. They can only be obtained by successful call to [Schema.Validate].

func (*Instance) AsPredecessor

func (i *Instance) AsPredecessor() (*Instance, TranslationLacunas, error)

AsPredecessor translates the instance into the form specified by the predecessor schema.

func (*Instance) AsSuccessor

func (i *Instance) AsSuccessor() (*Instance, TranslationLacunas, error)

AsSuccessor translates the instance into the form specified by the successor schema.

func (*Instance) Dehydrate

func (i *Instance) Dehydrate() *Instance

Dehydrate returns a copy of the Instance with all default values specified by the schema removed.

NOTE dehydration implementation is a WIP. If errors are encountered, the original input is returned unchanged.

func (*Instance) Hydrate

func (i *Instance) Hydrate() *Instance

Hydrate returns a copy of the Instance with all default values specified by the schema included.

NOTE hydration implementation is a WIP. If errors are encountered, the original input is returned unchanged.

func (*Instance) Schema

func (i *Instance) Schema() Schema

Schema returns the Schema corresponding to this instance.

func (*Instance) Translate

Translate transforms the provided Instance to an Instance of a different Schema from the same Lineage. A new *Instance is returned representing the transformed value, along with any lacunas accumulated along the way.

Forward translation within a major version (e.g. 0.0 to 0.7) is trivial, as all those schema changes are established as backwards compatible by Thema's lineage invariants. In such cases, the lens is referred to as implicit, as the lineage author does not write it, with translation relying on simple unification. Lacunas cannot be emitted from such translations.

Forward translation across major versions (e.g. 0.0 to 1.0), and all reverse translation regardless of sequence boundaries (e.g. 1.1 to either 1.0 or 0.0), is nontrivial and relies on explicitly defined lenses, which introduce room for lacunas and author judgment.

Thema translation is non-invertible by design. That is, Thema does not seek to generally guarantee that translating an instance from 0.0->1.0->0.0 will result in the exact original data. Input state preservation can be fully achieved in the program depending on Thema, so we avoid introducing complexity into Thema that is not essential for all use cases.

Errors only occur in cases where lenses were written in an unexpected way - for example, not all fields were mapped over, and the resulting object is not concrete. All errors returned from this func will children of terrors.ErrInvalidLens.

func (*Instance) Underlying

func (i *Instance) Underlying() cue.Value

Underlying returns the cue.Value representing the data contained in the Instance.

type Lacuna

type Lacuna struct {
	// The field path(s) and their value(s) in the pre-translation instance
	// that are relevant to the lacuna.
	SourceFields []FieldRef `json:"sourceFields,omitempty"`

	// The field path(s) and their value(s) in the post-translation instance
	// that are relevant to the lacuna.
	TargetFields []FieldRef `json:"targetFields,omitempty"`
	Type         LacunaType `json:"type"`

	// A human-readable message describing the gap in translation.
	Message string `json:"message"`
}

A Lacuna represents a semantic gap in a Lens's mapping between schemas.

For any given mapping between schema, there may exist some valid values and intended semantics on either side that are impossible to precisely translate. When such gaps occur, and an actual schema instance falls into such a gap, the Lens is expected to emit Lacuna that describe the general nature of the translation gap.

A lacuna may be unconditional (the gap exists for all possible instances being translated between the schema pair) or conditional (the gap only exists when certain values appear in the instance being translated between schema). However, the conditionality of lacunas is expected to be expressed at the level of the lens, and determines whether a particular lacuna object is created; the production of a lacuna object as the output of the translation of a particular instance indicates the lacuna applies to that specific translation.

type LacunaType

type LacunaType uint16

LacunaType assigns numeric identifiers to different classes of Lacunas.

FIXME this is a terrible way of doing this and needs to change

type Lineage

type Lineage interface {
	CUEWrapper

	// Name returns the name of the object schematized by the lineage, as declared
	// in the lineage's `name` field.
	Name() string

	// ValidateAny checks that the provided data is valid with respect to at
	// least one of the schemas in the lineage. The oldest (smallest) schema against
	// which the data validates is chosen. A nil return indicates no validating
	// schema was found.
	//
	// While this method takes a cue.Value, this is only to avoid having to trigger
	// the translation internally; input values must be concrete. To use
	// incomplete CUE values with Thema schemas, prefer working directly in CUE,
	// or if you must, rely on Underlying().
	//
	// TODO should this instead be interface{} (ugh ugh wish Go had tagged unions) like FillPath?
	ValidateAny(data cue.Value) *Instance

	// Schema returns the schema identified by the provided version, if one exists.
	//
	// Only the [0, 0] schema is guaranteed to exist in all valid lineages.
	Schema(v SyntacticVersion) (Schema, error)

	// First returns the first Schema in the lineage (v0.0). Thema requires that all
	// valid lineages contain at least one schema, so this is guaranteed to exist.
	First() Schema

	// Latest returns the newest Schema in the lineage - largest minor version
	// within the largest major version.
	//
	// Thema requires that all valid lineages contain at least one schema, so schema
	// is is guaranteed to exist, even if it's the 0.0 version.
	//
	// EXERCISE CAUTION WITH THIS METHOD. Relying on Latest is appropriate and
	// necessary for some use cases, such as keeping Thema declarations and
	// generated code in sync within a single repository. But use in the wrong
	// context - usually cross repository, loosely coupled, dependency
	// management-like contexts - can completely undermine Thema's translatability
	// invariants.
	//
	// If you're not sure, ask yourself: when a breaking change to this lineage is
	// published, what would that break downstream, and will the users who experience
	// that breakage be expecting it to happen?
	//
	// If the user would be expecting the breakage, using Latest is probably appropriate.
	// Otherwise, it is probably preferable to pick an explicit version number.
	Latest() Schema

	// All returns all Schemas in the lineage. Thema requires that all valid lineages
	// contain at least one schema, so this is guaranteed to contain at least one element.
	All() []Schema

	// Runtime returns the thema.Runtime instance with which this lineage was built.
	Runtime() *Runtime
	// contains filtered or unexported methods
}

A Lineage is the top-level container in Thema, holding the complete evolutionary history of a particular kind of object: every schema that has ever existed for that object, and the lenses that allow translating between those schema versions.

Lineages may only be produced by calling BindLineage.

func BindLineage

func BindLineage(v cue.Value, rt *Runtime, opts ...BindOption) (Lineage, error)

BindLineage takes a raw cue.Value, checks that it correctly follows Thema's invariants, such as translatability and backwards compatibility version numbering. If these checks succeed, a Lineage is returned.

This function is the only way to create non-nil Lineage objects. As a result, all non-nil instances of Lineage in any Go program are guaranteed to follow Thema invariants.

type LineageFactory deprecated

type LineageFactory func(*Runtime, ...BindOption) (Lineage, error)

A LineageFactory returns a Lineage, which is immutably bound to a single instance of #Lineage declared in CUE.

LineageFactory funcs are intended to be the main Go entrypoint to all of the operations, guarantees, and capabilities of Thema lineages. Lineage authors should generally define and export one instance of LineageFactory per #Lineage instance.

It is idiomatic to name LineageFactory funcs after the "name" field on the lineage they return:

func <name>Lineage ...

If the Go package and lineage name are the same, the name should be omitted from the builder func to reduce stutter:

func Lineage ...

Deprecated: having an explicit type for this adds little value.

type Runtime

type Runtime struct {
	// contains filtered or unexported fields
}

Runtime holds the set of CUE constructs available in the Thema CUE package, allowing Thema's Go code to internally reuse the same native CUE functionality.

Each Thema Runtime is bound to a single cue.Context, determined by the parameter passed to NewRuntime.

func NewRuntime

func NewRuntime(ctx *cue.Context) *Runtime

NewRuntime parses, loads and builds a full CUE instance/value representing all of the logic in the Thema CUE package (github.com/grafana/thema), and returns a Runtime instance ready for use.

Building is performed using the provided cue.Context. Passing a nil context will panic.

This function is the canonical way to make Thema logic callable from Go code.

func (*Runtime) Context

func (rt *Runtime) Context() *cue.Context

Context returns the *cue.Context in which this runtime was built.

func (*Runtime) Underlying

func (rt *Runtime) Underlying() cue.Value

Underlying returns the underlying cue.Value representing the whole Thema CUE library (github.com/grafana/thema).

type Schema

type Schema interface {
	CUEWrapper

	// Validate checks that the provided data is valid with respect to the
	// schema. If valid, the data is wrapped in an [Instance] and returned.
	// Otherwise, a nil Instance is returned along with an error detailing the
	// validation failure.
	//
	// While Validate takes a cue.Value, this is only to avoid having to trigger
	// the translation internally; input values must be concrete. Behavior of
	// this method is undefined for incomplete values.
	//
	// The concreteness requirement may be loosened in future versions of Thema. To
	// use incomplete CUE values with Thema schemas, prefer working directly in CUE,
	// or call [Schema.Underlying] to work directly with the underlying CUE API.
	//
	// TODO should this instead be interface{} (ugh ugh wish Go had tagged unions) like FillPath?
	Validate(data cue.Value) (*Instance, error)

	// Successor returns the next schema in the lineage, or nil if it is the last schema.
	Successor() Schema

	// Predecessor returns the previous schema in the lineage, or nil if it is the first schema.
	Predecessor() Schema

	// LatestInMajor returns the Schema with the newest (largest) minor version
	// within this Schema's major version. If the receiver Schema is the latest, it
	// will return itself.
	LatestInMajor() Schema

	// Version returns the schema's version number.
	Version() SyntacticVersion

	// Lineage returns the lineage that contains this schema.
	Lineage() Lineage

	// Examples returns the set of examples of this schema defined in the original
	// lineage. The string key is the name given to the example.
	Examples() map[string]*Instance
	// contains filtered or unexported methods
}

Schema represents a single, complete schema from a thema lineage. A Schema's Validate() method determines whether some data constitutes an Instance.

func SchemaP

func SchemaP(lin Lineage, v SyntacticVersion) Schema

SchemaP returns the schema identified by the provided version. If no schema exists in the lineage with the provided version, it panics.

This is a simple convenience wrapper on the Lineage.Schema() method.

type SyntacticVersion

type SyntacticVersion [2]uint

SyntacticVersion is a two-tuple of uints describing the position of a schema within a lineage. Syntactic versions are Thema's canonical version numbering system.

The first element is the index of the sequence containing the schema within the lineage, and the second element is the index of the schema within that sequence.

func LatestVersion deprecated

func LatestVersion(lin Lineage) SyntacticVersion

LatestVersion returns the version number of the newest (largest) schema version in the provided lineage.

Deprecated: call Lineage.Latest().Version().

func LatestVersionInSequence deprecated

func LatestVersionInSequence(lin Lineage, seqv uint) (SyntacticVersion, error)

LatestVersionInSequence returns the version number of the newest (largest) schema version in the provided sequence number.

An error indicates the number of the provided sequence does not exist.

Deprecated: call Schema.LatestInMajor().Version() after loading a schema in the desired major version.

func ParseSyntacticVersion

func ParseSyntacticVersion(s string) (SyntacticVersion, error)

ParseSyntacticVersion parses a canonical representation of a SyntacticVersion (e.g. "0.0") from a string.

func SV

func SV(seqv, schv uint) SyntacticVersion

SV creates a SyntacticVersion.

A trivial helper to avoid repetitive Go-stress disorder from countless instances of typing:

SyntacticVersion{0, 0}

func (SyntacticVersion) Less

func (sv SyntacticVersion) Less(osv SyntacticVersion) bool

Less reports whether the receiver SyntacticVersion is less than the provided one, consistent with the expectations of the stdlib sort package.

func (SyntacticVersion) String

func (sv SyntacticVersion) String() string

type TranslationLacunas

type TranslationLacunas interface {
	AsList() []Lacuna
}

TranslationLacunas defines common patterns for unary and composite lineages in the lacunas their translations emit.

type TypedInstance

type TypedInstance[T Assignee] struct {
	*Instance
	// contains filtered or unexported fields
}

TypedInstance represents data that is a valid instance of a Thema TypedSchema.

A TypedInstance is to a TypedSchema as an Instance is to a Schema.

It is not possible to create a valid TypedInstance directly. They can only be obtained by successful call to [TypedSchema.Validate].

func BindInstanceType

func BindInstanceType[T Assignee](inst *Instance, tsch TypedSchema[T]) (*TypedInstance[T], error)

BindInstanceType produces a TypedInstance, given an Instance and a TypedSchema derived from its Instance.Schema().

The only possible error occurs if the TypedSchema is not derived from the Instance.Schema().

func (*TypedInstance[T]) TypedSchema

func (inst *TypedInstance[T]) TypedSchema() TypedSchema[T]

TypedSchema returns the TypedSchema corresponding to this instance.

This method is identical to Instance.Schema, except that it returns the already-typed variant.

func (*TypedInstance[T]) Value

func (inst *TypedInstance[T]) Value() (T, error)

Value returns a Go struct of this TypedInstance's generic Assignee type, populated with the data contained in this instance, including default values, etc.

This method is similar to json.Unmarshal - it decodes serialized data into a standard Go type for working with in all the usual ways.

func (*TypedInstance[T]) ValueP

func (inst *TypedInstance[T]) ValueP() T

ValueP is the same as Value, but panics if an error is encountered.

type TypedSchema

type TypedSchema[T Assignee] interface {
	Schema

	// NewT returns a new instance of T, but with schema-specified defaults for its
	// field values instead of Go zero values. Fields without a schema-specified default
	// are populated with standard Go zero values.
	NewT() T

	// ValidateTyped performs validation identically to [Schema.Validate], but
	// returns a TypedInstance on success.
	ValidateTyped(data cue.Value) (*TypedInstance[T], error)

	// ConvergentLineage returns the ConvergentLineage that contains this schema.
	ConvergentLineage() ConvergentLineage[T]
}

TypedSchema is a Thema schema that has been bound to a particular Go type, per Thema's assignability rules.

func BindType

func BindType[T Assignee](sch Schema, t T) (TypedSchema[T], error)

BindType produces a TypedSchema, given a Schema that is AssignableTo the Assignee type parameter T. T must be struct-kinded, and at most one level of pointer indirection is allowed.

An error is returned if the provided Schema is not assignable to the given struct type.

Directories

Path Synopsis
cmd
encoding
cue
gocode
Package tgo provides tools for generating native Go types from Thema's lineage and schema abstractions.
Package tgo provides tools for generating native Go types from Thema's lineage and schema abstractions.
typescript
Package typescript provides tools for generating native TypeScript types from Thema's lineage and schema abstractions.
Package typescript provides tools for generating native TypeScript types from Thema's lineage and schema abstractions.
internal
Package vmux provides utilities that make it easy to implement "version multiplexing" with your Thema lineages.
Package vmux provides utilities that make it easy to implement "version multiplexing" with your Thema lineages.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL