cuetils

Cuetils

CUE utilities and helpers for working with tree based objects in any combination of CUE, Yaml, and JSON.

Using

As a command line binary

The cuetils CLI is useful for bulk operations or when you don't want to write extra CUE or Go.

Download a release from GitHub.

cuetils -h

As a Go library

The Go libraries are optimized and have more capabilities.*

go get github.com/hofstadter-io/cuetils@latest

import "github.com/hofstadter-io/cuetils/structural"

* work in progress, unoptimized use the CUE helper and RecurseN

As a CUE library

The CUE libraries all use the RecurseN helper and make use of the "function" pattern. You can also write custom operators.

Add to your project with hof mod or another method. See: https://cuetorials.com/first-steps/modules-and-packages/#dependency-management

import "github.com/hofstadter-io/cuetils/structural"

Structural Helpers

• Count the nodes in an object
• Depth how deep an object is
• Diff two objects, producing a structured diff
• Patch an object, producing a new object
• Pick a subojbect from another, selecting only the parts you want
• Maks a subobject from another, filtering out parts you don't want
• Replace with a subobject, updating fields found
• Upsert with a subobject, updating and adding fields
• Transform one or more objects into another using CUE
• Validate one or more objects with the power of CUE

The helpers work by checking if two operands unify. We try to make note of the edge cases where appropriate, as it depends on both the operation and the method you are using (CUE, Go, or cuetils).

Count

#Count calculates how many nodes are in an object.

a: {
	foo: "bar"
	a: b: c: "d"
}
cow: "moo"

$ cuetils count tree.cue
9

import "github.com/hofstadter-io/cuetils/structural"

tree: {
	a: {
		foo: "bar"
		a: b: c: "d"
	}
	cow: "moo"
}

depth: (structural.#Count & { #in: tree }).out
depth: 9

Depth

#Depth calculates the deepest branch of an object.

a: {
	foo: "bar"
	a: b: c: "d"
}
cow: "moo"

$ cuetils depth tree.cue
5

import "github.com/hofstadter-io/cuetils/structural"

tree: {
	a: {
		foo: "bar"
		a: b: c: "d"
	}
	cow: "moo"
}

depth: (structural.#Depth & { #in: tree }).out
depth: 5

import "github.com/hofstadter-io/cuetils/structural"

Diff

#Diff computes a semantic diff object

-- a.json --
{
	"a": {
		"b": "B"
	}
}
-- b.yaml --
a:
  c: C
b: B

$ cuetils diff a.json b.yaml
{
	"+": {
		b: "B"
	}
	a: {
		"-": {
			b: "B"
		}
		"+": {
			c: "C"
		}
	}
}

import "github.com/hofstadter-io/cuetils/structural"

x: {
	a: "a"
	b: "b"
	d: "d"
	e: {
		a: "a"
		b: "b"
		d: "d"
	}
}

y: {
	b: "b"
	c: "c"
	d: "D"
	e:  {
		b: "b"
		c: "c"
		d: 1
	}
}

diff: (structural.#Diff & { #X: x, #Y: y }).diff
diff: {
	"-": {
		a: "a"
		d: "d"
	}
	e: {
		"-": {
			a: "a"
			d: "d"
		}
		"+": {
			d: 1
			c: "c"
		}
	}
	"+": {
		d: "D"
		c: "c"
	}
}

import "github.com/hofstadter-io/cuetils/structural"

For diff and patch, int & 1 like expressions will not be detected. Lists are not currently supported for diff and patch. It may be workable if the list sizes are known and order consistent. Associative Lists may solve this issue. We don't currently have good syntax for specifying the key to match elements on.

Patch

#Patch applies a diff object

-- patch.json --
{
  "+": {
    b: "B"
  }
  a: {
    "-": {
      b: "B"
    }
    "+": {
      c: "C"
    }
  }
}
-- a.json --
{
	"a": {
		"b": "B"
	}
}

$ cuetils patch patch.json a.json
{
	b: "B"
	a: {
		c: "C"
	}
}

import "github.com/hofstadter-io/cuetils/structural"

x: {
	a: "a"
	b: "b"
	d: "d"
	e: {
		a: "a"
		b: "b"
		d: "d"
	}
}

p: {
	"-": {
		a: "a"
		d: "d"
	}
	e: {
		"-": {
			a: "a"
			d: "d"
		}
		"+": {
			d: 1
			c: "c"
		}
	}
	"+": {
		d: "D"
		c: "c"
	}
}

patch: (structural.#Patch & { #X: x, #Y: y }).patch
patch: {
	b: "b"
	c: "c"
	d: "D"
	e:  {
		b: "b"
		c: "c"
		d: 1
	}
}

import "github.com/hofstadter-io/cuetils/structural"

Pick

#Pick extracts a subobject

-- pick.cue --
{
	a: {
		b: string
	}
	c: int
	d: "D"
}
-- a.json --
{
	"a": {
		"b": "B"
	},
	"b": 1,
	"c": 2,
	"d": "D"
}

$ cuetils pick pick.cue a.json
{
	a: {
		b: "B"
	}
	c: 2
	d: "D"
}

import "github.com/hofstadter-io/cuetils/structural"

x: {
	a: "a"
	b: "b"
	d: "d"
	e: {
		a: "a"
		b: "b1"
		d: "cd"
	}
}
p: {
	b: string
	d: int
	e: {
		a: _
		b: =~"^b"
		d: =~"^d"
	}
}
pick: (structural.#Pick & { #X: x, #P: p }).pick
pick: {
	b: "b"
	e:  {
		a: "a"
		b: "b1"
	}
}

import "github.com/hofstadter-io/cuetils/structural"

Mask

#Mask removes a subobject

-- mask.cue --
{
	a: {
		b: string
	}
	c: int
	d: "D"
}
-- a.json --
{
	"a": {
		"b": "B"
		"c": "C"
	},
	"b": 1,
	"c": 2,
	"d": "D"
}

$ cuetils mask mask.cue a.json
{
	a: {
		c: "C"
	}
	b: 1
}

import "github.com/hofstadter-io/cuetils/structural"

x: {
	a: "a"
	b: "b"
	d: "d"
	e: {
		a: "a"
		b: "b1"
		d: "cd"
	}
}
m: {
	b: string
	d: int
	e: {
		a: _
		b: =~"^b"
		d: =~"^d"
	}
}
mask: (structural.#Mask & { #X: x, #M: m }).mask
mask: {
	a: "a"
	d: "d"
	e:  {
		d: "cd"
	}
}

import "github.com/hofstadter-io/cuetils/structural"

Replace

-- replace.cue --
{
	a: {
		b: "b"
	}
	d: "d"
	e: "E"
}
-- a.json --
{
	"a": {
		"b": "B"
	},
	"b": 1,
	"c": 2,
	"d": "D"
}

$ cuetils replace replace.cue a.json
{
	b: 1
	c: 2
	a: {
		b: "b"
	}
	d: "d"
}

import "github.com/hofstadter-io/cuetils/structural"

import "github.com/hofstadter-io/cuetils/structural"

Upsert

-- upsert.cue --
{
	a: {
		b: "b"
	}
	d: "d"
	e: "E"
}
-- a.json --
{
	"a": {
		"b": "B"
	},
	"b": 1,
	"d": "D"
}

$ cuetils upsert upsert.cue a.json
{
	b: 1
	a: {
		b: "b"
	}
	d: "d"
	e: "E"
}

import "github.com/hofstadter-io/cuetils/structural"

import "github.com/hofstadter-io/cuetils/structural"

Transform

-- t.cue --
#In: _        // required, filled in during processing
{
	B: #In.a.b
	C: #In.a.c
	D: #In.d
}

-- a.json --
{
	"a": {
		"b": "b"
		"c": "c"
	}
	"d": "d"
}

$ cuetils transform t.cue a.json
{
	B: "b"
	C: "c"
	D: "d"
}

import "github.com/hofstadter-io/cuetils/structural"

Validate

-- schema.cue --
{
	a: {
		b: int
	}
	c: int
	d: "D"
}
-- a.json --
{
	"a": {
		"b": "B"
	},
	"b": 1,
	"c": 2,
	"d": "D"
}

$ cuetils validate schema.cue a.json
a.json
----------------------
a.b: conflicting values "B" and int (mismatched types string and int):
    ./schema.cue:1:1
    ./schema.cue:3:6
    a.json:3:8


Errors in 1 file(s)

import "github.com/hofstadter-io/cuetils/structural"

RecurseN

A function factory for bounded recursion, defaulting to 20. This is a pattern to get around CUE's cycle detection by creating a struct with fields named for each iteration. See https://cuetorials.com/deep-dives/recursion/ for more details.

#RecurseN: {
	#maxiter: uint | *20
	#funcFactory: {
		#next: _
		#func: _
	}

	for k, v in list.Range(0, #maxiter, 1) {
		#funcs: "\(k)": (#funcFactory & {#next: #funcs["\(k+1)"]}).#func
	}
	#funcs: "\(#maxiter)": null

	#funcs["0"]
}

The core of the bounded recursion is this structural comprehension (unrolled for loop). The "recursive call" is made with the following pattern.

(#funcFactory & {#next: #funcs["\(k+1)"]}).#func

You can override the iterations with { #maxdepth: 100 } at the point of usage or by creating new helpers from the existing ones. You may need to adjust this

• up for deep objects
• down if runtime is an issue

import "github.com/hofstadter-io/cuetils/structural"

#LeaguesDeep: structural.#Depth & { #maxdepth: 10000 }

Custom Helpers

You can make new helpers by building on the #RecurseN pattern. You need two definitions, a factory and the user facing, recursed version.

package structural

import (
	"list"

	"github.com/hofstadter-io/cuetils/recurse"
)

// A function factory
#depthF: {
	// always required
	#next: _
	
	// the actual computation, must be named #func
	#func: {
		// you can have any args
		#in: _
		// or internal helpers
		#multi: {...} | [...]
		
		// the result, can be named anything
    depth: {
			// detect leafs
			if (#in & #multi) == _|_ { 1 }
			// detect struct
			if (#in & {...}) != _|_ {
				list.Max([for k,v in #in {(#next & {#in: v}).depth}]) + 1
			}
			// detect list
			if (#in & [...]) != _|_ {
				list.Max([for k,v in #in {(#next & {#in: v}).depth}])
			}
    }
	}
}

// The user facing, recursed version
#Depth: recurse.#RecurseN & {#funcFactory: #depthF}

The core of the recursive calling is:

(#next & {#in: v}).depth