rx

rx - Rego eXpression

Disclaimer: I am a Styra (the creators of OPA) employee. This project is not endorsed, affiliated with, or supported by Styra or the OPA project. You have no expectation of support from Styra or the OPA team if you use this software.

Early Stage Project: this project is still very much experimental. Expect things to change and break without notice.

rx (pronounced "rex") is inspired by jq, and aims to offer a similar set of capabilities oriented around running Rego queries. In the long run, rx is intended to make it as easy as possible to utilize Rego both interactively as a shell command, and within shell scripts. Though it is already possible to use Rego in this with via opa eval using the official opa command, rx is intended to be easier and simpler to use specifically in a shell context, and has additional convenience features to support this use case, such as more input and output formats, syntax highlighted output, and so on.

A simple demonstration (see the Showcase section for more):

$ data='{"foo": 7, "bar": 3, "baz": 12, "quux": 4, "spam": 6}'
$ # find all keys in data where the value is at least 5
$ echo "$data" | rx '{k | some k; input[k] > 5}'
[
	"baz",
	"foo",
	"spam"
]

Features

• Supported input formats
  • CSV (including comments, arbitrary delimiters, headers, and leading line skip)
  • JSON
  • YAML
• Supported output formats
  • JSON
  • YAML
• Syntax-highlighted output using chroma

Installation Instructions

With go install:

$ go install git.sr.ht/~charles/rx/cmd/rx@latest

With GNU Make:

$ git clone https://git.sr.ht/~charles/rx
$ cd rx
$ git checkout <tag name>  # skip if you want to use the latest development version
$ make install

Usage Instructions

See rx --help.

Showcase

Sample data:

$ cat sample_data/books.json
[
    {
        "title": "Writing An Interpreter In Go",
        "authors": ["Thorsten Ball"],
        "isbn": "978-3982016115",
        "year": 2018
    },
    {
        "title": "Writing A Compiler In Go",
        "authors": ["Thorsten Ball"],
        "isbn": "978-3982016108",
        "year": 2018
    },
    {
        "title": "The Go Programming Language",
        "authors": ["Alan A. A. Donovan", "Brian W. Kernighan"],
        "isbn": "978-0134190440",
        "year": 2015
    },
    {
        "title": "Hands-On GUI Application Development in Go",
        "authors": ["Andrew Williams"],
        "isbn": "978-1789138412",
        "year": 2019
    },
    {
        "title": "Building Cross-Platform GUI Applications with Fyne",
        "authors": ["Andrew Williams"],
        "isbn": "1800563167",
        "year": 2021
    }
]
$ cat sample_data/books.csv
title,"first author",isbn,year
Writing An Interpreter In Go,Thorsten Ball,978-3982016115,2018
Writing A Compiler In Go,Thorsten Ball,978-3982016108,2018
"The Go Programming Language","Alan A. A. Donovan",978-0134190440,2015
Hands-On GUI Application Development in Go,Andrew Williams,978-1789138412,2019
Building Cross-Platform GUI Applications with Fyne,Andrew Williams,1800563167,2021

Read JSON data:

$ rx < sample_data/books.json
[
	{
		"authors": [
			"Thorsten Ball"
		],
		"isbn": "978-3982016115",
		"title": "Writing An Interpreter In Go",
		"year": 2018
	},
	{
		"authors": [
			"Thorsten Ball"
		],
		"isbn": "978-3982016108",
		"title": "Writing A Compiler In Go",
		"year": 2018
	},
	{
		"authors": [
			"Alan A. A. Donovan",
			"Brian W. Kernighan"
		],
		"isbn": "978-0134190440",
		"title": "The Go Programming Language",
		"year": 2015
	},
	{
		"authors": [
			"Andrew Williams"
		],
		"isbn": "978-1789138412",
		"title": "Hands-On GUI Application Development in Go",
		"year": 2019
	},
	{
		"authors": [
			"Andrew Williams"
		],
		"isbn": "1800563167",
		"title": "Building Cross-Platform GUI Applications with Fyne",
		"year": 2021
	}
]

Read CSV data, with headers:

$ rx -i csv -H < sample_data/books.csv
[
	{
		"first author": "Thorsten Ball",
		"isbn": "978-3982016115",
		"title": "Writing An Interpreter In Go",
		"year": 2018
	},
	{
		"first author": "Thorsten Ball",
		"isbn": "978-3982016108",
		"title": "Writing A Compiler In Go",
		"year": 2018
	},
	{
		"first author": "Alan A. A. Donovan",
		"isbn": "978-0134190440",
		"title": "The Go Programming Language",
		"year": 2015
	},
	{
		"first author": "Andrew Williams",
		"isbn": "978-1789138412",
		"title": "Hands-On GUI Application Development in Go",
		"year": 2019
	},
	{
		"first author": "Andrew Williams",
		"isbn": 1800563167,
		"title": "Building Cross-Platform GUI Applications with Fyne",
		"year": 2021
	}
]

Select all books published after 2018:

$ rx -i csv -H '{book.title | book := input[_]; book.year > 2018}' < sample_data/books.csv
[
	"Building Cross-Platform GUI Applications with Fyne",
	"Hands-On GUI Application Development in Go"
]

Select all authors who have published at least 2 books:

$ rx '{author | book := input[_]; author := book.authors[_]; count({b | b := input[_]; author in b.authors}) > 1}' < sample_data/books.json
[
	"Andrew Williams",
	"Thorsten Ball"
]

As above, but with YAML output:

$ rx -o yaml '{author | book := input[_]; author := book.authors[_]; count({b | b := input[_]; author in b.authors}) > 1}' < sample_data/books.json
- Andrew Williams
- Thorsten Ball

Performance

Performance numbers use the jeopardy.json from this blog post. Each test was run 6 times, with the first run dropped and the remaining 5 runs averaged. jq-1.6 is used. The test system is running Arch Linux with kernel 5.17.4-zen1-1-zen on a Framework laptop with an i7-1165G7, 32GB memory, and a WD Black NVMe SSD with ZFS. Time is measured using the time command, and only the real time is considered. rx is build against go version go1.18.1 linux/amd64. The version of rx used for this test was level with b2e830f006fa4051c8c027ae41aea86a8af7745c. Output is written to disk, so this implies no colorization for either jq or rx.

Note that by default, the input file is parsed as YAML. I also show performance when using -i json which uses the standard library's JSON implementation, which is faster than parsing YAML. --ugly is used for some tests to disable pretty-printing output as well.

• Identity, read the input in JSON, write it to disk in JSON
  • jq: mean=1.757s, stdev=0.08641s
  • rx: mean=4.126s, stdev=0.06777s
  • rx --ugly: mean=3.821s, stdev=0.6064s
  • rx -i json --ugly: 1.927s, stdev=0.07547s
• Select only the subset of the question which has category HISTORY
  • jq '.[] | select(.category=="HISTORY")': mean=0.6974s, stdev=0.01293s
  • rx '{q | q := input[_]; q.category == "HISTORY"}': mean=3.327s, stdev=0.04233s
  • rx -i json --ugly '{q | q := input[_]; q.category == "HISTORY"}': mean=1.305s, stdev=0.02595s
• Select only the question field of those questions which have category HISTORy
  • jq '.[] | select(.category=="HISTORY") | .question': mean=0.7524s, stdev=0.1202s
  • rx '{q.question | q := input[_]; q.category == "HISTORY"}': mean=3.303s, stdev=0.02696s
  • rx -i json --ugly '{q.question | q := input[_]; q.category == "HISTORY"}': mean=1.305s, stdev=0.01077s

License

See ./LICENSE.

Community

Feel free to discuss or send patches via my public inbox.