rq

Rego Query

Early Stage Project: this project is still very much experimental. Expect things to change and break without notice.

rq (Rego Query) is inspired by jq, and aims to offer a similar set of capabilities oriented around running Rego queries. In the long run, rq is intended to make it as easy as possible to utilize Rego both interactively as a shell command, and within shell scripts. Though it is already possible to use Rego in this way with via opa eval using the official opa command, rq is intended to be easier and simpler to use specifically in a shell context, and has additional convenience features to support this use case, such as more input and output formats, syntax highlighted output, and so on.

A simple demonstration (see the Showcase section for more):

$ data='{"foo": 7, "bar": 3, "baz": 12, "quux": 4, "spam": 6}'
$ # find all keys in data where the value is at least 5
$ echo "$data" | rq '{k | some k; input[k] > 5}'
[
	"baz",
	"foo",
	"spam"
]

A short (10 minute) talk providing a brief overview of rq: rq: Datalog for your shell pipelines.

Features

• Supported input formats
  • awk (field and record splitting similar to awk's defaults)
  • CSV (including comments, arbitrary delimiters, headers, and leading line skip)
  • dotenv (shell-like variables, via gotenv)
  • Dhall
  • HCL
  • HJSON
  • INI
  • JSON
  • NDJSON
  • lines (input is split into non-empty lines)
  • raw (plain UTF-8 string)
  • tabular (input is parsed as a table with header and type inference, uses field and record splitting similar to the awk format)
  • TOML
  • XML
  • YAML
• Supported output formats
  • JSON
  • NDJSON
  • null (discards output data without formatting it)
  • YAML
  • CSV
  • raw (primitive types, and lists of primitive types, are printed one per line without quotes, otherwise behaves like the JSON output format)
  • md-table (markdown compatible tables)
  • XML
  • dotenv (POSIX shell variable assignments, suitable for use with eval)
  • sh (alias for dotenv)
  • template (output is used as the data for a Go string template, see -t / --template)
  • TOML
• Syntax-highlighted output using chroma
• Larger programs can be written as scripts, with shebang support via rq script
• Additional builtins to augment Rego's standard library, many with useful side-effects
  • rq.run() - run arbitrary shell commands
  • rq.tree() - recursively search the local filesystem
  • rq.env() - access environment variables
  • rq.args() - access CLI arguments (only for rq script)
  • rq.error() - crash the Rego evaluator with an error
  • rq.encode() - serialize a Rego object to a string using one of rq's output formats
  • rq.decode() - deserialize a string to a Rego object using one of rq's input formats
  • rq.write() - write a Rego object to disk using one of rq's output formats
  • rq.read() - read a Rego object from disk using one of rq's input formats
  • rq.parsedate() - parse a date string without needing to explicitly provide a layout
  • rq.version() - access rq version information
  • rq.convert() - performs unit conversions
  • rq.abs() - convert a path to absolute form
  • rq.base() - extract the final element from a path
  • rq.ext() - extract the file extension from a path
  • rq.dir() - extract all but the final element of a path
  • rq.splitpath() - split a path to individual elements
  • rq.joinpath() - join path elements
  • rq.getwd() - retrieve rq's working directory
  • rq.chdir() - change rq's working directory
  • rq.scriptpath() - retrieve the path to the running script (only for rq script)
  • rq.template() - template a string with data from a Rego object
  • rq.fake() - generate fake data using jaswdr/faker
  • rq.sfake() - similar to rq.fake(), but allows referencing previously generated data symbolically

Installation Instructions

With go install:

$ go install git.sr.ht/~charles/rq/cmd/rq@latest

With Make:

$ git clone https://git.sr.ht/~charles/rq
$ cd rq
$ git checkout <tag name>  # skip if you want to use the latest development version
$ make install

With asdf (note: you must have a Go compiler installed):

$ asdf plugin add rq https://git.sr.ht/~charles/asdf-rq
$ asdf install rq latest
$ asdf global rq latest

On Linux or MacOS, you can also install the latest binary version to /usr/bin with:

$ curl -LSsf 'https://git.sr.ht/~charles/rq/blob/master/scripts/install.sh' | sudo bash

The installation script supports overriding the OS, version, platform, and installation directory via environment variables. These are documented in script/install.sh.

Usage Instructions

See rq --help. You might also like the getting started guide.

Showcase

For more rq usage examples, check out the cookbook.

Sample data:

$ cat sample_data/books.json
[
    {
        "title": "Writing An Interpreter In Go",
        "authors": ["Thorsten Ball"],
        "isbn": "978-3982016115",
        "year": 2018
    },
    {
        "title": "Writing A Compiler In Go",
        "authors": ["Thorsten Ball"],
        "isbn": "978-3982016108",
        "year": 2018
    },
    {
        "title": "The Go Programming Language",
        "authors": ["Alan A. A. Donovan", "Brian W. Kernighan"],
        "isbn": "978-0134190440",
        "year": 2015
    },
    {
        "title": "Hands-On GUI Application Development in Go",
        "authors": ["Andrew Williams"],
        "isbn": "978-1789138412",
        "year": 2019
    },
    {
        "title": "Building Cross-Platform GUI Applications with Fyne",
        "authors": ["Andrew Williams"],
        "isbn": "1800563167",
        "year": 2021
    }
]
$ cat sample_data/books.csv
title,"first author",isbn,year
Writing An Interpreter In Go,Thorsten Ball,978-3982016115,2018
Writing A Compiler In Go,Thorsten Ball,978-3982016108,2018
"The Go Programming Language","Alan A. A. Donovan",978-0134190440,2015
Hands-On GUI Application Development in Go,Andrew Williams,978-1789138412,2019
Building Cross-Platform GUI Applications with Fyne,Andrew Williams,1800563167,2021

Read JSON data:

$ rq < sample_data/books.json
[
	{
		"authors": [
			"Thorsten Ball"
		],
		"isbn": "978-3982016115",
		"title": "Writing An Interpreter In Go",
		"year": 2018
	},
	{
		"authors": [
			"Thorsten Ball"
		],
		"isbn": "978-3982016108",
		"title": "Writing A Compiler In Go",
		"year": 2018
	},
	{
		"authors": [
			"Alan A. A. Donovan",
			"Brian W. Kernighan"
		],
		"isbn": "978-0134190440",
		"title": "The Go Programming Language",
		"year": 2015
	},
	{
		"authors": [
			"Andrew Williams"
		],
		"isbn": "978-1789138412",
		"title": "Hands-On GUI Application Development in Go",
		"year": 2019
	},
	{
		"authors": [
			"Andrew Williams"
		],
		"isbn": "1800563167",
		"title": "Building Cross-Platform GUI Applications with Fyne",
		"year": 2021
	}
]

Read CSV data, with headers:

$ rq -i csv -H < sample_data/books.csv
[
	{
		"first author": "Thorsten Ball",
		"isbn": "978-3982016115",
		"title": "Writing An Interpreter In Go",
		"year": 2018
	},
	{
		"first author": "Thorsten Ball",
		"isbn": "978-3982016108",
		"title": "Writing A Compiler In Go",
		"year": 2018
	},
	{
		"first author": "Alan A. A. Donovan",
		"isbn": "978-0134190440",
		"title": "The Go Programming Language",
		"year": 2015
	},
	{
		"first author": "Andrew Williams",
		"isbn": "978-1789138412",
		"title": "Hands-On GUI Application Development in Go",
		"year": 2019
	},
	{
		"first author": "Andrew Williams",
		"isbn": 1800563167,
		"title": "Building Cross-Platform GUI Applications with Fyne",
		"year": 2021
	}
]

Select all books published after 2018:

$ rq -i csv -H '{book.title | book := input[_]; book.year > 2018}' < sample_data/books.csv
[
	"Building Cross-Platform GUI Applications with Fyne",
	"Hands-On GUI Application Development in Go"
]

As above, but with CSV output:

authors.0,isbn,title,year
Andrew Williams,1800563167,Building Cross-Platform GUI Applications with Fyne,2021
Andrew Williams,978-1789138412,Hands-On GUI Application Development in Go,2019

Select all authors who have published at least 2 books:

$ rq '{author | book := input[_]; author := book.authors[_]; count({b | b := input[_]; author in b.authors}) > 1}' < sample_data/books.json
[
	"Andrew Williams",
	"Thorsten Ball"
]

As above, but with YAML output:

$ rq -o yaml '{author | book := input[_]; author := book.authors[_]; count({b | b := input[_]; author in b.authors}) > 1}' < sample_data/books.json
- Andrew Williams
- Thorsten Ball

As above, but with "raw" output:

$ rq -o raw '{author | book := input[_]; author := book.authors[_]; count({b | b := input[_]; author in b.authors}) > 1}' < sample_data/books.json
Andrew Williams
Thorsten Ball

Run a shell command and parse the output as JSON:

$ rq 'rq.run(["jq", ".[0]"], {"stdin_spec": {"file_path": "sample_data/books.json"}, "stdout_spec": "json:"})'
{
	"exitcode": 0,
	"stderr": "",
	"stdout": {
		"authors": [
			"Thorsten Ball"
		],
		"isbn": "978-3982016115",
		"title": "Writing An Interpreter In Go",
		"year": 2018
	}
}

Generate fake data for unit tests:

$ rq '{rq.fake("internet.user"): {"name": rq.fake("name"), "email": rq.fake("email")} | i := numbers.range(1, 5)[_]}'
{
	"gulgowski.russel": {
		"email": "pauline@example.org",
		"name": "Carmel Franecki"
	},
	"jacquelyn": {
		"email": "arden.stamm@example.org",
		"name": "Olga Toy V"
	},
	"marcus.larkin": {
		"email": "murazik@example.org",
		"name": "Aniya Graham Jr."
	},
	"rogahn": {
		"email": "cole.magdalen@example.org",
		"name": "Imani Gislason"
	},
	"swaniawski.torey": {
		"email": "abshire@example.org",
		"name": "Ms. Halie Herzog"
	}
}

License

See ./LICENSE.

Changelog

See ./CHANGELOG.md.

Documentation

See ./doc/README.md.

Community & Contribution

Feel free to discuss or send patches via the rq mailing list.

If you have found an issue with rq, please report it via the rq issue tracker.

For instructions on how to properly prepare patches for use with SourceHut projects such as rq, see the git.sr.ht docs.

Supporting rq

If you would like to support the project financially, donations are accepted via buy me a coffee. Your support is greatly appreciated.

Disclaimer

I am a Styra (the creators of OPA) employee. This project is not endorsed, affiliated with, or supported by Styra or the OPA project. You have no expectation of support from Styra or the OPA team if you use this software.