starfig

Starfig is a programmatic configuration (config) generator. It helps create static configs using Starlark, a deterministic Python-like language.

Features

• 👍 Guranteed correctness. Catch invalid config updates by building and testing the source in CI.
• ♻️ Code reuse. Create shared schemas and configs.
• 🚨 Validation next door. Validations live with the schema definitions, written in the same language.
• 🤠 Good ol' Python. Sort of. Starfig uses Starlark, a Python-like language created by Google.
• 💥 Transpile to your language. Generate static configs into multiple languages. Coming soon.

Table of Contents

• Table of Contents
• Intro
  • Installation
  • What is it?
  • Use Cases
• Getting Started
  • Creating a Universe
  • Creating a Schema
  • Instantiating the schema
  • Generating the Configs
  • Validations
  • Sharing Configs
• API
  • Files
    • STARVERSE
    • .star File
    • STARFIG File
  • Target
  • load
  • Schema
  • Validations
  • Primitives
    • Bool
    • Float
    • Int
    • String
    • Object
    • List
• CLI
• Development

Intro

Installation

• macOS
  • brew install starfig It's in the process: https://github.com/Homebrew/homebrew-core/pull/102655
• Go
  • go install github.com/jathu/starfig@latest
• Linux
  • github.com/jathu/starfig/releases

What is it?

Starfig is a command line tool that helps build and test static configs using Starlark, a high level deterministic language. It helps you to create and share config schemas in large projects and organizations.

It currently generates a JSON output. Generating configs into different languages is coming soon.

load("//config/measurements.star", "Size")

def color_between_zero_and_one(value):
    if value < 0 or value > 1:
        return "Color must be between 0 and 1."

Star = Schema(
    fields = {
        "name": String(required = True),
        "spectral": String(validations = [
            spectral_validation
        ])
        "color": Float(validations = [
            color_between_zero_and_one
        ])
    }
)

Planet = Schema(
    fields = {
        "radius": Object(Size, required = True),
        "stars": List(Star)
    }
)

# ---------------------------------------------

sun = Star(
    name = "Sun",
    spectral = "G2V",
    color = 0.63,
)

earth = Planet(
    radius = Size(
        value = 6378,
        unit = "km"
    )
    stars = [sun]
)

mars = Planet(
    radius = Size(
        value = 3396,
        unit = "km"
    )
    stars = [sun]
)

{
  "//config:sun": {
    "name": "Sun",
    "spectral": "G2V",
    "color": 0.63
  },
  "//config:earth": {
    "radius": {
      "value": 6378,
      "unit": "km"
    },
    "stars": [
      {
        "name": "Sun",
        "spectral": "G2V",
        "color": 0.63
      }
    ]
  },
  "//config:mars": {
    "radius": {
      "value": 3396,
      "unit": "km"
    },
    "stars": [
      {
        "name": "Sun",
        "spectral": "G2V",
        "color": 0.63
      }
    ]
  }
}

Use Cases

Starfig can be used wherever you have configs — especially JSON/YAML configs:

• Use it for feature flagging or remote configs to ensure correctness and prevent regressions
• If you're an infra team
  • Infra configs (i.e. CloudFormation, Envoy, k8s) can be generated correctly and with custom constraints. It can be built and tested in CI before pushing to prod
  • Provide Starfig schemas to allow application teams to self-serve their own infra (example below)
• If your platform/tool (i.e. CLI tools) relies on users generting configs, providing them Starfig schemas can offload validation and correctness checks to Starfig

As an example:

• Starfig's version and changelog is built using Starfig:
  • Schema: tool/changelog/defs.star
  • Definition: tool/changelog/STARFIG
• This changlog config is also used as the source of version management during release: .github/workflows/populate_release.sh

⬆️ Back Up

Getting Started

Let's walkthrough Starfig by running through an example. Suppose we are an infra team and we want to help other teams create long-running jobs.

Before we start, learn a bit about Starlark. Starlark is the language we'll use in Starfig. Essentially, Starlark is Python without unbounded loops or nondeterminism. This helps us ensure deterministic builds that produce the same results everytime. If you know Python, you're good to go.

Let's go.

Creating a Universe

Starfig uses a universe to understand its workspace. A universe is defined as a folder that contains a STARVERSE file and all its subdirectories. The folder containing the STARVERSE file is the root of the universe.

Let's create a STARVERSE file in our root folder. For now, it will be empty.

~/bookface-corp $ touch STARVERSE

Creating a Schema

To help other teams create long-running jobs, we want to make it as simple and safe as possible. Let's expect users to provide a minimal set of configs to describe their long-running jobs.

Starfig allows schema definitions in .star files. This can be named anything and live anywhere in the universe.

~/bookface-corp $ touch infra/configs/jobs/defs.star

Within the defs.star file, we can define our schema using Starlark and Starfig provided methods.

# File: ~/bookface-corp/infra/configs/jobs/defs.star

Job = Schema(
  fields = {
    "name": String(required = True),

    # Time to allow the job to run.
    "time_to_live_minutes": Int(default = 60),

    # List of commands to run in the long-running jobs.
    "run_commands": List(String)
  }
)

We defined our job as a simple list of commands with a name and time to live (max runtime).

Instantiating the schema

Teams can now start creating a Job anywhere in the universe. Starfig schemas are instantiated in a STARFIG file. This file can live anywhere in the universe.

Suppose the Growth team creates a job to send emails out:

~/bookface-corp $ touch growth/jobs/STARFIG

# File: ~/bookface-corp/growth/jobs/STARFIG

load("//infra/configs/jobs/defs.star", "Job")

def days_to_minutes(days):
  return 60 * 24 * days

email_sender = Job(
  name = "weekly_email_sender",
  # Let's let this run for a week
  time_to_live_minutes = days_to_minutes(7),
  run_commands = [
    "./send_emails.sh",
    "./log_email_results.sh"
  ]
)

We see a lot going on here:

• We can load schemas from anywhere in the universe. Since it's a depedency, any breaking changes to Job will cause this config to fail during build
• Since this is Starlark, a subset of Python, we can do things like creating functions and computing the time_to_live_minutes

Generating the Configs

Let's look at the current state of our universe:

~/bookface-corp $ ls .

.
├── STARVERSE
├── ...
├── infra
│   └── configs
│       └── jobs
│           └── defs.star
├── ...
├── growth
│   └── jobs
│       ├── ...
│       └── STARFIG
└── ...

Starfig can now generate the configs.

~/bookface-corp $ starfig build //...
{
  "//growth/jobs:email_sender": {
    "name": "weekly_email_sender",
    "time_to_live_minutes": 10080,
    "run_commands": [
      "./send_emails.sh",
      "./log_email_results.sh"
    ]
  }
}

Alternatively, we can also build specific configs: starfig build //growth/jobs:email_sender.

Validations

A job that runs for a week seems unreasonable and unreliable. Let's set the upperbound to a day. We can enforce this by adding a validation to our Job schema:

# File: ~/bookface-corp/infra/configs/jobs/defs.star

def limit_ttl(user_set_ttl):
  if user_set_ttl > (24 * 60):
    return "A job cannot run longer than a day."

Job = Schema(
  fields = {
    "name": String(required = True),
    "time_to_live_minutes": Int(
      default = 60,
      validations = [limit_ttl]
    ),
    "run_commands": List(String)
  }
)

Now any instances of a Job with a time_to_live_minutes longer than a day will fail to build.

~/bookface-corp $ starfig build //...

12:34:56.789 error ~/bookface-corp/growth/jobs/STARFIG:6: Invalid field time_to_live_minutes in Job: "A job cannot run longer than a day."

Sharing Configs

Let's allow jobs to be run in specific regions — regions we, as an infra team, will provide. Start by providing a list of existing regions:

# File: ~/bookface-corp/infra/configs/region/defs.star

Region = Schema(
  fields = {
    "country": String(required = True),
    "zone": String(required = True)
  }
)

# File: ~/bookface-corp/infra/configs/region/STARFIG

load("//infra/configs/region/defs.star", "Region")

us_east_1 = Region(country = "US", zone = "Northern Virginia")

us_east_2 = Region(country = "US", zone = "Ohio")

us_west_1 = Region(country = "US", zone = "Northern California")

us_west_2 = Region(country = "US", zone = "Oregon")

Update Job to accept a list of regions:

# File: ~/bookface-corp/infra/configs/jobs/defs.star

load("//infra/configs/region/defs.star", "Region")

def limit_ttl(user_set_ttl):
  if user_set_ttl > (24 * 60):
    return "A job cannot run longer than a day."

Job = Schema(
  fields = {
    "name": String(required = True),
    "time_to_live_minutes": Int(
      default = 60,
      validations = [limit_ttl]
    ),
    "run_commands": List(String),

    # Regions your jobs will be run in.
    "regions": List(Region)
  }
)

We can now update our email_sender to run on some specifc regions:

# File: ~/bookface-corp/growth/jobs/STARFIG

load("//infra/configs/jobs/defs.star", "Job")
load("//infra/configs/region/STARFIG", "us_east_1", "us_west_1")

def days_to_minutes(days):
  return 60 * 24 * days

email_sender = Job(
  name = "weekly_email_sender",
  time_to_live_minutes = days_to_minutes(1),
  run_commands = [
    "./send_emails.sh",
    "./log_email_results.sh"
  ],
  regions = [us_east_1, us_west_1]
)

Building this should now include the regions:

~/bookface-corp $ starfig build //growth/jobs:email_sender
{
  "//growth/jobs:email_sender": {
    "name": "weekly_email_sender",
    "time_to_live_minutes": 10080,
    "run_commands": [
      "./send_emails.sh",
      "./log_email_results.sh"
    ],
    "regions": [
      {
        "country": "US",
        "zone": "Northern Virginia"
      },
      {
        "country": "US",
        "zone": "Northern California"
      }
    ]
  }
}

⬆️ Back Up

API

Since Starfig uses Starlark, the runtime includes everything in it: github.com/bazelbuild/starlark. In addition to Starfig provided libs.

Files

Starfig cares about three types of files.

STARVERSE

Starfig uses a universe to understand its workspace. A universe is defined as a folder that contains a STARVERSE file and all its subdirectories. The folder containing the STARVERSE file is the root of the universe.

.star File

• .star files (i.e. defs.star, jobs.star etc.) allow schema, function and constant definitions
• .star files can be imported into other .star files to define schemas or imported into STARFIG files to instantiate schemas
• A package can have an arbitrary number of .star files
  • Variables or functions starting with an underscore (_) are implicitly private and will not be exported

STARFIG File

• A STARFIG file only contains schema instantiations — the actual configs
  • It can also contain functions, however they will not be exported
• STARFIG files can be only imported into other STARFIG files to share configs
• Implicit in the name, a package can only have a single STARFIG file — which can contain multiple targets within

Target

A target is the name of an object in the Starfig universe. All targets must start with //, which indicates the root of the universe. Composed as:

          Target
│-------------------------│

//some/package/path:example

│ │---------------│ │-----│
│ │                 │   
│ └ Package         └ Target name 
│
└ Root of the starfig universe

• File Target
  • A file target is a path to a starfig file. i.e. //some/path/def.star, //some/other/path/STARFIG
  • Package: the relative path of the directory from the root to the file
  • Target name: the name of the .star or STARFIG file in the package
• Build Target
  • A build target is the name of a config within the Starfig universe
  • Package: the relative path of the directory from the root to the STARFIG file
  • Target name: the name of the variables created by schema instantiations in the STARFIG file

load

The load function allows importing dependencies.

load("//example/config/defs.star", "Example", "Sample", "Another")

• The first argument of the load function is a file target
  • .star files can be loaded into other .star files or STARFIG files
  • STARFIG files can only be loaded into other STARFIG files
• The remaining arguments are the dependencies being loaded from the file target
• Variables or functions starting with an underscore, _, are implicitly private and will not be importable

Schema

Schema is a function that helps define a schema. A schema is a structure of grouped fields — it's a way to define a custom type.

# Example

Fruit = Schema(
  fields = {
    "name": String(),
  },
  validations = []
)

Validations

Validations are custome user defined functions that validate a schema instantiation during build time. A validation error is thrown if the validation functions returns anything but None.

# This validation is run on just the name field.
def name_requirement(fruit_name):
  if len(fruit_name) < 2:
    return "A fruit name must have 2 or more characters."
  else:
    return None

# This validation is run on the whole schema instantiation. 
# This allows cross field validations.
def name_color_requirement(fruit):
  if fruit["color"] == "red" and "sweeet" not in fruit["name"]:
    return "Red fruit names must contain sweeet."
  else:
    return None

Fruit = Schema(
  fields = {
    "name": String(validations = [name_requirement])
    "color": String()
  },
  validations = [name_color_requirement]
)

Primitives

Bool

# Example

Fruit = Schema(
  fields = {
    "is_sweet": Bool(default = True, required = True, validations = []),
  }
)

Float

# Example

Fruit = Schema(
  fields = {
    "weight": Float(default = 0, required = True, validations = []),
  }
)

Int

# Example

Fruit = Schema(
  fields = {
    "cost": Int(default = 1, required = True, validations = []),
  }
)

String

# Example

Fruit = Schema(
  fields = {
    "name": String(default = "", required = True, validations = []),
  }
)

Object

An Object is a special function that allows fields to expect other schema types.

# Example

load("//example/defs.star", "Country")

Fruit = Schema(
  fields = {
    "country": Object(Country, required = True, validations = [])
  }
)

List

A List is a special function that allows fields to expect a list of other schema types.

# Example

load("//example/defs.star", "Country")

Fruit = Schema(
  fields = {
    "country": List(Country, validations = [])
  }
)

⬆️ Back Up

CLI

Run starfig --help to learn more.

⬆️ Back Up

Development

Starfig is built using Go and its standard built/test commands. For convenience, Starfig also has fx commands — run fx list to see the commands. jq is required for some fx commands.

May 2022 — San Francisco