prowgen

Istio Prow Job Config Generator

prowgen is a tool that can be used to generate a set of Prow job configuration files from the templated meta config files. It can help simplify the complex Prow job configuration files by allowing users to provide opinionated defaults and shared config fields.

Config Reference

prowgen simplifies the configuration by allowing users to define multiple layers of configs, with each overlays on the other. Currently it supports defining 3 layers:

• BaseConfig contains common fields that are shared by all meta config files under the same folder.
• JobsConfig contains fields that can be defined in a meta config file, and it can contain multiple Jobs.
• Job is the last layer for defining the actual Prow jobs.

All the supported fields for these structs can be checked from spec.go, but below are the concrete examples on how to configure them in yaml format.

Base config

In the root folder, a .base.yaml file can be added to define the config fields that are shared by all the meta config files.

Below is an example base config file:

# The header line that will be added to each generated config file.
autogen_header: "# THIS FILE IS AUTOGENERATED. See prow/config/README.md\n"

# A map of org:alias.
# Jobs configured with the org in this map will have its `path_alias` field.
path_aliases:
  istio: istio.io

# Cluster and node pool to schedule the Prow job pods.
cluster: istio-build
node_selector:
  testing: test-pool

# The GCS bucket to upload the logs and artifacts.
gcs_log_bucket: istio-testing

# Testgrid config for all the jobs.
# Note num_failures_to_alert will only be set for postsubmit and periodic jobs.
testgrid_config:
  enabled: true
  alert_email: istio-oncall@googlegroups.com
  num_failures_to_alert: "1"

# A map of preset resource allocations that can be referenced in each meta config file.
resources_presets:
  default:
    limits:
      cpu: 3000m
      memory: 24Gi
    requests:
      cpu: 1000m
      memory: 3Gi

# The default dependencies for all the jobs.
requirements: [cache]
# A map of dependency presets that can be referenced in each meta config file.
requirement_presets:
  kind:
    volumeMounts:
    - mountPath: /lib/modules
      name: modules
      readOnly: true
    - mountPath: /sys/fs/cgroup
      name: cgroup
      readOnly: true
    - mountPath: /var/lib/docker
      name: docker-root
    volumes:
    - hostPath:
        path: /lib/modules
        type: Directory
      name: modules
    - hostPath:
        path: /sys/fs/cgroup
        type: Directory
      name: cgroup
    - emptyDir: {}
      name: docker-root
  cache:
    volumeMounts:
    - mountPath: /home/prow/go/pkg
      name: build-cache
      subPath: gomod
    volumes:
    - hostPath:
        path: /var/tmp/prow/cache
        type: DirectoryOrCreate
      name: build-cache
  gcp:
    labels:
      preset-service-account: "true"
  commonargs:
    args:
    - "--up"
    - "--down"
    - "--test"

In each sub-folder, a .base.yaml file can also be added which'll overlay the config fields in the root .base.yaml. Please note for now the overlay logic is not recursive, which means only the .base.yaml file in the current folder and the root folder will be overlaid, any other .base.yaml files in folders between them will be ignored.

Job Syntax

Any number of yaml files can be added to the root and subfolder(s) to configure the Prow jobs for each org/repo:branches.

Before is an example, annotated job config file:

# REQUIRED. Defines what org these jobs should run for
org: istio
# REQUIRED. Defines what repo these jobs should run for
repo: istio

# Defines what branches to run these jobs for. Multiple can be provided
# The branch name will be appended to the job name (e.g tests -> tests-master)
# If this is not supplied, it defaults to master
branches:
  - master

# REQUIRED. Defines the image that will be used to run the jobs
image: gcr.io/istio-testing/build-tools:master

# The policy and secrets for pulling the image.
image_pull_policy: Always
image_pull_secrets: ["gcr-secret"]

# The interval to schedule the periodic Prow jobs.
interval: 5h
# cron can also be used to schedule the periodic Prow jobs.
# interval and cron cannot be specified together.

# Determines whether this configuration can be automatically cloned to create a release branch
# version. Only used for Istio to generate meta config files for the new release branch.
supports_release_branching: false

# A matrix can contain arbitrary number of dimensions, and can be used to easily define a combination of Prow jobs.
# Each dimension will only be respected for computation if they are referenced in the Prow job config, and the syntax
# to use the dimension is $(matrix.dimension_name)
matrix:
  greet: [hey, hello, hi]
  name: [foo, bar]

# Defines the actual jobs
jobs:
  # A basic test requires just a name and a command to run
  - name: unit-tests
    command: [make, test]
  - name: integration-tests
    # types defines when the job will run. Valid options are [presubmit, postsubmit, periodic].
    # by default a presubmit and postsubmit job will be created with the same config
    types: [postsubmit, periodic]
    # resources determines what resource requests and limits to use.
    # It can be one of the preset resource allocations defined in the global config and file config.
    # If omitted, default will be used if it is provided.
    resources: large
    # timeout is how long the prow job will be kept before being aborted.
    timeout: 10h
    command: [prow/istio-lint.sh]
    # requirements specify what dependencies a test has.
    # The options must be the preset requirement names specified in the requirement_presets field in the global config and file config.
    requirements: [gcp, commonargs]
    # excluded_requirements specify what dependencies a test should not have.
    # The options must be the preset requirement names specified in the requirement_presets field in the global config and file config.
    excluded_requirements: [cache]
  - name: hello-world
    command: [echo, "hello world"]
    # modifiers change various parts of the test config. See the values below
    modifiers:
    - presubmit_skipped # if set, the test will only be run in presubmit by explicitly calling /test on it
    - presubmit_optional # if set, the test will not be required in presubmit
    - hidden # if set, the test will run but not be reported to the GitHub UI
  - name: $(matrix.greet)-$(matrix.name)
    # Prow jobs will be generated based on the combinations of each dimension.
    # In this case 3*2=6 Prow jobs will be generated.
    command: [echo, "${matrix.greet} $(matrix.name)"]

# Defines preset resource allocations for tests
# The map here will be intersected with the map in the global config (if there is),
# and overwrite the value if the names are duplicated.
resources_presets:
  default:
    requests:
      memory: "3Gi"
      cpu: "3000m"
    limits:
      memory: "24Gi"
      cpu: "3000m"
  # Define another preset, "large", with higher allocations
  large:
    requests:
      memory: "16Gi"
      cpu: "3000m"
    limits:
      memory: "24Gi"
      cpu: "3000m"
# Defines preset dependencies for tests
# The map here will be intersected with the map in the base config (if there is),
# and overwrite the value if the names are duplicated.
requirement_presets:
  github:
    volumeMounts:
    - mountPath: /etc/github-token
      name: github
      readOnly: true
    volumes:
    - name: github
      secret:
        secretName: oauth-token

More of the examples can be checked from testdata and Istio Prow jobs.

How to use the tool

make command

Istio Prow jobs config files can be generated with:

make generate-config

go run command

You can also run the command directly, which provides more options:

cd prow/config/cmd
go run generate.go \
  --input-dir=/path/to/meta/config --output-dir=/path/to/generated/config \
  [print|write|check|branch]

• print will print out all generated config to stdout
• write will write out generated config to the appropriate job file
• check will strictly compare the generated config to the current config, and fail if there are any differences. This is useful for a CI gate to ensure config is up to date
• branch will create new job configurations for a new release branch. Invoke with a release name (e.g. "1.4"). Currently only usable for the Istio project.

docker run command

The prowgen tool has been automatically published as a Docker image at gcr.io/istio-testing/prowgen:latest, so you can also run it with docker run command if you don't want to install the binary on your local machine:

export INPUT="/path/to/meta/config"
export OUTPUT="/path/to/generated/config"
docker run --mount type=bind,source=${INPUT},target=${INPUT} --mount type=bind,source=${OUTPUT},target=${OUTPUT} \
  gcr.io/istio-testing/prowgen:latest \
  prowgen --input-dir=${INPUT} --output-dir=${OUTPUT} write

Use it as a library

Since all the core structs and functions for the prowgen tool are public, you can also import and use it as a library. For an example, check how Knative configgen is implemented.

Pre/Post process command

The --pre-process-command and --post-process-command flag may be used to execute a command before and after the config files are generated, in case the users need customized config generation logic that cannot be supported by prowgen. The binary will be run with the following environment variables set:

INPUT        # Path of the meta config files folder
OUTPUT       # Path of the generated config files folder