gh-actions-demo

How to Create GitOps Pipelines with GitHub Actions

GitHub Actions will become a major player on the CI SaaS market. It can easily replace most CI tools out there especially if you ship code as container images. With GitHub Actions you can do more than CI/CD. Most tasks performed today with bots (code sign validations, issue management, notifications, etc) can be made into workflows and run solely by GitHub.

Why would you give up your current CI SaaS and self hosted bots for GitHub Actions? For one, GitHub Actions simplifies automation tasks by offering a serverless platform that is capable of handling most development tasks. As a developer you don't want to jump from one SaaS to another in order to diagnose a build error. The fewer environments you have to use on a regular basis, the more productive you'll be. Not to mention that as a developer you probably spend most of your time on GitHub anyway.

Should you use GitHub Action to deploy workloads on Kubernetes? I think continuous deployment should not be part of the CI workflow, instead CD could be performed by a Kubernetes operator that implements a control loop that continuously applies the desired state to your cluster, offering protection against harmful actions like deployments deletion or network policies altering. You can read more about why we consider CiOps a Kubernetes anti-pattern here.

The anatomy of a GitHub Action

In many ways GitHub Actions are similar to FaaS, and like serverless functions, a GitHub Action can be triggered by an event. Multiple actions can be chained together to create a workflow that defines how you want to react to that particular event.

Most FaaS solutions made for Kubernetes let you package a function as a container image. A Github Action is no more than that with a piece of code packaged as a container image that GitHub runs for you.

In order to make a GitHub Action all you need to do is to create a Dockerfile. Here is an example of a GitHub Action that runs unit tests for a Golang project:

FROM golang:1.10

LABEL "name"="go test action"
LABEL "maintainer"="Stefan Prodan <support@weave.works>"
LABEL "version"="1.0.0"

LABEL "com.github.actions.icon"="code"
LABEL "com.github.actions.color"="green-dark"
LABEL "com.github.actions.name"="gotest"
LABEL "com.github.actions.description"="This is an action to run go test."

COPY entrypoint.sh /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]

And the container entry point script that runs the tests:

#!/usr/bin/env bash

set -e +o pipefail

APP_DIR="/go/src/github.com/${GITHUB_REPOSITORY}/"

mkdir -p ${APP_DIR} && cp -r ./ ${APP_DIR} && cd ${APP_DIR}

go test $(go list ./... | grep -v /vendor/) -race -coverprofile=coverage.txt -covermode=atomic
mv coverage.txt ${GITHUB_WORKSPACE}/coverage.txt

If the tests pass, the coverage report is copied to the GitHub workspace. This workspace is shared between all the actions part of the same workflow. This means that you can use the coverage report generated by the go test action in another action and then publish that report to Codecov.

This is how you can define a workflow that will publish the test coverage after a git push:

workflow "Publish test coverage" {
  on = "push"
  resolves = ["Publish coverage"]
}

action "Run tests" {
  uses = "./.github/actions/gotest"
}

action "Publish coverage" {
  needs = ["Run tests"]
  uses = "./.github/actions/codecov"
  args = "-f ${GITHUB_WORKSPACE}/coverage.txt"
}

The above workflow uses actions defined in the same repository as the app code. However, you could also define GitHub Actions in a dedicated repo. For example instead of ./.github/actions/gotest it could be stefanprodan/gh-actions/gotest@master or even a container image hosted on Docker Hub docker://stefanprodan/gotest:latest.

Running each workflow step in a container is not a novelty. Jenkins docker pipelines, Drone, GCP builder and other tools are doing the same thing, but what I like about GitHub Actions is that you don't need to build and publish the action container image to a registry. Github lets you reference a git repo where your action Dockerfile is and it will then build the container image before running the workflow.

Building a GitOps pipeline for Kubernetes

GitOps has a different approach on how you define and deploy your workloads.

In the GitOps model, the state of your Kubernetes cluster(s) is kept in a dedicated git repo (I will refer to this as the config repository). This means the app deployments, Helm releases, network polices and any other Kubernetes custom resources are managed from a single repo that defines your cluster's desired state.

If you look at the GKE CI/CD example created by GitHub, the deployment file is in the same repo as the app code and the Docker image tag is injected from an environment variable. This works fine until your cluster melts down or you need to rollback a deployment to a previous version. In the case of a major incident, you will have to rerun the last deployment workflow (in every app repo), but since you will be building and deploying new images, it means that you won't end up with exactly the same state as before, .

GitOps solves this problem by reapplying the config repo every time the cluster diverges from the state defined in git. For this to work you'll need a GitOps operator that runs in your cluster and a container registry where GitHub Actions are publishing immutable images (no latest tags, use semantic versioning or git sha).

I will use gh-actions-demo to demonstrate a GitOps pipeline that includes promoting releases between environments.

GitHub Workflows for a Golang App

This is what the GitHub workflow looks like for a Golang app:

When commits are pushed to the master branch, the GitHub workflow produces a container image as in repo/app:branch-commitsha. When you do a GitHub release, the build action tags the container image as repo/app:git-tag.

The GitHub Actions used in this workflow can be found here.

workflow "Publish container" {
  on = "push"
  resolves = ["Push"]
}

action "Lint" {
  uses = "./.github/actions/golang"
  args = "fmt"
}

action "Test" {
  needs = ["Lint"]
  uses = "./.github/actions/golang"
  args = "test"
}

action "Build" {
  needs = ["Test"]
  uses = "./.github/actions/docker"
  secrets = ["DOCKER_IMAGE"]
  args = ["build", "Dockerfile"]
}

action "Login" {
  needs = ["Build"]
  uses = "actions/docker/login@master"
  secrets = ["DOCKER_USERNAME", "DOCKER_PASSWORD"]
}

action "Push" {
  needs = ["Login"]
  uses = "./.github/actions/docker"
  secrets = ["DOCKER_IMAGE"]
  args = "push"
}

If you have access to GitHub Actions, use this to bootstrap a private repository with the above workflow.

Structuring your config repository

Now that the code repository workflow produces immutable container images, let's create a config repository with the following structure:

├── namespaces
│   ├── production.yaml
│   └── staging.yaml
└── workloads
    ├── production
    │   ├── deployment.yaml
    │   └── service.yaml
    └── staging
        ├── deployment.yaml
        └── service.yaml

You can find the repository at gh-actions-demo-cluster.

The cluster repo contains two namespaces and the deployment YAMLs for my demo app. The staging deployment is for when I push to the master branch and the production one is for when I do a GitHub release.

Connecting the Weave Cloud Agents

In order to create these objects in my cluster, I'll install the Weave Cloud agents and connect the GitOps operator to my cluster repo.

When Weave Cloud Deploy runs for the first time, it applies all of the YAMLs it finds inside the repo: namespaces first, then the services and finally the deployments.

Automating and rolling back deployments

Weave Cloud Deploy pulls and applies any changes made to the repository every five minutes. To trigger a deploy on every commit I can setup a GitHub web hook from the Weave Cloud UI:

Now that my workloads are running in both namespaces, I want to automate the deployment so that every time I push to the master branch, the resulting image runs on staging and every time I do a GitHub release the production workload is updated.

Weave Cloud Deploy allows you to define automated deployment polices based on container image tags. These policies can be specified using annotations in the deployment manifest YAML file.

In order to automate the master branch deployment, I can edit the staging/deployment.yaml and create a glog filter:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: podinfo
  namespace: staging
  annotations:
    flux.weave.works/tag.podinfod: glog:master-*
    flux.weave.works/automated: 'true'
spec:
  template:
    spec:
      containers:
      - name: podinfod
        image: stefanprodan/podinfo:master-a9a1252

For GitHub releases, I can create a semantic version filter and instruct Weave Cloud Deploy to update my production deployment on every patch release:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: podinfo
  namespace: production
  annotations:
    flux.weave.works/tag.podinfod: semver:~1.3
    flux.weave.works/automated: 'true'
spec:
  template:
    spec:
      containers:
      - name: podinfod
        image: stefanprodan/podinfo:1.3.0

If I push commits to the demo app master branch, GitHub actions publish an image as stefanprodan/podinfo:master-48761af, Weave Cloud then updates the cluster repo and deploys that image to the staging namespace.

If I make a new release by pushing a tag to my demo app repo with git tag 1.3.2 && git push origin 1.3.2 the GitHub actions will test, build and push a container image as in stefanprodan/podinfo:1.3.2 to the registry. Weave Cloud Deploy will then fetch the new image tag from Docker Hub, update the production deployment in the cluster repo and apply the new deployment spec.

If I want to rollback this deployment, I can either undo the git commit or I can use Weave Cloud and do a manual release of a previous version:

Weave Cloud will then commit the new image tag to git and report the deployment progress in the UI:

Conclusions

Once GitHub Actions are generally available, I expect to see an explosion of custom actions that range from simple linting tools to security scanners and bots. With a polished UI and the compute power that Azure offers, chances are GitHub Actions will become the go-to platform for CI automation.

Getting Help

If you have any questions about GitOps or Weave Cloud:

• Invite yourself to the Weave community slack.
• Ask a question on the #flux slack channel.
• Join the Weave User Group and get invited to online talks, hands-on training and meetups in your area.
• Send an email to weave-users@weave.works