README
¶
Azathoth
A pipeline manager on Kubernetes.
Purpose
This project aims to provide a simple jobs deployment tool based on the Kubernetes environment.
It is a way for me to learn about Kubernetes, so it is very experimental.
Pipelines
A pipelines is a Go programs that defines a graph of dependency of jobs passed to a Scheduler. A job can be any Docker image.
Currently, a pipeline is built in a Docker image and deployed on a regular basis through a Kubernetes CronJob.
When it starts, it performs a DFS through the jobs dependencies graph. It schedules only jobs whose have no dependencies or that have dependencies already executed. It also starts a server listening to events sent by jobs pods.
When starting a job, it creates a Kubernetes Job resource. It attaches to it lifecycle hooks that send requests to its server at start and stop. This warns the Scheduler about a Job state change. When it receives such an event, the Scheduler uses the Kubernetes API to get more info about the corresponding Job and then update its status.
Example
Requirements
- a Kubernetes cluster (I used minikube for my tests)
- Docker
Build
First of all, make sure to target your cluster's Docker registry:
eval $(minikube docker-env)
Then build all the jobs Docker images with:
make build
Deploy
Deploy the pipelines CronJobs with:
make deploy
make clean
deletes the CronJobs objects.
Directories