README
¶
= Directed Acyclic Graph Build System
image:https://pkg.go.dev/badge/github.com/DavidGamba/go-getoptions/dag.svg["Go Reference", link="https://pkg.go.dev/github.com/DavidGamba/go-getoptions/dag"]
Lightweight Directed Acyclic Graph (DAG) Build System.
It allows building a list of tasks and then running the tasks in different DAG trees.
The tree dependencies are calculated and tasks that have met their dependencies are run in parallel.
There is an option to run it serially for cases where user interaction is required.
== Usage
A detailed example can be found in link:../examples/dag/main.go[]
To see what the build system would do run:
----
$ go run main.go build --quiet --dot
digraph G {
label = "build graph";
rankdir = TB;
"bt3";
"bt1";
"bt3" -> "bt1";
"bt2";
"bt3" -> "bt2";
}
----
----
$ go run main.go clean --quiet --dot
digraph G {
label = "clean graph";
rankdir = TB;
"ct1";
"ct3";
"ct1" -> "ct3";
"ct2";
"ct2" -> "ct3";
}
----
----
$ go run main.go
SYNOPSIS:
main [--dot] [--help|-?] [--quiet] <command> [<args>]
COMMANDS:
build build project artifacts
clean clean project artifacts
help Use 'main help <command>' for extra details.
OPTIONS:
--dot Generate graphviz dot diagram (default: false)
--help|-? (default: false)
--quiet (default: false)
Use 'main help <command>' for extra details.
exit status 1
----
== Motivation
I want something better than Bash scripts and Makefiles but not as limiting as Bazel, Buck and Please (never tried Pants but they are all inspired by Blaze).
Scons and Gradle are hard to work with.
Mage is awesome but the first thing I saw a developer do with it was run it in a debugger which is not possible with the current design.
Mage builds a separate go file that it then compiles and runs. It also relies on panic/recover for its control flow.
This build system leverages go-getoptions and a DAG tree to provide an alternative.
== Build System
In order to have a build system you need a couple of pieces:
* Define task dependencies (This package takes care of that).
It builds a tree for you that it then runs in parallel when possible.
* Define target and sources dependencies.
In other words, if my sources have changed I need to rebuild my targets.
Use the https://github.com/DavidGamba/dgtools/tree/master/fsmodtime["github.com/DavidGamba/dgtools/fsmodtime"] package.
* Task idempotency.
This so you can run your build system tasks over and over without risk.
This one is on you!
Finally, having an easy to use `os/exec` wrapper also helps a lot: https://github.com/DavidGamba/dgtools/tree/master/run["github.com/DavidGamba/dgtools/run"]
== ROADMAP
* Allow changing the ticker duration
* Add message every 30 seconds on what task is running.
== License
This file is part of go-getoptions.
Copyright (C) 2015-2024 David Gamba Rios
This Source Code Form is subject to the terms of the Mozilla Public
License, v. 2.0. If a copy of the MPL was not distributed with this
file, You can obtain one at http://mozilla.org/MPL/2.0/.
Documentation
¶
Overview ¶
Package dag - Lightweight Directed Acyclic Graph (DAG) Build System.
It allows building a list of tasks and then running the tasks in different DAG trees.
The tree dependencies are calculated and tasks that have met their dependencies are run in parallel. Max parallelism can be set and there is an option to run it serially for cases where user interaction is required.
Index ¶
- Variables
- func Stderr(ctx context.Context) io.Writer
- func Stdout(ctx context.Context) io.Writer
- type ContextKey
- type Errors
- type Graph
- func (g *Graph) AddTask(t *Task)
- func (g *Graph) DepthFirstSort() ([]*Vertex, error)
- func (g *Graph) Run(ctx context.Context, opt *getoptions.GetOpt, args []string) error
- func (g *Graph) SetMaxParallel(max int) *Graph
- func (g *Graph) SetOutputBuffer(w io.Writer) *Graph
- func (g *Graph) SetSerial() *Graph
- func (g *Graph) String() string
- func (g *Graph) Task(id string) *Task
- func (g *Graph) TaskDependensOn(t *Task, tDependencies ...*Task)
- func (g *Graph) Validate(tm *TaskMap) error
- type ID
- type Task
- type TaskMap
- type Vertex
Constants ¶
This section is empty.
Variables ¶
var ( ErrorTaskNil = fmt.Errorf("nil task given") ErrorTaskID = fmt.Errorf("missing task ID") ErrorTaskFn = fmt.Errorf("missing task function") ErrorTaskDuplicate = fmt.Errorf("task definition already exists") ErrorTaskNotFound = fmt.Errorf("task not found") ErrorTaskDependencyDuplicate = fmt.Errorf("task dependency already defined") ErrorGraphHasCycle = fmt.Errorf("graph has a cycle") ErrorTaskSkipped = fmt.Errorf("skipped") )
var ErrorSkipParents = fmt.Errorf("skip parents without failing")
ErrorSkipParents - Allows for conditional tasks that allow a task to Skip all parent tasks without failing the run
Functions ¶
Types ¶
type ContextKey ¶ added in v0.24.0
type ContextKey string
type Graph ¶
type Graph struct {
Name string
TickerDuration time.Duration
Vertices map[ID]*Vertex
// contains filtered or unexported fields
}
func (*Graph) DepthFirstSort ¶ added in v0.24.0
DepthFirstSort - Returns a sorted list with the Vertices https://en.wikipedia.org/wiki/Topological_sorting#Depth-first_search It returns ErrorGraphHasCycle is there are cycles.
func (*Graph) Run ¶
Run - Execute the graph tasks in parallel where possible. It checks for tasks updates every 1 Millisecond by default. Modify using the graph.TickerDuration
func (*Graph) SetMaxParallel ¶ added in v0.24.0
SetMaxParallel - Limit concurrency.
func (*Graph) SetOutputBuffer ¶ added in v0.24.0
SetOutputBuffer - Adds a buffer in the context passed to the task that allows the task logging to be sent to that buffer. At the end of the task, the in memory buffered output will be written to the given io.Writer.
The context keys are 'StdoutBuffer' and 'StderrBuffer' which can be retrieved with the helper functions. The helper functions default to os.Stdout and os.Stderr when no buffering is defined.
g.SetOutputBuffer(os.Stdout) stdout := dag.Stdout(ctx) stderr := dag.Stderr(ctx) fmt.Fprintf(stdout, "Output") fmt.Fprintf(stderr, "Error")
NOTE: Even though both stdout and stderr contex keys are provided, currently both are combined into a single output and written into the given io.Writer.
func (*Graph) SetSerial ¶
SetSerial - The call to Run() will run the Tasks serially. Useful when the tasks require user input and the user needs to see logs in order to make a decision.
func (*Graph) TaskDependensOn ¶
TaskDependensOn - Allows adding tasks to the graph and defining their edges.
type Task ¶
type Task struct {
ID ID
Fn getoptions.CommandFn
// contains filtered or unexported fields
}
type TaskMap ¶
type TaskMap struct {
// contains filtered or unexported fields
}
func (*TaskMap) Add ¶
Add - Adds a new task to the TaskMap. Errors collected for TaskMap.Validate().
Source Files