README
¶
Maintainable automation.
🚧 Status: alpha 🚧
Early development
Please expect breaking changes. Feel free to file an issue if you
- find a bug
- need a feature added or tweaked for your use case
- have any other suggestions
PipeDream is an open-source, general-purpose automation tool.
It is an alternative to shell scripts - just as powerful, but more portable, composable, testable and reliable.
How it works
Define a pipeline in a yaml file. Using a simple, but extensible syntax, you can call, chain, merge, and split pipelines, invoke shell commands, handle errors, write tests, create mocks, and much more.
Getting started
Have a look at the Quick Start Guide, the Documentation or just run the program. It will offer to create a sample pipeline file in your current folder.
Installation
From npm
npm i -g @l9/pipedream
Using homebrew (Mac OS/linux)
brew install layer9berlin/tap/pipedream
From source (requires Go installed)
git clone https://github.com/Layer9Berlin/PipeDream
cd PipeDream
go build
pipedream
then select Install when promted.
This will compile PipeDream and run it locally, allowing you to execute the install.pipe file (very meta), which in turn installs everything you need to run PipeDream anywhere on your machine. If you make changes to the code, just run PipeDream and execute the installation.pipe again to update your installed version. You can skip the installation step entirely if you only want to run PipeDream in the project folder and don't need any of the utility tools.
If you use PipeDream a lot, it might be convenient to set up an alias like p that is faster to type.
Uninstall
Installed via npm
npm uninstall -g @l9/pipedream
Installed via homebrew
brew uninstall pipedream
Compiled from source
Simply run
pipedream
in the repo source directory and select Uninstall.
Not sure
If you don't recall how you installed PipeDream, run
pipedream version
and check the via field.
Why wouldn't I just use a shell script?
Shell scripting is an art that takes years to master. Tools like shellcheck and bats can help you improve the code quality of your scripts, but they don't solve the core problems:
- There are many traps for beginners to fall into
- Each command has its own syntax (not to mention subtle differences between shells)
- Error handling is so difficult it's rarely done well
- Even simple tasks often require contorted solutions that break for edge cases
- A lack of tests and documentation means you generally don't want to adapt an existing script
If you have used a modern CI/CD, orchestration or containerization tool, you are probably already familiar with a different way to write a script: defining tasks in yaml format. PipeDream is the natural extension of these tools to your localhost. Automate anything, anywhere, using a simple syntax that follows the principle of least astonishment.
"It's like Ansible for localhost!"
Anonymous
We have created PipeDream to simplify maintenance and dependency management tasks that used to involve a large number of steps, combining many different tools running both locally and on remote servers.
To illustrate its potential, we have used PipeDream to define a universal dependency manager pipe. It offers a single, consistent interface pulling together results from different package managers.
Documentation
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package cmd implements the `pipedream` command line tool
|
Package cmd implements the `pipedream` command line tool |
|
Package src contains the implementation of the `pipedream` functionalities
|
Package src contains the implementation of the `pipedream` functionalities |
|
custom
Package custom contains extensions and customizations of native/third party libraries
|
Package custom contains extensions and customizations of native/third party libraries |
|
custom/evaluate
Package evaluate provides custom functions extending the `govaluate` package
|
Package evaluate provides custom functions extending the `govaluate` package |
|
custom/filepath
Package filepath contains custom functions extending the native `filepath` package
|
Package filepath contains custom functions extending the native `filepath` package |
|
custom/io
Package io contains custom functions extending the native `io` package
|
Package io contains custom functions extending the native `io` package |
|
custom/math
Package math contains custom functions extending the native `math` package
|
Package math contains custom functions extending the native `math` package |
|
custom/stringmap
Package stringmap contains convenience functions for maps with string keys
|
Package stringmap contains convenience functions for maps with string keys |
|
custom/strings
Package strings contains custom functions extending the native `strings` package
|
Package strings contains custom functions extending the native `strings` package |
|
datastream
Package datastream provides a composable data stream, which pipes multiple inputs to multiple outputs according to the defined compositions
|
Package datastream provides a composable data stream, which pipes multiple inputs to multiple outputs according to the defined compositions |
|
logging
Package logging provides custom logging and activity indication
|
Package logging provides custom logging and activity indication |
|
logging/fields
Package fields provides custom log entry fields
|
Package fields provides custom log entry fields |
|
middleware
Package middleware provides middlewares that together implement pipedream's core functionalities
|
Package middleware provides middlewares that together implement pipedream's core functionalities |
|
middleware/catch
Package catch provides a middleware for graceful handling of stderr output
|
Package catch provides a middleware for graceful handling of stderr output |
|
middleware/collect
Package collect provides a middleware for gathering values into a map and saving them to disk
|
Package collect provides a middleware for gathering values into a map and saving them to disk |
|
middleware/dir
Package dir provides a middleware for changing the current working directory
|
Package dir provides a middleware for changing the current working directory |
|
middleware/docker
Package docker provides a middleware for execution within a Docker (Compose) container
|
Package docker provides a middleware for execution within a Docker (Compose) container |
|
middleware/each
Package each provides a middleware that copies some input into several child pipes running simultaneously
|
Package each provides a middleware that copies some input into several child pipes running simultaneously |
|
middleware/env
Package env provides a middleware managing environment variables
|
Package env provides a middleware managing environment variables |
|
middleware/extract
Package extract provides a middleware merging values from nested maps into pipe arguments
|
Package extract provides a middleware merging values from nested maps into pipe arguments |
|
middleware/inherit
Package inherit provides a middleware that passes arguments from a parent to its children
|
Package inherit provides a middleware that passes arguments from a parent to its children |
|
middleware/input
Package _input provides a middleware to overwrite a pipe's input directly
|
Package _input provides a middleware to overwrite a pipe's input directly |
|
middleware/interpolate
Package interpolate provides a middleware to substitute arguments or inputs into other arguments
|
Package interpolate provides a middleware to substitute arguments or inputs into other arguments |
|
middleware/output
Package _output provides a middleware to overwrite a pipe's output directly
|
Package _output provides a middleware to overwrite a pipe's output directly |
|
middleware/pipe
Package pipe provides a middleware to execute several commands in sequence
|
Package pipe provides a middleware to execute several commands in sequence |
|
middleware/select
Package _select provides a middleware that shows a selection prompt to the user
|
Package _select provides a middleware that shows a selection prompt to the user |
|
middleware/sequence
Package sequence provides a middleware that executes pipes synchronously, one after the other
|
Package sequence provides a middleware that executes pipes synchronously, one after the other |
|
middleware/shell
Package shell provides a middleware to execute commands in a shell
|
Package shell provides a middleware to execute commands in a shell |
|
middleware/ssh
Package ssh provides a middleware enabling remote command execution via SSH
|
Package ssh provides a middleware enabling remote command execution via SSH |
|
middleware/stack
Package stack defines a list of middleware items to be executed when a pipeline is run
|
Package stack defines a list of middleware items to be executed when a pipeline is run |
|
middleware/switch
Package _switch provides a middleware that replaces the output using the first match from a list of regexes
|
Package _switch provides a middleware that replaces the output using the first match from a list of regexes |
|
middleware/sync
Package sync provides a middleware that makes the execution function wait until the run has completed
|
Package sync provides a middleware that makes the execution function wait until the run has completed |
|
middleware/timer
Package timer provides a middleware that records execution time
|
Package timer provides a middleware that records execution time |
|
middleware/when
Package when provides a middleware that enables conditional execution
|
Package when provides a middleware that enables conditional execution |
|
middleware/with
Package with provides a middleware that extracts and processes input patterns
|
Package with provides a middleware that extracts and processes input patterns |
|
parsing
Package parsing provides a parser for pipeline yaml files
|
Package parsing provides a parser for pipeline yaml files |
|
pipeline
Package pipeline provides models to represent pipeline files and runs
|
Package pipeline provides models to represent pipeline files and runs |
|
run
Package run contains the implementation of the default `pipedream` shell command, selecting, parsing and executing pipelines
|
Package run contains the implementation of the default `pipedream` shell command, selecting, parsing and executing pipelines |
|
version
Package version provides the implementation of the version command, providing information about the current `pipedream` command line tool installation
|
Package version provides the implementation of the version command, providing information about the current `pipedream` command line tool installation |
|
Package util provides additional command line tools to handle specific, specialized tasks not covered by pipes or middleware
|
Package util provides additional command line tools to handle specific, specialized tasks not covered by pipes or middleware |
|
semver-compare
Parse and compare two or more semantic versions
|
Parse and compare two or more semantic versions |
|
traffic-light
Augment output with a green, amber or red status indicator
|
Augment output with a green, amber or red status indicator |