README
¶
Linear Orchestration Packages
Introduction
Go has a great system for doing intricate pipelining via goroutines and channels. The problem is that managing channel input/output and error conditions in a pipeline can get dicey, especially if you introduce parrallel operations at certain stages in the pipeline.
Problems such as pipelines freezing and your not sure where or you get the old panic "sending on a closed channel", etc...
Then there are problems such as if data fed into the pipeline represents part of a whole, but there are mutliple sources feeding into the pipeline (aka you cannot close the input channel), how do you know when you are done?
These packages are designed around the "concurrency isn't parallelism" idea, but we throw in a dose of parallelism too.
Concurrent AND Parallel
The packages here are both concurrent and parallel. You can create multiple of the same pipeline with a shared input channel (and output in the case of the streaming package).
Inside each stage of a pipeline, you can have parallel processors.
Having concurrent Pipelines with parallel Stages is overkill for CPU bound pipelines but the parallel stages can alleviate IO bound Stages.
Two packages to choose from
- promise/ should be used when each request into the Pipeline represents a single response
- streaming/ should be used when requests may be part of a whole that result in a single outcome
Note: These packages are for linear Pipelines
These packages officially only supports two paths:
- linear processing from Stage A-Z
- breaking out of the Pipeline for an error
Directories
¶
| Path | Synopsis |
|---|---|
|
Package promise provides a processing pipeline where every input results in a response as a promise.
|
Package promise provides a processing pipeline where every input results in a response as a promise. |
|
examples/stroke
DON'T USE THIS EXAMPLE FOR PROMISE....
|
DON'T USE THIS EXAMPLE FOR PROMISE.... |
|
examples/stroke/pipeline
This is really the wrong use case for this type of Pipeline, but I wanted to see if I could make it work and wanted to test various error conditions I knew I would discover while I wrote out all the enumerators.
|
This is really the wrong use case for this type of Pipeline, but I wanted to see if I could make it work and wanted to test various error conditions I knew I would discover while I wrote out all the enumerators. |
|
Package streaming provides a processing pipeline where your stream a group of inputs to be processed by a Pipeline.
|
Package streaming provides a processing pipeline where your stream a group of inputs to be processed by a Pipeline. |
|
examples/stroke
This takes some health data on strokes you can find at: https://www.kaggle.com/fedesoriano/stroke-prediction-dataset It converts that data to a standard format with enumerators in the first stage and then writes out some stats in the second stage (how many were Males, Females, Married, ...).
|
This takes some health data on strokes you can find at: https://www.kaggle.com/fedesoriano/stroke-prediction-dataset It converts that data to a standard format with enumerators in the first stage and then writes out some stats in the second stage (how many were Males, Females, Married, ...). |
|
examples/stroke/pipeline
This takes some health data on strokes you can find at: https://www.kaggle.com/fedesoriano/stroke-prediction-dataset It converts that data to a standard format with enumerators in the first stage and then writes out some stats in the second stage (how many were Males, Females, Married, ...).
|
This takes some health data on strokes you can find at: https://www.kaggle.com/fedesoriano/stroke-prediction-dataset It converts that data to a standard format with enumerators in the first stage and then writes out some stats in the second stage (how many were Males, Females, Married, ...). |