streams

package module

v0.10.0 Latest Latest Go to latest Published: Jan 14, 2024 License: MIT Imports: 0 Imported by: 11

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

github.com/reugn/go-streams

Links

Open Source Insights

README ¶

go-streams

A lightweight stream processing library for Go.
go-streams provides a simple and concise DSL to build data pipelines. pipeline-architecture-example

Wiki
In computing, a pipeline, also known as a data pipeline, is a set of data processing elements connected in series, where the output of one element is the input of the next one. The elements of a pipeline are often executed in parallel or in time-sliced fashion. Some amount of buffer storage is often inserted between elements.

Overview

Building blocks:

Source - A Source is a set of stream processing steps that has one open output.
Flow - A Flow is a set of stream processing steps that has one open input and one open output.
Sink - A Sink is a set of stream processing steps that has one open input. Can be used as a Subscriber.

Implemented Flows (flow package):

Map
FlatMap
Filter
Reduce
PassThrough
Split
FanOut
RoundRobin
Merge
Flatten
Batch
Throttler
SlidingWindow
TumblingWindow
SessionWindow

Supported Connectors:

Go channels
File system
Network (TCP, UDP)
WebSocket
Aerospike
Apache Kafka
Apache Pulsar
NATS
Redis

Examples

Usage samples are available in the examples directory.

License

Licensed under the MIT License.

Documentation ¶

Overview ¶

Package streams specifies interfaces to be implemented by the streaming connectors and operators.

Constants ¶

This section is empty.

Variables ¶

This section is empty.

Functions ¶

This section is empty.

Types ¶

type Flow ¶

type Flow interface {
	Inlet
	Outlet
	Via(Flow) Flow
	To(Sink)
}

Flow represents a set of stream processing steps that has one open input and one open output. Programs can combine multiple Flows into sophisticated dataflow topologies. Implement this interface to create a custom stream transformation operator.

type Inlet ¶

type Inlet interface {
	In() chan<- any
}

Inlet represents a type that exposes one open input.

type Outlet ¶

type Outlet interface {
	Out() <-chan any
}

Outlet represents a type that exposes one open output.

type Sink ¶

type Sink interface {
	Inlet
}

Sink represents a set of stream processing steps that has one open input. A Sink will usually connect to a database or streaming platform to flush data from a pipeline. Implement this interface to create a custom sink connector.

type Source ¶

type Source interface {
	Outlet
	Via(Flow) Flow
}

Source represents a set of stream processing steps that has one open output. A Source will usually connect to a database or streaming platform to produce a stream of events/records. Implement this interface to create a custom source connector.

Source Files ¶

View all Source files

Directories ¶

Path	Synopsis
aerospike module
examples module
extension Package extension provides basic connector implementations.	Package extension provides basic connector implementations.
flow Package flow provides streams.Flow implementations.	Package flow provides streams.Flow implementations.
internal
assert
ospkg Package ospkg provides platform-specific utility functions and constants.	Package ospkg provides platform-specific utility functions and constants.
kafka module
nats module
pulsar module
redis module
ws module

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL