numaflow

package module

v1.2.1 Latest Latest Go to latest Published: May 7, 2024 License: Apache-2.0 Imports: 2 Imported by: 0

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

github.com/numaproj/numaflow

Links

Open Source Insights

README ¶

Numaflow

Summary

Numaflow is a Kubernetes-native tool for running massively parallel stream processing. A Numaflow Pipeline is implemented as a Kubernetes custom resource and consists of one or more source, data processing, and sink vertices.

Numaflow installs in a few minutes and is easier and cheaper to use for simple data processing applications than a full-featured stream processing platforms.

Use Cases

Real-time data analytics applications.
Event driven applications such as anomaly detection, monitoring, and alerting.
Streaming applications such as data instrumentation and data movement.
Workflows running in a streaming manner.

Key Features

Kubernetes-native: If you know Kubernetes, you already know how to use Numaflow.
Language agnostic: Use your favorite programming language.
Exactly-Once semantics: No input element is duplicated or lost even as pods are rescheduled or restarted.
Auto-scaling with back-pressure: Each vertex automatically scales from zero to whatever is needed.

Data Integrity Guarantees:

Minimally provide at-least-once semantics
Provide exactly-once semantics for unbounded and near real-time data sources
Preserving order is not required

Roadmap

Session Window (1.2)

Demo

Resources

Documentation ¶

Constants ¶

This section is empty.

Variables ¶

This section is empty.

Functions ¶

This section is empty.

Types ¶

type Version ¶

type Version struct {
	Version      string
	BuildDate    string
	GitCommit    string
	GitTag       string
	GitTreeState string
	GoVersion    string
	Compiler     string
	Platform     string
}

Version contains version information

func GetVersion ¶

func GetVersion() Version

GetVersion returns the version information

func (Version) String ¶

func (v Version) String() string

String outputs the version as a string

Source Files ¶

View all Source files

version.go

Directories ¶

Path	Synopsis
cmd
commands
hack
crdgen
gen-openapi-spec
json-schema
pkg
apis/numaflow/v1alpha1 +kubebuilder:object:generate=true +groupName=numaflow.numaproj.io +k8s:deepcopy-gen=package,register +k8s:openapi-gen=true	+kubebuilder:object:generate=true +groupName=numaflow.numaproj.io +k8s:deepcopy-gen=package,register +k8s:openapi-gen=true
apis/proto/daemon Package daemon is a reverse proxy.	Package daemon is a reverse proxy.
client/clientset/versioned
client/clientset/versioned/fake This package has the automatically generated fake clientset.	This package has the automatically generated fake clientset.
client/clientset/versioned/scheme This package contains the scheme of the automatically generated clientset.	This package contains the scheme of the automatically generated clientset.
client/clientset/versioned/typed/numaflow/v1alpha1 This package has the automatically generated typed clients.	This package has the automatically generated typed clients.
client/clientset/versioned/typed/numaflow/v1alpha1/fake Package fake has the automatically generated clients.	Package fake has the automatically generated clients.
client/informers/externalversions
client/informers/externalversions/internalinterfaces
client/informers/externalversions/numaflow
client/informers/externalversions/numaflow/v1alpha1
client/listers/numaflow/v1alpha1
daemon/client
daemon/server
daemon/server/service Package service is built for querying metadata and to expose it over daemon service.	Package service is built for querying metadata and to expose it over daemon service.
daemon/server/service/rater Package rater provides the functionality to calculate the processing rate of each vertex partition.	Package rater provides the functionality to calculate the processing rate of each vertex partition.
forwarder Package forwarder defines the interfaces for data forwarders in different type of vertices.	Package forwarder defines the interfaces for data forwarders in different type of vertices.
isb
isb/stores/jetstream
isb/stores/redis
isb/stores/simplebuffer
isb/testutils
isbsvc
metrics
reconciler Package reconciler defines implementations of the Reconciler interface defined at sigs.k8s.io/controller-runtime/pkg/reconcile.Reconciler.	Package reconciler defines implementations of the Reconciler interface defined at sigs.k8s.io/controller-runtime/pkg/reconcile.Reconciler.
reconciler/cmd
reconciler/isbsvc
reconciler/isbsvc/installer
reconciler/pipeline
reconciler/vertex
reconciler/vertex/scaling Package scaling provides the autoscaling capability for Numaflow.	Package scaling provides the autoscaling capability for Numaflow.
reduce Package reduce reads messages from isb and is responsible for the first part of reduce subsystem.	Package reduce reads messages from isb and is responsible for the first part of reduce subsystem.
reduce/applier
reduce/pbq
reduce/pbq/partition Package partition is a tuple containing (start, end) time and an optional slot.	Package partition is a tuple containing (start, end) time and an optional slot.
reduce/pbq/wal
reduce/pbq/wal/aligned
reduce/pbq/wal/aligned/fs Package fs implements write-ahead-log on FileSystem.	Package fs implements write-ahead-log on FileSystem.
reduce/pbq/wal/aligned/memory
reduce/pbq/wal/noop
reduce/pbq/wal/unaligned
reduce/pbq/wal/unaligned/fs Package fs implements write-ahead-log on FileSystem.	Package fs implements write-ahead-log on FileSystem.
reduce/pbq/wal/unaligned/noop
reduce/pnf
sdkclient
sdkclient/error
sdkclient/grpc
sdkclient/grpc_resolver
sdkclient/mapper
sdkclient/mapstreamer
sdkclient/reducer
sdkclient/serverinfo
sdkclient/sessionreducer
sdkclient/sideinput
sdkclient/sinker
sdkclient/source
sdkclient/sourcetransformer
shared/clients/nats
shared/clients/nats/test
shared/clients/redis
shared/ewma
shared/expr
shared/health-status-code
shared/idlehandler
shared/kvs
shared/kvs/inmem Package inmem package implements the watermark progression using in mem store as the KV store.	Package inmem package implements the watermark progression using in mem store as the KV store.
shared/kvs/jetstream Package jetstream package implements the kv store and watcher using Jetstream.	Package jetstream package implements the kv store and watcher using Jetstream.
shared/kvs/noop
shared/logging
shared/queue
shared/tls
shared/util
shuffle
sideinputs Package sideinputs is used for Side Inputs features.	Package sideinputs is used for Side Inputs features.
sideinputs/initializer Package initializer is used for init container on the vertex pod to initialize the Side Inputs data.	Package initializer is used for init container on the vertex pod to initialize the Side Inputs data.
sideinputs/manager Package manager is used for run the service in the numa container of a Side Inputs Manager.	Package manager is used for run the service in the numa container of a Side Inputs Manager.
sideinputs/synchronizer Package synchronizer is used for the service in the sidecar container of a vertex pod for watching Side Inputs data changes.	Package synchronizer is used for the service in the sidecar container of a vertex pod for watching Side Inputs data changes.
sideinputs/utils
sinks
sinks/blackhole
sinks/forward
sinks/kafka
sinks/logger
sinks/sinker
sinks/udsink
sources
sources/forward Package forward is used for creating a data forwarder for the source vertex.	Package forward is used for creating a data forwarder for the source vertex.
sources/forward/applier
sources/generator Package generator contains an implementation of an in-memory generator that generates payloads in json format.	Package generator contains an implementation of an in-memory generator that generates payloads in json format.
sources/http
sources/kafka
sources/nats
sources/sourcer
sources/transformer
sources/transformer/builtin
sources/transformer/builtin/event_time
sources/transformer/builtin/filter
sources/transformer/builtin/time_extraction_filter
sources/udsource Package udsource implements user-defined source, enabling the platform to communicate with user-defined data sources through gRPC.	Package udsource implements user-defined source, enabling the platform to communicate with user-defined data sources through gRPC.
udf
udf/builtin
udf/builtin/cat
udf/builtin/filter
udf/forward Package forward is used for creating a data forwarder for the UDF vertex.	Package forward is used for creating a data forwarder for the UDF vertex.
udf/forward/applier
udf/rpc Package rpc provides the interface to invoke UDFs (map, mapstream and reduce).	Package rpc provides the interface to invoke UDFs (map, mapstream and reduce).
watermark/entity Package entity is the smallest processor entity for which the watermark will strictly monotonically increase.	Package entity is the smallest processor entity for which the watermark will strictly monotonically increase.
watermark/fetch
watermark/generic
watermark/generic/jetstream
watermark/publish
watermark/store
watermark/timeline
watermark/wmb Package wmb represents the offset-timeline pair and its corresponding encoder and decoder.	Package wmb represents the offset-timeline pair and its corresponding encoder and decoder.
window Package window implements windowing constructs.	Package window implements windowing constructs.
window/strategy/fixed Package fixed implements Fixed windows.	Package fixed implements Fixed windows.
window/strategy/session
window/strategy/sliding Package sliding implements Sliding windows.	Package sliding implements Sliding windows.
server Package server is used for running UX backend server.	Package server is used for running UX backend server.
apis
apis/v1
authn
authz
cmd/server
cmd/serversecretinit
common
routes
test
api-e2e
e2e-api
fixtures
webhook Package webhook is used for running validating admission webhook.	Package webhook is used for running validating admission webhook.
cmd
validator

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL