vmware-go-kcl

module
v0.0.0-...-80ae6ab Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 17, 2019 License: MIT

README

VMware-Go-KCL

A Fork? Why?

The upstream project does not allow for the use of a single DynamoDB table for N many workers. This fork has changed the schema for the DynamoDB checkpointer to allow for this.

Overview

Amazon Kinesis enables real-time processing of streaming data at massive scale. Kinesis Streams is useful for rapidly moving data off data producers and then continuously processing the data, be it to transform the data before emitting to a data store, run real-time metrics and analytics, or derive more complex data streams for further processing.

The VMware Kinesis Client Library for GO (VMware-Go-KCL) enables Go developers to easily consume and process data from [Amazon Kinesis][kinesis].

VMware-Go-KCL brings Go/Kubernetes community with Go language native implementation of KCL matching exactly the same API and functional spec of original Java KCL v2.0 without the resource overhead of installing Java based MultiLangDaemon.

Try it out

Prerequisites

Make sure hmake is version 1.3.1 or above and go is version 1.11 or above

hmake --version
1.3.1

Make sure to launch Docker daemon with specified DNS server --dns DNS-SERVER-IP

On Ubuntu, update the file /etc/default/docker to put --dns DNS-SERVER-IP in DOCKER_OPTS.

On Mac, set DNS in Docker PreferencesDaemonInsecure registries

Build & Run
hmake

# security scan
hmake scanast

# run test
hmake check

# run integration test
# update the worker_test.go to let it point to your Kinesis stream
hmake test

Documentation

VMware-Go-KCL matches exactly the same interface and programming model from original Amazon KCL, the best place for getting reference, tutorial is from Amazon itself:

Contributing

The vmware-go-kcl project team welcomes contributions from the community. Before you start working with vmware-go-kcl, please read our Developer Certificate of Origin. All contributions to this repository must be signed as described on that page. Your signature certifies that you wrote the patch or have the right to pass it on as an open-source patch. For more detailed information, refer to CONTRIBUTING.md.

License

MIT License

Directories

Path Synopsis
clientlibrary
checkpoint
* Copyright (c) 2018 VMware, Inc.
* Copyright (c) 2018 VMware, Inc.
common
* Copyright (c) 2018 VMware, Inc.
* Copyright (c) 2018 VMware, Inc.
config
* Copyright (c) 2018 VMware, Inc.
* Copyright (c) 2018 VMware, Inc.
interfaces
* Copyright (c) 2018 VMware, Inc.
* Copyright (c) 2018 VMware, Inc.
metrics
* Copyright (c) 2018 VMware, Inc.
* Copyright (c) 2018 VMware, Inc.
partition
* Copyright (c) 2018 VMware, Inc.
* Copyright (c) 2018 VMware, Inc.
utils
* Copyright (c) 2018 VMware, Inc.
* Copyright (c) 2018 VMware, Inc.
worker
* Copyright (c) 2018 VMware, Inc.
* Copyright (c) 2018 VMware, Inc.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL