goguardian-go-kcl

Amazon Kinesis Client Library for Go

This package provides an interface to the Amazon Kinesis Client Library (KCL) MultiLangDaemon, which is part of the Amazon KCL for Java. Developers can use the Amazon KCL to build distributed applications that process streaming data reliably at scale. The Amazon KCL takes care of many of the complex tasks associated with distributed computing, such as load-balancing across multiple instances, responding to instance failures, checkpointing processed records, and reacting to changes in stream volume. This interface manages the interaction with the MultiLangDaemon so that developers can focus on implementing their record processor executable. A record processor executable typically looks something like:

package main

import (
	"github.com/goguardian/goguardian-go-kcl/kcl"
)

type myProcessor struct{}

func (m *myProcessor) Initialize(*kcl.InitializationInput)           { /* handle init */ }
func (m *myProcessor) ProcessRecords(*kcl.ProcessRecordsInput)       { /* handle process */ }
func (m *myProcessor) LeaseLost(*kcl.LeaseLostInput)                 { /* handle lease lost */ }
func (m *myProcessor) ShardEnded(*kcl.ShardEndedInput)               { /* handle shard end */ }
func (m *myProcessor) ShutdownRequested(*kcl.ShutdownRequestedInput) { /* handle shutdown */ }

func main() {
	processor := &myProcessor{}
	process := kcl.GetKCLProcess(processor)
	err := process.Run()
	if err != nil {
		panic(err)
	}
}

Before You Get Started

Install Go and make sure your go version matches the go version in the go.mod file.

Before running the sample, you'll want to make sure that your environment is configured to allow the sample to use your AWS Security Credentials.

By default, the sample uses the DefaultCredentialsProvider so you'll want to make your credentials available to one of the credentials providers in that provider chain. There are several ways to do this such as providing a ~/.aws/credentials file, or if you're running on EC2, you can associate an IAM role with your instance with appropriate access.

Running the Sample

There is a single Makefile target to run the sample. Simply run:

make run_sample

This command will do the following:

1. Download jars necessary to run the MultiLangDaemon into a ./jar folder.
2. Build the runner app located at ./runner which is used to run the Java MultiLangDaemon.
3. Build the sample processor executable located at ./sample.
4. Run the Java MultiLangDaemon which will spawn the sample processor.

Running integration tests

Ensure you have docker-compose. We leverage LocalStack to emulate Kinesis locally. Also ensure you have the $JAVA_HOME environment variable set. For MacOS see this guide.

make run_integ_test

Under the Hood - What You Should Know about Amazon KCL's MultiLangDaemon

Amazon KCL for Go uses Amazon KCL for Java internally. AWS implemented a Java-based daemon, called the MultiLangDaemon that does all the heavy lifting. This approach has the daemon spawn the user-defined record processor script/program as a sub-process. The MultiLangDaemon communicates with this sub-process over standard input/output using a simple protocol, and therefore the record processor script/program can be written in any language.

At runtime, there will always be a one-to-one correspondence between a record processor, a child process, and an Amazon Kinesis Shard. The MultiLangDaemon will make sure of that, without any need for the developer to intervene.

In this release, we have abstracted these implementation details away and exposed an interface that enables you to focus on writing record processing logic in Go. This approach enables Amazon KCL to be language agnostic, while providing identical features and similar parallel processing model across all languages.

See Also

• Developing Consumer Applications for Amazon Kinesis Using the Amazon Kinesis Client Library
• The Amazon KCL for Java
• The Amazon KCL for Ruby
• The Amazon Kinesis Documentation
• The Amazon Kinesis Forum

Release Notes

Upgraded go version in go.mod

License

License