gogreengrass

Overview of gogreengrass

💡 Deploy your cloud go lambda onto greengrass core devices without alteration

This library is a enabler to deploy standard aws go lambdas onto greengrass device lambdas. It also exposes the greengrass local API functions (greengrass SDK) to e.g. communicate with MQTT, local device shadow / secrets manager etc.

It also enables a go programmer to create much more efficient greengrass specific lambdas using the simplified lambda model if that is required.

NOTE: This is still very much in development!

Modes of operation

This library has two modes of operation to deploy and execute the go lambdas.

1. Primary mode is to use GGC C Runtime and deploy golang lambdas as greengrass lambda executable. In this mode the go lambda is dynamically linked to the GGC C runtime and is much more optimal.

2. One is standard lambda and thus wrapped as a python 3.7 and may be managed in AWS console and deployed onto GGC. The go lambda will be invoked through the python wrapper. This mode is good when you want to do scripting and use go to realize the lambda.

Both may use e.g. CDK to deploy the lambda.

For example, create this simple lambda that you want to execute in same thread as the main function.

//go:generate gogreengrass -sdkc

func main() {
	type MyEvent struct {
		Data  int    `json:"data"`
		Hello string `json:"hello"`
	}

	type MyResponse struct {
		Age   int    `json:"age"`
		Topic string `json:"topic"`
	}

	sdkc.Start(func(c context.Context, data MyEvent) (MyResponse, error) {

		lc, _ := lambdacontext.FromContext(c)

		fmt.Printf(
			"context: %v, topic: %s, data: '%v'\n",
			lc, lc.ClientContext.Custom["subject"], data,
		)

		resp := MyResponse{Age: 19, Topic: "feed/myfunc"}

		sdkc.NewQueue().PublishObject(
			"feed/testlambda", sdkc.QueueFullPolicyOptionAllOrError, &resp,
		)

		return resp, nil
	})
}

Make sure to have the shared library shim installed by gogreengrass -sdkc - see Command Line Tool. Since this file is decorated with generator pattern, go generate will create the library shim. Just do a standard go build go build -o testlambda main.go and include it into your deployment.

The following CDK definition can be used to deploy the above lambda (see sample: internal/test/sdkc/lambda).

import * as cdk from '@aws-cdk/core';
import * as lambda from '@aws-cdk/aws-lambda';
import path = require("path");

const GREENGRASS_EXECUTABLE = new lambda.Runtime('arn:aws:greengrass:::runtime/function/executable')

export class TestLambda extends cdk.Stack {

  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const testlambda = new lambda.Function(this, 'testlambda', {
      runtime: GREENGRASS_EXECUTABLE,
      functionName: 'testlambda',
      handler: 'testlambda',
      code: lambda.Code.fromAsset(path.join(__dirname, '../../_out/testlambda')),
      timeout: cdk.Duration.seconds(30),
      currentVersionOptions: {
        removalPolicy: cdk.RemovalPolicy.RETAIN,
      }
    });

    testlambda.currentVersion.addAlias('live')
  }
}

Note that the lambda runtime is in this case arn:aws:greengrass:::runtime/function/executable (if python use Runtime.PYTHON_3_7).

When doing npm run deploy it will show up in the IoT Core Greengrass console lambda for the greengrass group.

Command Line Tool

Install the command line tool by go get -u github.com/mariotoffia/gogreengrass. This tool may be used in order to generate the necessary go and python shim or to install the mock shared library if using C runtime.

gogreengrass -h emits the following:

gogreengrass v0.0.5
Usage: gogreengrass [--out PATH] [--package PACKAGE] [--binary BINARY] [--downloadsdk] [--force] [--sdkp] [--sdkc]

Options:
  --out PATH, -o PATH    The out path to write the generated go and python files. Default is current directory
  --package PACKAGE, -p PACKAGE
                         an optional package name instead of main
  --binary BINARY, -b BINARY
                         an optional name of the binary that the build system produces, default is foldername.o
  --downloadsdk, -d      If set to true, it will download the python sdk (it is needed to be in current folder)
  --force, -f            Force downloads the SDK (even if exists in target folder)
  --sdkp                 Writes the python/go shims to current folder (or out folder)
  --sdkc                 Installs the c runtime shared library in /tmp/gogreengrass
  --help, -h             display this help and exit
  --version              display version and exit

C Runtime

This is the preferred method to create your go lambdas. Use the sdkc package to interact with the lambda runtime and greengrass specific APIs such as local device shadow / secrets manager or publish data on MQTT etc.

Lambda

The go version of the lambda runtime is layered. The "slim" or simple layer and the standard AWS lambda layer. Depending on the use-case and performance on the device you may choose one over the other.

Standard Lambda

The standard AWS lambda layer is behaving exactly the same as a standard cloud lambda, hence portable.

When using the convenience function Start it starts the lambda dispatcher on the main thread and the lambda gets invoked on the main thread. This is more or less the standard cloud version of it.

	sdkc.Start(func(c context.Context, data MyEvent) (MyResponse, error) {
		// process the data here
	})

If you want to continue on main thread and fire up a dispatcher on a separate thread, you may use StartWithOpts to control this behavior.

	sdkc.StartWithOpts(
		func(c context.Context, data MyEvent) (MyResponse, error) { // <1>
		// process the data here
		}, 
		RuntimeOptionSeparateThread, // <2>
		true  // <3>
	)

<1> This function is executed on a single background thread. Hence, the invocations is serialized on that thread. The main thread continues. <2> The specifies the separate thread behavior. <3> If set to true, it will always fetch the payload. If false it is up to the lambda to fetch the data (see below).

Since the lambda function do take MyEvent the payload must be set to true in order to Unmarshal into that object. If lambda wants no payload or want to handle this itself, specify false.

// registered lambda
func(c context.Context) error {
	r := NewRequestReader()
	b := make([]byte, 256)
	for {
		n, err := 	r.Read(b)
		if n > 0 {
			// process the b[:n]
		}

		if err == io.EOF {
			break
		}

		if err != nil {
			return err
		}
	}
}

If you only want to read it all directly, e.g. want to do custom Unmarshal or other.

if buf, err := ioutil.ReadAll(NewRequestReader()); err == nil {
   // the complete request payload is in buf
}

In short, you may use standard portable go lambdas or very specialized on the "standard" track. You may even do more lightweight lambdas using the more low level lambda.

Slim Lambda

The slim lambda is a non reflective and no Unmarshal path and hence is more optimized. You have two options to register the lambda (as with regular lambda). The GGStart and GGStartWithOpts, it works exactly the same on registration part, except that the lambda function is fixed. You have to do all reading, writing and others yourself.

	sdkc.GGStart(func(lc *sdkc.LambdaContextSlim) { // <1>

		sdk.Log(sdkc.LogLevelInfo, "%s, %s\n", lc.ClientContext, lc.FunctionARN) // <2>
		sdk.Log(sdkc.LogLevelInfo, "Payload: %s\n", string(lc.Payload)) // <3>
       // <4>
	})

<1> The one and only function type to register. <2> ClientContext is a string that you may unmarshal yourself. <3> The payload is a []byte (in this case it will be populated since GGStart do set payload to true) <4> If you want to return data, you have to write either error output or return payload yourself.

As with standard lambda, one may register using GGStartWithOpts to change if background thread or read / write data yourself.

To write a response, do create a NewResponseWriter and do a Write(buff). If any errors is returned, use the NewErrorResponseWriter and do a Write(buff). Both of them implements the io.Writer interface.

Install C Runtime SDK Mock Library

You need to have the mock version of the shared library. Either follow the instructions in the greengrass core C SDK or use the gogreengrass ability to store libaws-greengrass-core-sdk-c.so in your /tmp/gogreengrass folder.

gogreengrass -sdkc

This writes the shared library (shim) that makes your go lambdas build and run. When deployed onto the greengrass core device, the real shared library is already present (this library shall never be part of the package) - see the greengrass core C SDK for more information.

Python wrapper

Python wrapper to deploy go lambdas into green grass. It also encapsulates the python greengrass SDK so go code may invoke the python SDK to do device shadow operations, secrets manager and publish data onto MQTT.

The main.go contains the lambda code in this sample.

package main

import (
	"context"
	"fmt"

	"github.com/aws/aws-lambda-go/lambdacontext"
	"github.com/mariotoffia/gogreengrass/sdk"
)

// once is invoked once when ggc lambda startup
func once() {

	type MyEvent struct {
		Data  int    `json:"data"`
		Hello string `json:"hello"`
	}

	type MyResponse struct {
		Age   int    `json:"age"`
		Topic string `json:"topic"`
	}

	sdk.Register(func(c context.Context, data MyEvent) (MyResponse, error) {

		lc, _ := lambdacontext.FromContext(c)

		fmt.Printf(
			"context: %v, (from-)topic: %s, data: '%v'\n",
			lc, lc.ClientContext.Custom["subject"], data,
		)

		return MyResponse{Age: 19, Topic: "feed/myfunc"}, nil
	})
}
func main() {
	// Standard AWS cloud lambda initialization code
	// Your standard lambda SDK registration code here
}

When you invoke the gogreengrass -sdkp -d, it will generate glue.py and glue.go where the python code has the lambda entry

# The actual function that you need to bind the lambda entry point to
def function_handler(event, context):
    result = invokeJSON(context, event)
    return result

You may remove that and provide with your own. The glue.go give the python runtime the ability to invoke your lambda code as well as able to call python SDK functions such as publish to MQTT etc. You may modify the glue code if you wish (you always have the ability to re-generate using gogreengrass -sdkp).

This is only needed to be done once (if not upgraded the gogreengrass sdk and breaking changes has been introduced) and thus, you may check-in the code. Publish the lambda as python 3.x runtime and then it will be deployable in a greengrass core group.

If you stick with the auto generated, make sure that your handler is set to: glue.function_handler. If you bind the IoT Core -> Lambda on e.g. topic: invoke/ggtest, open the test utility to and post to invoke/ggtest:

{
  "data": 49,
  "hello": "world!"
}

The log should print out the following in greengrass/ggc/var/log/user/your region/account/ggtest.log

[2020-11-16T21:55:08.122Z][INFO]-context: &{c760d735-5e69-4ffc-5a76-1460197dcd71 arn:aws:lambda:eu-central-1:033549287452:function:ggtest:11 { } {{   } map[] map[subject:invoke/ggtest]}}, (from-)topic: invoke/ggtest, data: '{49 world!}'

The above line is logged out from the following statement (in sample code)

fmt.Printf(
			"context: %v, (from-)topic: %s, data: '%v'\n",
			lc, lc.ClientContext.Custom["subject"], data,
		)