messages

Contents

1. Overview
   1.1. Purpose
   1.2. Definitions
   
2. Configuration
   
3. Deployment
   3.1. Prerequisites
   3.2. Bare
   3.3. Docker
   3.4. K8s
      3.4.1. Helm
   
4. Usage
   4.1. Put Batch
   4.2. Get Batch
   4.3. Delete Batch
   4.4. Clean
   
5. Design
   5.1. Requirements
   5.2. Approach
      5.2.1. Data Schema
   5.3. Limitations
   
6. Contributing
   6.1. Versioning
   6.2. Issue Reporting
   6.3. Building
   6.4. Testing
      6.4.1. Functional
      6.4.2. Performance
   6.5. Releasing
   

1. Overview

Temporary message storage service.

1.1. Purpose

The service purpose is to store all incoming messages.

1.2. Definitions

TODO

2. Configuration

The service is configurable using the environment variables:

3. Deployment

3.1. Prerequisites

A general note is that there should be a MongoDB cluster deployed to be used for storing the quota data. It's also possible to obtain a free cluster for testing purposes using Atlas.

3.2. Bare

Preconditions:

1. Build messages executive using make build

Then run the command:

API_PORT=50051 \
DB_URI=mongodb+srv://localhost/\?retryWrites=true\&w=majority \
DB_NAME=messages \
DB_TABLE_NAME=messages \
  ./messages

3.3. Docker

make run

3.4. K8s

Prepare the image pull secret:

kubectl create secret docker-registry github-registry \
  --docker-server ghcr.io \
  --docker-username=<GITHUB_USER_NAME> \ 
  --docker-password=<GITHUB_CR_PAT> \
  --docker-email=<USER_EMAIL>

3.4.1. Helm

Create a helm package from the sources:

helm package helm/messages/

Install the helm chart:

helm install messages ./messages-<CHART_VERSION>.tgz \
  --values helm/messages/values-db-uri.yaml

where

• values-db-uri.yaml contains the value override for the DB URI
• <CHART_VERSION> is the helm chart version

4. Usage

The service provides basic gRPC interface to perform public and private operations on messages.

4.1. Put Batch

Example command:

grpcurl \
  -plaintext \
  -proto api/grpc/service.proto \
  -d @ \
  localhost:50051 \
  awakari.messages.Service/PutBatch

Payload:

{
   "msgs": [
      {
         "id": "3426d090-1b8a-4a09-ac9c-41f2de24d5ac",
         "type": "example.type",
         "source": "example/uri",
         "spec_version": "1.0",
         "attributes": {
            "subject": {
               "ce_string": "Obi-Wan Kenobi"
            },
            "time": {
               "ce_timestamp": "1985-04-12T23:20:50.52Z"
            }
         },
         "text_data": "I felt a great disturbance in the force"
      },
      {
         "id": "3426d090-1b8a-4a09-ac9c-41f2de24d5ad",
         "type": "example.type",
         "source": "example/uri",
         "spec_version": "1.0",
         "attributes": {
            "subject": {
               "ce_string": "Yoda"
            },
            "time": {
               "ce_timestamp": "1985-05-11T12:02:05.25Z"
            }
         },
         "text_data": "Try not. Do or do not. There is no try."
      },
      {
         "id": "3426d090-1b8a-4a09-ac9c-41f2de24d5ae",
         "type": "example.type",
         "source": "example/uri",
         "spec_version": "1.0",
         "attributes": {
            "subject": {
               "ce_string": "Qui-Gon Jinn"
            },
            "time": {
               "ce_timestamp": "1985-06-08T14:31:41.16Z"
            }
         },
         "text_data": "The ability to speak does not make you intelligent."
      }
   ]
}

4.2. Get Batch

Example command:

grpcurl \
  -plaintext \
  -proto api/grpc/service.proto \
  -d @ \
  localhost:50051 \
  awakari.messages.Service/GetBatch

Payload:

{
   "ids": [
      "3426d090-1b8a-4a09-ac9c-41f2de24d5ac",
      "3426d090-1b8a-4a09-ac9c-41f2de24d5ad",
      "3426d090-1b8a-4a09-ac9c-41f2de24d5ae"
   ]
}

4.3. Delete Batch

Batch deletion API function is to be used internally by the writer service.

Example command:

grpcurl \
  -plaintext \
  -proto api/grpc/service.proto \
  -d @ \
  localhost:50051 \
  awakari.messages.Service/DeleteBatch

Payload:

{
   "ids": [
      "3426d090-1b8a-4a09-ac9c-41f2de24d5ac",
      "3426d090-1b8a-4a09-ac9c-41f2de24d5ad",
      "3426d090-1b8a-4a09-ac9c-41f2de24d5ae"
   ]
}

4.4. Clean

Example command:

grpcurl \
  -plaintext \
  -proto api/grpc/cleaner/service.proto \
  -d @ \
  localhost:50051 \
  awakari.messages.cleaner.Service/Clean

5. Design

5.1. Requirements

5.2. Approach

The messages service may be considered as a pipeline with:

• input, consuming the messages those have any resolved subscription matches
• output, delivering the new matching messages by a subscription

5.2.1. Data Schema

5.3. Limitations

6. Contributing

6.1. Versioning

The service uses the semantic versioning. The single source of the version info is the git tag:

git describe --tags --abbrev=0

6.2. Issue Reporting

TODO

6.3. Building

make build

Generates the sources from proto files, compiles and creates the messages executable.

6.4. Testing

6.4.1. Functional

make test

6.4.2. Performance

TODO

6.5. Releasing

To release a new version (e.g. 1.2.3) it's enough to put a git tag:

git tag -v1.2.3
git push --tags

The corresponding CI job is started to build a docker image and push it with the specified tag (+latest).