subscriptions

Contents

1. Overview
   1.1. Purpose
   1.2. Definitions
      1.2.1. Route
         1.2.1.1. Destination
         1.2.1.2. Condition
            1.2.1.2.1. Group Condition
            1.2.1.2.2. Key Condition
            1.2.1.2.3. Kiwi Condition
            1.2.1.2.4. Kiwi Tree Condition
            1.2.1.2.5. Kiwi Bird Condition
      1.2.2. Subscription
   
2. Configuration
   
3. Deployment
   3.1. Prerequisites
   3.2. Bare
   3.3. Docker
   3.4. K8s
      3.4.1. Helm
   
4. Usage
   4.1. Create
   4.2. Read
   4.3. Delete
   4.4. Search
      4.4.1. By Condition
      4.4.2. By Metadata
   
5. Design
   5.1. Requirements
   5.2. Approach
      5.2.1. Data Schema
         5.2.1.1. Subscription
         5.2.1.2. Route
         5.2.1.3. Group Condition
         5.2.1.4. Kiwi Condition
      5.2.2 Results Pagination
   5.3. Limitations
   
6. Contributing
   6.1. Versioning
   6.2. Issue Reporting
   6.3. Building
   6.4. Testing
      6.4.1. Functional
      6.4.2. Performance
   6.5. Releasing
   

1. Overview

Subscriptions storage service.

1.1. Purpose

The main function of the service is to find all subscriptions by a condition. For example, a message metadata is matching a certain pattern. Then it's necessary to find all subscriptions those have this pattern in the corresponding condition.

1.2. Definitions

1.2.1. Route

A subscription route describes the conditions to match an incoming message and a destination where the matching message should be routed to.

1.2.1.1. Destination

Destination is a free-form string describing a target (usually topic or subject) for a message matching the corresponding subscription. An incoming message should be routed to the specified destination when and only when subscription condition matches.

1.2.1.2. Condition

A condition represents a message matching criteria. The common properties for any type of condition are:

• unique id
• negation flag (Not). When the negation flag set to true the condition is treated as a negation, otherwise it's a proposition.

A group condition represents a group of child conditions coupled with a certain logic: And, Or, Xor.

A key condition is an abstract condition specifying a key that should match the message metadata key.

A kiwi (Key-Input WIldcard) condition is a key condition containing the metadata value pattern. Also, kiwi condition has a partial attribute to represent whether a value part is allowed to match the pattern.

A kiwi-tree specific condition implementation of a kiwi condition. It comes with additional limitations on the pattern syntax but allows resolving a condition by a key/value pair in a O(log(N)) time.

A specific kiwi condition implementation. It should come without any limitation on the pattern syntax but will resolve a condition in O(N) time. Not implemented yet.

1.2.2. Subscription

Subscriptions is an entity linking the condition with the routes. A subscription also has unique id generated on creation and human-readable metadata.

2. Configuration

The service is configurable using the environment variables:

3. Deployment

3.1. Prerequisites

A general note is that there should be a MongoDB cluster deployed to be used for storing the pattern data. It's possible to obtain a free cluster for testing purposes using Atlas.

3.2. Bare

Preconditions:

1. Build patterns executive using make build
2. Run the kiwi-tree dependency services (x2: complete/partial)

Then run the command:

API_PORT=8080 \
DB_URI=mongodb+srv://localhost/\?retryWrites=true\&w=majority \
DB_NAME=subscriptions \
DB_TABLE_NAME=subscriptions \
API_KIWI_TREE_COMPLETE_URI=http://localhost:8081 \
API_KIWI_TREE_PARTIAL_URI=http://localhost:8082 \
./subscriptions

3.3. Docker

TODO: run the kiwi-tree (x2) and subscriptions in the same network

alternatively, it's possible to build and run the new docker image in place using the command: (note that the command below requires all env vars to be set in the file env.txt)

make run

3.4. K8s

TODO

3.4.1. Helm

Create a helm package from the sources:

helm package helm/subscriptions/

Install the helm chart:

helm install subscriptions ./subscriptions-<CHART_VERSION>.tgz \
  --values helm/subscriptions/values-db-uri.yaml

where

• values-db-uri.yaml contains the value override for the DB URI
• <CHART_VERSION> is the helm chart version

4. Usage

The service provides basic gRPC interface to perform the operation on subscriptions.

4.1. Create

Example:

grpcurl \
  -plaintext \
  -proto api/grpc/service.proto \
  -d '{"metadata" :{"description": "my subscription"}, "route": { "destinations": ["dst0"], "condition": {"kiwiTreeCondition": {"not" :false, "key": "key0", "pattern": "pattern*", "partial": false}}}}' \
  localhost:8080 \
  subscriptions.Service/Create

Yet another example:

grpcurl \
  -plaintext \
  -proto api/grpc/service.proto \
  -d '{"metadata": {"name": "sub1", "description": "my subscription 1"}, "route": {"destinations": ["dst1"], "condition": {"groupCondition": {"not": false, "logic": 0, "group": [{"kiwiTreeCondition": {"not": false, "key": "key0", "pattern": "pattern?", "partial": false}}, {"kiwiTreeCondition": {"not": true, "key": "key1", "pattern": "pattern1", "partial": true}}]}}}}' \
  localhost:8080 \
  subscriptions.Service/Create

4.2. Read

Example:

grpcurl \
  -plaintext \
  -proto api/grpc/service.proto \
  -d '{"id": "3426d090-1b8a-4a09-ac9c-41f2de24d5ac"}' \
  localhost:8080 \
  subscriptions.Service/Read

4.3. Delete

Example:

grpcurl \
  -plaintext \
  -proto api/grpc/service.proto \
  -d '{"id": "f7102c87-3ce4-4bb0-8527-b4644f685b13"}' \
  localhost:8080 \
  subscriptions.Service/Delete

4.4. Search

4.4.1. By Condition

The search by condition purpose is to be used by a resolver to find the matching subscriptions.

Example:

grpcurl \
  -plaintext \
  -proto api/grpc/service.proto \
  -d '{"limit": 100, "kiwiConditionQuery": {"key": "key0", "pattern": "pattern*", "partial": false}}' \
  localhost:8080 \
  subscriptions.Service/SearchByCondition

4.4.2. By Metadata

The search by metadata purpose is to be used by a user to find own subscriptions.

Example:

grpcurl \
  -plaintext \
  -proto api/grpc/service.proto \
  -d '{"limit": 100, "metadata": {"description": "my subscription"}}' \
  localhost:8080 \
  subscriptions.Service/SearchByMetadata

5. Design

5.1. Requirements

5.2. Approach

5.2.1. Data Schema

Subscriptions are stored in the single table under the denormalized schema.

Example data:

- id: "2f63ea52-a66c-4b93-92f1-12aa2831cd2c"
  metadata:
    name: subscription0
    description: Anything related to orders that are not in Helsinki
    user: "e7fae6df-f0e7-4a6b-b8ae-3802a7927f7e"  
  route:
    destinations:
      - /dev/null
    condition:
      base:
        id: "123e4567-e89b-12d3-a456-426614174000"
        not: false
      logic: "And"
      group:
        - id: "14cadd71-c662-4f1a-8b0f-3b17dfb107f5"
          base:
            not: false
          partial: true
          key: "subject"
          pattern: "orders"
        - id: "c00e1228-fd78-4761-8f59-fbbfa690b9a9"
          base:
            not: true
          partial: false
          key: "location"
          pattern: "Helsinki"

5.2.1.1. Subscription

A subscription is immutable by design, hence there's no update operation for this. If a user wants to change a subscription they need to delete it 1st and then create again.

5.2.1.2. Route

5.2.1.3. Group Condition

5.2.1.4. Kiwi Condition

5.2.2. Results Pagination

The limit and cursor search parameters are used to support the results' pagination.

5.3. Limitations

6. Contributing

6.1. Versioning

The service uses the semantic versioning. The single source of the version info is the git tag:

git describe --tags --abbrev=0

6.2. Issue Reporting

TODO

6.3. Building

make build

Generates the sources from proto files, compiles and creates the subscriptions executable.

6.4. Testing

6.4.1. Functional

make test

6.4.2. Performance

TODO

6.5. Releasing

To release a new version (e.g. 1.2.3) it's enough to put a git tag:

git tag -v1.2.3
git push --tags

The corresponding CI job is started to build a docker image and push it with the specified tag (+latest).