volgre_flow_processor

Multi Rule Engine Flow Processor in Golang poc 2

An asynchronous, non-blocking, concurrent, distributed, fault-tolerant, event-driven rule engine based flow processor using kafka events.

Install volgre application + dependencies:

$ git clone https://github.com/boomerlang/volgre_flow_processor

Run the application:

$ cd volgre_flow_processor

$ go build

$ ./volgre_flow_processor > run.log 2>&1 &

To see the flow the in action:

$ tail -f run_log

Abstract

This poc is totally different from the previous one. This implements the flow of processisng a set of tasks based on the evaluation of set of rule engine rule sets as shown in Hari's demo.

Every rule engine has an input and output state. In input a rule engine receives a data structure, evaluates its own rule sets against that data structure's fields, and in the output it return the data structure with modified fields.

Every rule engine logs the methods it executes founded in the rule set and evaluation time of the rule set.

The output from a rule engine becomes the input for the next rule engine selected for running its own rule set on this input.

The selection of the next task to be processed is controlled by the value of object that is passed from current rule engine to the next.

The flow:

1. The first rule engine in the EntryPoints map with a kafka listener waits for an event to appear on the topic it is bound to.

2. When the event appears , the data from the topic becomes the intial state and it is evaluated by the rule engine with the corresponding rule sets and then the input object is mutated and a new state of the object is generated at output.

3. This new state of the object decides which rule engine from the Task set runs its rule sets against this new state of the object.

4. This continues until there are no more tasks in Task set and based on the evaluation of the last task the flow switches in the ExitPoints from where it selects an appropiate entry that contains a kafka topic on which it will publish something.

The process and simulation flows are cooperative flows and they call each other strategically through kafka topics and the transition is done by mutating state by the rule sets evaluations.

The processing jumps from one flow to the other using kafka topics and state transitions dictated by the rule engine evaluating their own private rule sets.

Trigger the flow

Now the applications communicates exclusively through kafka topics (input and output). No more rest api.

To test the application send a message on an input topic and then listen with a consumer on an output topic where the application publishes the result of processing each entry in the EntryPoints map.

The input topic (where the applications waits for kafka messages):

On a console type:

$ kafka-console-producer.sh --bootstrap-server 141.147.22.26:9092 --topic volpay.instruction.receive < SampleData/payment-sample.json

The output topic (where the application publishes the result):

On a different console type:

$ kafka-console-consumer.sh --bootstrap-server 141.147.22.26:9092 --topic volgre.process.sim.flow.output

This acts as a debug topic where the application publish a message as a trace of what it is doing in that moment of time.

The output will be a sequence of json documents that contain information about current state, the rule engine that it is executed, the next state, the type of the rule engine that evalutes its own rule sets (process or simulation). The next json document in the sequence represents the next state and so on unil the last state it is reached and the flow ends. It will be seen that the processing jumps from process flow to simulation flow and back. In the simulation flow the state is preserved. Only in the process flow the state is mutated. This will be observed in the output json document sequence.

Internals

Internally, every file of rule set is mapped to a distinct rule engine.

Every rule engine acts as a distinct entity isolated from the other rule engines, whith its own name and internal, private state.

The rule engines processes comunicates with each other by means of the messages. A process cannot directly access the internal state of other processes, so no need for locks for synchronization orchestration. Processes communicates with each other through messages. Message can have any form and value.

This is a lock-free architecture.

This is an actor model that leverages the Go's concurrency building blocks: goroutines, channels, mutexes and contexts, and glues them together in a meaningful and easy to reason about unit of execution called 'process' or actor.

All rule engine workers are part of a supevision tree such that if a worker crashes it is automatically restarted by its supervisor and the error is logged such that if there is a bug to be able to tackle it asap.

The framework used is:

https://github.com/ergo-services/ergo

With this approach we can build massivelly concurent, distribuuted and fault-tolerant systems in Golang like in Erlang.

Motivation

Rather that stitch together different design patterns that may or may not play well with each other and most important, with go concurrency model, that would eventually lead to a system that is made from parts that competets with each other rather than work harmoniously with each other, and hard to debug leading to the so called Frankenstein syndrome.

Instead, we can walk on the shoulders of the giants and take advantage of the 40+ years battle-tested technology in production, that is Erlang/OTP platform, now available in Golang.

Using the actor based model we can implement whatever design pattern we want encapsulated in the so-called behaviour of the actor.

Using that behaviour we can create actors with that behaviour.