workflow-event-aggregator

workflow-event-aggregator

Currently not used

Aggregate events from multiple NATS Streaming clusters and forward them into a different NATS Streaming cluster. Basically a fan-in pattern.

Use case

When there are microservices deployed to the edge (a different location than the where the majority of microservices are deployed to), there will likely be a separate NATS Streaming cluster deployed to the edge act as a buffer for edge microservices to publish events into. the workflow-event-aggregator then proxy/forward those events published into the edge NATS Streaming cluster into the central NATS Streaming cluster.

Cloud events

the aggregator will only forward NATS Streaming msg that is valid cloudevents.

Lost connection

If the connection between aggregator and destination cluster is broken for an extended period of time, the aggregator will terminate. If the connections between aggregator and source clusters are broken, the aggregator will keep retrying to reconnect to the failed clusters for the lifetime of the aggregator. While aggregator attempting the reconnection, existing connection with other working source clusters continue to accepting events.

The assumption is that the aggregator will be deploy closer to the destination cluster than the source clusters. Thus connection between aggregator and the destination cluster is more stable. Whereas the connections with source clusters are more fragile, and there exists the possiblity of source clusters being unreachble for a period of time.

Simple local testing/dev setup

This will setup 3 NATS Streaming instances, stan1 is the "central" one where aggregator forward events into; stan2 and stan3 are the "edge" where the aggregator subscribe events from. There is also a nats-box(stan_sub) setup to subcribe to stan1 so events forwarded to stan1 will be visible in the logs of the nats-box.

prepare config file

cp example_config.yml config.yml

start the workflow-event-aggregator and other NATS Streaming servers.

docker-compose -f docker-compose.local.yml up --build -d

Note: this uses a different Dockerfile (Dockerfile.local)

restart stan_sub (a nats-box that subscribe to stan1), since it will fail on start up due to stan1 not being ready.

docker-compose -f docker-compose.local.yml restart stan_sub

monitor logs

docker-compose -f docker-compose.local.yml logs -f

run nats-box to publish msgs

docker run --rm -ti --network=host synadia/nats-box

publish example event (inside nats-box container)

stan-pub -s localhost:4222 -c stan -id nats-box-123 cyverse.events '{"specversion":"1.0","type":"org.cyverse.events.FooBar","source":"/example","id":"abcdef12345","time":"2018-04-05T17:31:00Z","datacontenttype":"application/json"}'

Note: localhost:4222 corrsponds to stan2, localhost:4333 corrsponds to stan3 in docker-compose.yml