kassette-server

Kassette provides secure data pipelines that make it easy to collect data from every application and site, then activate it in your warehouse and business tools.

Why Kassette?

Microservices and enterprise applications are difficult to integrate together. Data needs to be transformed, API integrations written, recovery implemented to handle failures. Development teams spend a long time implementing non core functionality. Allowing business to bring new processes and tools forces significant time developing integrations.

Bringing in the iPaas

"Integration platform as a service" enable a new suite of tools to be built on top of the data pipeline. Kassette is a data pipeline that enables you to build your own integrations and data flows.

Getting Started

Setup a postgres DB with an Admin user and add to .env properties. Then simply run:

go run main.go

Installation

Kassette is available as a helm chart. You can run it using the following command:

helm repo add metaops https://metaops-solutions.github.io/helm-charts

helm install kassette-server metaops/kassette-server 

Run locally using docker-compose

This will create 2 Postgres Databaases: kassette and warehouse + builds and runs kassette-server kassette-server will run and listen on port 8088

docker-compose up

Configuration

Configuration is done in app but the application can be scaled

Configure destionations in the kassette database:

INSERT into source_config ( id, source, write_key ) VALUES ( 1, '{ "id": "1", "config": {}, "enabled": true, "writeKey": "write_key", "sourceName": "Camunda-test", "destinations": [ { "ID": "1", "Name": "Camunda-test", "Config": null, "Enabled": true, "RevisionID": "", "WorkspaceID": "", "Transformations": { "ID": "1", "Config": {}, "VersionID": "1" }, "IsProcessorEnabled": true, "DestinationDefinition": { "ID": "1", "Name": "power_bi", "Config": { "endpoint": "https://api.powerbi.com/beta/c4ae8b92-c69e-4f24-a16b-9a034ffa7e79/datasets/5998ed2b-b43c-4281-bdd5-1bc851b2697a/rows?experience=power-bi&key=kQ3YW5PG7ws1fy%2FUU3aau2P8wdeYxhB6yzBfbg6EtJ5WUrix6xgJl%2BRwnzTqooDAncwnc9poWOUijC1CnBue8g%3D%3D", "requestFormat": "ARRAY", "requestMethod": "WAREHOUSE" }, "DisplayName": "Power BI", "ResponseRules": null } } ], "sourceDefinition": { "category": "Workflow", "sourceId": "1", "sourceName": "Camunda" } }', 'write_key');

Releasing

We will use git tags to track the versions

git tag -a v1.0.0 -m "First tagged release of the kassette-server"
git push --tags

Usage

Documentation

The Kassette architecture is composed of 2 main components: Kassette Server Contains a processor which accepts data, persists to Postgres in a dataset. This is a job queue which is then picked up by the processor. The processor figures out which transformations and destinations are relevant for the data, performs these actions and maintains a job status table. A transformer which accepts data from the gateway, collects context and then returns to the gateway which places the

Kassette trasformer This is manages the sources, destinations and reports on health of the feeds. It's a web portal into the data pipeline.