cassandra

Conduit Connector for Cassandra

Conduit destination connector for Cassandra.

How to build?

Run make build to build the connector.

Testing

Run make test to run all the unit tests. Run make test-integration to run the integration tests.

The Docker compose file at test/docker-compose.yml can be used to run the required resource locally.

Destination

This destination connector pushes data from upstream resources to Cassandra via Conduit.

It parses the record received into a CQL (Cassandra Query Language) and executes that CQL into Cassandra using gocql. If the record has the operation create or snapshot then it's parsed into an INSERT query, if the operation is update then it's parsed into an UPDATE query, and DELETE query for delete operation.

Make sure that the destination table that the connector will write the records to, has the same schema as the payload for the records received. so if the payload looks like this:

{
  "id": 1,
  "name": "john",
  "full-time": true,
  "salary": 1000.1,
  "age": 25
}

then you should have a Cassandra table that looks like:

CREATE TABLE table_name ( id int NOT NULL, name varchar(255), full_time bool, salary double, age int, PRIMARY KEY (id));

Configuration

Table name

If a record contains a cassandra.table property in its metadata it will be inserted in that table, otherwise it will fall back to use the table configured in the connector. Thus, a Destination can support multiple tables in a single connector.

Example pipeline configuration file

   pipelines:
   cassandra-pipeline:
     status: running
     name: example-pipeline
     description: write data into Cassandra.
     connectors:
       postgres-con:
         type: source
         plugin: builtin:postgres # you can use any other source connector, this is just an example.
         name: postgres-source
         settings:
           url: postgresql://username:pass@127.0.0.1:5432/mydb #postgresql://{username}:{password}@{host}:{port}/{database}
           table: employees
           orderingColumn: id
       cassandra-con:
         type: destination
         plugin: standalone:cassandra
         name: cassandra-dest
         settings:
           nodes: 127.0.0.1:9042 #{host}:{port}
           keyspace: company
           table: employees
     processors:
       proc1:
         type: parsejsonpayload #postgres creates raw data payload, but json formatted, so this processor will convert the raw data into structured. 

Build your Cassandra connector, then place the connector binary in the connectors directory relative to Conduit, check connectors for more details. Also, check Pipeline Configuration Files Docs for more details about how to run this pipeline.

Known Issues & Limitations

• Supports only structured data format for the key and payload. If your data is raw and JSON formatted, then you can use conduit's builtin processor parsejsonpayload or parsejsonkey to parse your json data into structured data.

Planned work

• Support raw data formats for keys and payloads.