gateway

cassandra-http-gateway

Conveniently assemble read-only HTTP gateway services for Apache Cassandra.

This framework acts as a query builder for CQL SELECT statements. The resulting query objects bind request parameters to attributes in the database, process HTTP requests, and return JSON-serialized results in a response. It can be used to publish read-only dataset access to internal clients.

It is -by design- very minimalist in nature. No validation of queries are performed, and JSON serialization is handled by Cassandra (ala SELECT JSON ...). Results are returned as an array of zero or more objects (rows).

Rationale

It has become increasingly common to have datasets that are materialized -either via batch analytics, and/or streaming events- and persisted for low-latency access. At the WMF, we typically persist these datasets to a multi-tenant Cassandra cluster (though use of other clusters & stores is possible). What is for us a half-dozen or so use-cases (at the time of writing), is being projected to grow to many 10s or 100s in the near future. At this scale, and with datasets from disparate stakeholders, this will likely become unmanageable.

An HTTP data gateway is one way of decoupling client applications from the underlying database. There are many potential benefits from this -for example- preserving the future ability to transparently migrate clusters or redistribute datasets, and to better manage resource utilization (rate limiting etc). Even something as mundane as a fleet-wide upgrade of native drivers could be prohibitively expensive when code owners are many, varied in resources and priorities, or worse when code becomes orphaned entirely.

Issues

• There is (currently) no support for streaming; Results are buffered in memory before being returned. Make sure you are not expecting a result set large enough for this to be an issue.
• Because there is no validation of input, obvious client errors will manifest as query failures. For example, if a client were to pass an alphanumeric value in a parameter bound to an integer field in the database, the framework will happily pass this on to Cassandra, resulting in a status 500 (instead of the expected 400).

See also: T294468

Usage

With a dataset like...

CREATE KEYSPACE music WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 1};
CREATE TABLE music.tracks (artist text, album text, track text, release_date timestamp, PRIMARY KEY(artist, album, track));

INSERT INTO music.tracks (artist, album, track, release_date) VALUES ('Van Halen', '5150', 'Good Enough', '1986-03-24');
INSERT INTO music.tracks (artist, album, track, release_date) VALUES ('Van Halen', '5150', 'Why Can''t This Be Love', '1986-03-24');
INSERT INTO music.tracks (artist, album, track, release_date) VALUES ('Van Halen', '5150', 'Get Up', '1986-03-24');
INSERT INTO music.tracks (artist, album, track, release_date) VALUES ('Van Halen', '5150', 'Dreams', '1986-03-24');
INSERT INTO music.tracks (artist, album, track, release_date) VALUES ('Van Halen', '5150', 'Summer Nights', '1986-03-24');
INSERT INTO music.tracks (artist, album, track, release_date) VALUES ('Van Halen', '5150', 'Best of Both Worlds', '1986-03-24');
INSERT INTO music.tracks (artist, album, track, release_date) VALUES ('Van Halen', '5150', 'Love Walks In', '1986-03-24');
INSERT INTO music.tracks (artist, album, track, release_date) VALUES ('Van Halen', '5150', '"5150"', '1986-03-24');
INSERT INTO music.tracks (artist, album, track, release_date) VALUES ('Van Halen', '5150', 'Inside', '1986-03-24');

And code that looks like...

package main

import (
    "fmt"
    "os"

    "github.com/gocql/gocql"
    "github.com/julienschmidt/httprouter"
)

func main() {
    var cluster *gocql.ClusterConfig
    var err error
    var session *gocql.Session

    cluster = gocql.NewCluster("localhost")
    cluster.Consistency = gocql.One

    if session, err = cluster.CreateSession(); err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }

    var router = new httprouter.New()
    router.GET("/music/:artist/:album", func(w http.ResponseWriter, r *http.Request, ps httprouter.Params) {
        SelectBuilder.
            From("music", "tracks").
            Session(session).
            Bind(ps).
            Build().
            Handle(w, r)
    })
}

You get a response like...

$ http http://localhost:8080/music/Van%20Halen/5150
HTTP/1.1 200 OK
Content-Length: 987
Content-Type: application/json
Date: Sat, 02 Apr 2022 01:25:04 GMT

{
    "rows": [
        {
            "album": "5150",
            "artist": "Van Halen",
            "release_date": "1986-03-24 00:00:00.000Z",
            "track": "\"5150\""
        },
        {
            "album": "5150",
            "artist": "Van Halen",
            "release_date": "1986-03-24 00:00:00.000Z",
            "track": "Best of Both Worlds"
        },
        {
            "album": "5150",
            "artist": "Van Halen",
            "release_date": "1986-03-24 00:00:00.000Z",
            "track": "Dreams"
        },
        {
            "album": "5150",
            "artist": "Van Halen",
            "release_date": "1986-03-24 00:00:00.000Z",
            "track": "Get Up"
        },
        {
            "album": "5150",
            "artist": "Van Halen",
            "release_date": "1986-03-24 00:00:00.000Z",
            "track": "Good Enough"
        },
        {
            "album": "5150",
            "artist": "Van Halen",
            "release_date": "1986-03-24 00:00:00.000Z",
            "track": "Inside"
        },
        {
            "album": "5150",
            "artist": "Van Halen",
            "release_date": "1986-03-24 00:00:00.000Z",
            "track": "Love Walks In"
        },
        {
            "album": "5150",
            "artist": "Van Halen",
            "release_date": "1986-03-24 00:00:00.000Z",
            "track": "Summer Nights"
        },
        {
            "album": "5150",
            "artist": "Van Halen",
            "release_date": "1986-03-24 00:00:00.000Z",
            "track": "Why Can't This Be Love"
        }
    ]
}


$