nycab

NYCAB Challenge

Summary

Cab Data Researcher is a company that provides insights on the open data about NY cab trips. Cab trips in NY are public available as csv downloadable files. In order to make it more useful we want to wrap the data in a public API.

Data format is as follows:

medallion, hack_license, vendor_id, rate_code, store_and_fwd_flag, pickup_datetime, dropoff_datetime, passenger_count, trip_time_in_secs, trip_distance

• The medallion is cab identification.
• The API should provide a way to query how many trips a particular cab (medallion) has made given a particular pickup date ( using ​pickup_datetime​). Only consider the date and not the time.
• The API must receive one or more medallions and return how many trips each medallion has made.
• Considering that the query creates a heavy load on the database, the results must be cached.
• The API must allow user to ask for fresh data, ignoring the cache. There must be also be a method to clear the cache.

Primary feature

1. Support RESTful API with OpenAPI(Swagger).
2. Support Redis cache and in-process cache to accellerate fetching data.
3. API support enforce getting data from database(with refresh argument).
4. API support cache clean manually.
5. Support interactive command line tool to acess API service.
6. Support independent RPC service to provide access to database.

About Infrastructure

This challenge implement 3 main components:

1. cli.nycab (Commandline tool)
2. api.nycab (Restful API service)
3. service.nycab (GRPC service)

cli.nycab raises requests to api.nycab and retrieve json responses.

api.nycab handles these requests and decides whether to get data from cache or original database.

If cache data is needed, api.nycab will try to connect with a real cache service, which is either an in-process memory service or a Redis service.

If database is needed, api.nycab will get data through service.nycab GRPC connection.

In general, API service have public IP address so it can be accessed by any client in the Internet. GRPC services and its related database stay in internal network for security reason.

Components and their relation can be seen below.

About project file structure

This project structure is according to Go project convention (https://github.com/golang-standards/project-layout). The detail of the project is given below.

Run the whole stack

1. Change directory to the root of the project.
2. Start MySQL and Redis service.

make run-infra

This will start redis service and MySQL service with "nycab" database.
After finishing using the infrastructure, remember to stop them.

make stop-infra

1. Import sql file to database.

make load-data

Script code will extract sql file from "test/testdata/db/ny_cab_data_cab_trip_data_full.sql.zip".
And then load into "nycab" database.

1. At last, run each command in sequence in a seperate terminal.

make run-service
make run-api
make run-cli

How to use cli.nycab Command line

After running cli.nycab, type "help" for more details.

> help
count
        Description:    count medallions pickup in a day 
        Example:        count medallion=medallion1,medallion2,medallion... date=2013-01-12 refresh=true
cache
        Description:    remove all cache
        Example:        cache clean
quit
        Description:    quit current application
        Example:        quit

RESTful API request and response format

The description of this part can be seen through api/swagger/nycab.yaml.
This yaml file is well documented and easy to view.
A better way to view this document is to use swagger editor(https://editor.swagger.io/).
You can import nycab.yaml to this website.
The visualization of this yaml file will help you understand all the APIs and their request and response.

Run Basic test

make test