README
¶
NOTE: This project is no longer being updated publicly.
Blueprint
The schema server for the Twitch Science Data Pipeline.
See also the Schema Suggestor.
What it does
This permits you to create "schemas". A schema is essentially a confluence of two related concepts:
- A set of instructions for how to convert inbound data to the desired outbound data
- The table structure that the data will be inserted into
"Outbound data" and "table structure" are essentially the same in our world view.
Components
- An angularjs frontend
- An API
- A postgres db storing schema state
The frontend works with the API to create schemas in bpdb, the ingesters handle the creation of those tables later.
Building
apt-get -u install build-essential libgeoip-dev libgeoip1
go get github.com/tools/godep
./build.sh blueprint $GIT_BRANCH $BASEAMI $SECURITY_GROUP false
The arguments to build.sh are position, each position is as follows:
project: The project name, in this case "blueprint"branch: The name of your branchsource_ami: AMI to use as the base for resultant AMIsecurity_group: Security Group ID to useuse_private_ip: SSH to private IP (default: false)
Packer expects there to be two environment variables available:
AWS_ACCESS_KEYAWS_SECRET_KEY
Setup your security group to permit access from your IP address.
Set the use_private_ip to convey to packer whether to SSH to the
public IP address of the intermediate machine or the private one.
Either way your security group needs to allow access to port 22.
Maintenance mode
Members of the admin team on GitHub (number specified by the command-line flag
-adminTeam) can take Blueprint in and out of maintenance mode, during which
no modifications to the database are possible.
Caveat: After toggling maintenance mode, you will have to reload to see the relevant UI changes, but the backend is locked down regardless.
Angular Tests
First, install nodejs to so you can get npm as a side effect. Then you should be able to execute
./run_angular_tests.sh and that should take care of installing dependencies for you and running
all available Jasmine unit tests in the static/app/ folder with
Karma.
Documentation
¶
Overview ¶
Package blueprint provides an API server and UI for editing the Spade processing rules. Each schema defines processing rules for a particular event type, and results in that event being ingested by rs_ingester into Redshift. Processing rules transform one or more input fields to a TSV-compatible string. Systems can use the read- only endpoints to fetch the schema data without authentication. It listens on port 8080 for unauthenticated HTTP requests and on port 8081 for authenticated HTTP requests (assuming an Elastic Load Balancer already performed SSL termination).
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
_vendor
|
|
|
github.com/aws/aws-sdk-go/aws
Package aws provides core functionality for making requests to AWS services.
|
Package aws provides core functionality for making requests to AWS services. |
|
github.com/aws/aws-sdk-go/aws/awserr
Package awserr represents API error interface accessors for the SDK.
|
Package awserr represents API error interface accessors for the SDK. |
|
github.com/aws/aws-sdk-go/aws/credentials
Package credentials provides credential retrieval and management The Credentials is the primary method of getting access to and managing credentials Values.
|
Package credentials provides credential retrieval and management The Credentials is the primary method of getting access to and managing credentials Values. |
|
github.com/aws/aws-sdk-go/aws/credentials/endpointcreds
Package endpointcreds provides support for retrieving credentials from an arbitrary HTTP endpoint.
|
Package endpointcreds provides support for retrieving credentials from an arbitrary HTTP endpoint. |
|
github.com/aws/aws-sdk-go/aws/credentials/stscreds
Package stscreds are credential Providers to retrieve STS AWS credentials.
|
Package stscreds are credential Providers to retrieve STS AWS credentials. |
|
github.com/aws/aws-sdk-go/aws/defaults
Package defaults is a collection of helpers to retrieve the SDK's default configuration and handlers.
|
Package defaults is a collection of helpers to retrieve the SDK's default configuration and handlers. |
|
github.com/aws/aws-sdk-go/aws/ec2metadata
Package ec2metadata provides the client for making API calls to the EC2 Metadata service.
|
Package ec2metadata provides the client for making API calls to the EC2 Metadata service. |
|
github.com/aws/aws-sdk-go/aws/endpoints
Package endpoints provides the types and functionality for defining regions and endpoints, as well as querying those definitions.
|
Package endpoints provides the types and functionality for defining regions and endpoints, as well as querying those definitions. |
|
github.com/aws/aws-sdk-go/aws/session
Package session provides configuration for the SDK's service clients.
|
Package session provides configuration for the SDK's service clients. |
|
github.com/aws/aws-sdk-go/aws/signer/v4
Package v4 implements signing for AWS V4 signer Provides request signing for request that need to be signed with AWS V4 Signatures.
|
Package v4 implements signing for AWS V4 signer Provides request signing for request that need to be signed with AWS V4 Signatures. |
|
github.com/aws/aws-sdk-go/private/endpoints
Package endpoints validates regional endpoints for services.
|
Package endpoints validates regional endpoints for services. |
|
github.com/aws/aws-sdk-go/private/protocol/query
Package query provides serialization of AWS query requests, and responses.
|
Package query provides serialization of AWS query requests, and responses. |
|
github.com/aws/aws-sdk-go/private/protocol/rest
Package rest provides RESTful serialization of AWS requests and responses.
|
Package rest provides RESTful serialization of AWS requests and responses. |
|
github.com/aws/aws-sdk-go/private/protocol/restxml
Package restxml provides RESTful XML serialization of AWS requests and responses.
|
Package restxml provides RESTful XML serialization of AWS requests and responses. |
|
github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil
Package xmlutil provides XML serialization of AWS requests and responses.
|
Package xmlutil provides XML serialization of AWS requests and responses. |
|
github.com/aws/aws-sdk-go/private/signer/v4
Package v4 implements signing for AWS V4 signer
|
Package v4 implements signing for AWS V4 signer |
|
github.com/aws/aws-sdk-go/service/s3
Package s3 provides a client for Amazon Simple Storage Service.
|
Package s3 provides a client for Amazon Simple Storage Service. |
|
github.com/aws/aws-sdk-go/service/s3/s3iface
Package s3iface provides an interface to enable mocking the Amazon Simple Storage Service service client for testing your code.
|
Package s3iface provides an interface to enable mocking the Amazon Simple Storage Service service client for testing your code. |
|
github.com/aws/aws-sdk-go/service/s3/s3manager
Package s3manager provides utilities to upload and download objects from S3 concurrently.
|
Package s3manager provides utilities to upload and download objects from S3 concurrently. |
|
github.com/aws/aws-sdk-go/service/s3/s3manager/s3manageriface
Package s3manageriface provides an interface for the s3manager package
|
Package s3manageriface provides an interface for the s3manager package |
|
github.com/aws/aws-sdk-go/service/sqs
Package sqs provides a client for Amazon Simple Queue Service.
|
Package sqs provides a client for Amazon Simple Queue Service. |
|
github.com/aws/aws-sdk-go/service/sqs/sqsiface
Package sqsiface provides an interface to enable mocking the Amazon Simple Queue Service service client for testing your code.
|
Package sqsiface provides an interface to enable mocking the Amazon Simple Queue Service service client for testing your code. |
|
github.com/aws/aws-sdk-go/service/sts
Package sts provides a client for AWS Security Token Service.
|
Package sts provides a client for AWS Security Token Service. |
|
github.com/davecgh/go-spew/spew
Package spew implements a deep pretty printer for Go data structures to aid in debugging.
|
Package spew implements a deep pretty printer for Go data structures to aid in debugging. |
|
github.com/go-ini/ini
Package ini provides INI file read and write functionality in Go.
|
Package ini provides INI file read and write functionality in Go. |
|
github.com/goji/param
Package param deserializes parameter values into a given struct using magical reflection ponies.
|
Package param deserializes parameter values into a given struct using magical reflection ponies. |
|
github.com/gorilla/context
Package context stores values shared during a request lifetime.
|
Package context stores values shared during a request lifetime. |
|
github.com/gorilla/securecookie
Package gorilla/securecookie encodes and decodes authenticated and optionally encrypted cookie values.
|
Package gorilla/securecookie encodes and decodes authenticated and optionally encrypted cookie values. |
|
github.com/gorilla/sessions
Package gorilla/sessions provides cookie and filesystem sessions and infrastructure for custom session backends.
|
Package gorilla/sessions provides cookie and filesystem sessions and infrastructure for custom session backends. |
|
github.com/lib/pq
Package pq is a pure Go Postgres driver for the database/sql package.
|
Package pq is a pure Go Postgres driver for the database/sql package. |
|
github.com/lib/pq/oid
Package oid contains OID constants as defined by the Postgres server.
|
Package oid contains OID constants as defined by the Postgres server. |
|
github.com/pmezard/go-difflib/difflib
Package difflib is a partial port of Python difflib module.
|
Package difflib is a partial port of Python difflib module. |
|
github.com/sirupsen/logrus
Package logrus is a structured logger for Go, completely API compatible with the standard library logger.
|
Package logrus is a structured logger for Go, completely API compatible with the standard library logger. |
|
github.com/stretchr/testify/assert
Package assert provides a set of comprehensive testing tools for use with the normal Go testing system.
|
Package assert provides a set of comprehensive testing tools for use with the normal Go testing system. |
|
github.com/stretchr/testify/require
Package require implements the same assertions as the `assert` package but stops test execution when a test fails.
|
Package require implements the same assertions as the `assert` package but stops test execution when a test fails. |
|
github.com/twitchscience/aws_utils/logger
Package logger is a wrapper around logrus that logs in a structured JSON format and provides additional context keys.
|
Package logger is a wrapper around logrus that logs in a structured JSON format and provides additional context keys. |
|
github.com/zenazn/goji
Package goji provides an out-of-box web server with reasonable defaults.
|
Package goji provides an out-of-box web server with reasonable defaults. |
|
github.com/zenazn/goji/bind
Package bind provides a convenient way to bind to sockets.
|
Package bind provides a convenient way to bind to sockets. |
|
github.com/zenazn/goji/graceful
Package graceful implements graceful shutdown for HTTP servers by closing idle connections after receiving a signal.
|
Package graceful implements graceful shutdown for HTTP servers by closing idle connections after receiving a signal. |
|
github.com/zenazn/goji/graceful/listener
Package listener provides a way to incorporate graceful shutdown to any net.Listener.
|
Package listener provides a way to incorporate graceful shutdown to any net.Listener. |
|
github.com/zenazn/goji/web
Package web provides a fast and flexible middleware stack and mux.
|
Package web provides a fast and flexible middleware stack and mux. |
|
github.com/zenazn/goji/web/middleware
Package middleware provides several standard middleware implementations.
|
Package middleware provides several standard middleware implementations. |
|
github.com/zenazn/goji/web/mutil
Package mutil contains various functions that are helpful when writing http middleware.
|
Package mutil contains various functions that are helpful when writing http middleware. |
|
golang.org/x/net/context
Package context defines the Context type, which carries deadlines, cancelation signals, and other request-scoped values across API boundaries and between processes.
|
Package context defines the Context type, which carries deadlines, cancelation signals, and other request-scoped values across API boundaries and between processes. |
|
golang.org/x/net/context/ctxhttp
Package ctxhttp provides helper functions for performing context-aware HTTP requests.
|
Package ctxhttp provides helper functions for performing context-aware HTTP requests. |
|
golang.org/x/oauth2
Package oauth2 provides support for making OAuth2 authorized and authenticated HTTP requests.
|
Package oauth2 provides support for making OAuth2 authorized and authenticated HTTP requests. |
|
golang.org/x/oauth2/internal
Package internal contains support packages for oauth2 package.
|
Package internal contains support packages for oauth2 package. |
|
golang.org/x/sys/unix
Package unix contains an interface to the low-level operating system primitives.
|
Package unix contains an interface to the low-level operating system primitives. |
|
Package ingester provides an interface to the ingester's control endpoints.
|
Package ingester provides an interface to the ingester's control endpoints. |
|
processor
Package processor contains logic to process gzipped Mixpanel event data into SQL table schemas.
|
Package processor contains logic to process gzipped Mixpanel event data into SQL table schemas. |
Click to show internal directories.
Click to hide internal directories.