README
¶
redshifter
redshifter is responsible for getting data from any source into AWS Redshift for data analysis.
Motivation
AWS Redshift is a columnar storage based data warehouse solution. It is optimized for analysis and business intelligence and has integration with many tools.
Features
Redshifter contains libraries for transferring data from various data sources like postgres, mixpanel to s3 and a library to copy data into redshift from s3.
The details of each of the libraries can be found in their respective READMEs.
Running
Mixpanel data can be exported to redshift using the following command. Note that all the options specified are required to run the script.
$ AWS_ACCESS_KEY_ID=<access_key_id> \
AWS_SECRET_ACCESS_KEY=<secret_access_key> \
AWS_REGION=<s3_bucket_region> \
godep go run mixpanel_to_redshift.go \
-redshifthost=<redshift_host_url> \
-redshiftport=<redshift_port> \
-redshiftuser=<redshift_username> \
-redshiftpassword=<redshift_password> \
-redshiftdatabase=<redshift_database_name> \
-redshifttable=<redshift_table>
-jsonpathsfile=<jsonpaths_file> \
-mixpanelevents=<mixpanel_events_csv> \
-exportdir=<s3_export_dir> \
-mixpanelapikey=<api_key> \
-mixpanelapisecret=<api_secret>
There are other optional flags that can be found using:
godep go run mixpanel_to_redshift.go --help
Testing
Tests for the repository can be run using $ make test
Documentation
¶
There is no documentation for this package.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Godeps
|
|
|
_workspace/src/github.com/Clever/pathio
Package pathio is a package that allows writing to and reading from different types of paths transparently.
|
Package pathio is a package that allows writing to and reading from different types of paths transparently. |
|
_workspace/src/github.com/lib/pq
Package pq is a pure Go Postgres driver for the database/sql package.
|
Package pq is a pure Go Postgres driver for the database/sql package. |
|
_workspace/src/github.com/lib/pq/listen_example
Below you will find a self-contained Go program which uses the LISTEN / NOTIFY mechanism to avoid polling the database while waiting for more work to arrive.
|
Below you will find a self-contained Go program which uses the LISTEN / NOTIFY mechanism to avoid polling the database while waiting for more work to arrive. |
|
_workspace/src/github.com/lib/pq/oid
Package oid contains OID constants as defined by the Postgres server.
|
Package oid contains OID constants as defined by the Postgres server. |
|
_workspace/src/github.com/mitchellh/goamz/aws
goamz - Go packages to interact with the Amazon Web Services.
|
goamz - Go packages to interact with the Amazon Web Services. |
|
_workspace/src/github.com/vaughan0/go-ini
Package ini provides functions for parsing INI configuration files.
|
Package ini provides functions for parsing INI configuration files. |