transporter

module
v0.1.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 27, 2015 License: BSD-3-Clause

README

Circle CI

Compose helps with database transformations from one store to another. It can also sync from one to another or several stores.

Transporter

Build

make sure godep is installed, go get github.com/tools/godep and then build with godep restore godep go build -a ./cmd/...

Configure

There is a sample config in test/config.yaml. The config defines the endpoints, (either sources or sinks) that are available to the application.

api:
  interval: 60s # time interval between metrics posts to the api endpoint
  uri: "http://requestb.in/1a0zlf11"
nodes:
  localmongo:
    type: mongo
    uri: mongodb://localhost/boom
  supernick:
    type: elasticsearch
    uri: http://10.0.0.1,10.0.0.2:9200/indexname
  debug:
    type: file
    uri: stdout://
  foofile:
    type: file
    uri: file:///tmp/foo
  stdout:
    type: file
    uri: stdout://

There is also a sample 'application.js' in test/application.js. The application is responsible for building transporter pipelines. Given the above config, this Transporter application.js will copy from a file (in /tmp/foo) to stdout.

Source({name:"foofile"}).save({name:"stdout"})

This application.js will copy from the local mongo to a file on the local disk

Source({name:"localmongo", namespace: "boom.foo"}).save({name:"tofile"})

Transformers can also be configured in the application.js as follows

var pipeline = Source({name:"mongodb-production", namespace: "compose.milestones2"})
pipeline = pipeline.transform("transformers/transform1.js").transform("transformers/transform2.js")
pipeline.save({name:"supernick", namespace: "something/posts2"});

Run

  • list transporter list --config ./test/config.yaml
  • run transporter run --config ./test/config.yaml ./test/application.js
  • eval transporter eval --config ./test/config.yaml 'Source({name:"localmongo", namespace: "boom.foo"}).save({name:"tofile"})'
  • test transporter test --config ./test/config.yaml test/application.js

Complete beginners guide

OS X
  • ensure you have mercurial installed as it is required for a dependency
  • install the Mac OS X binary build from https://golang.org/dl/
  • follow instructions on http://golang.org/doc/install
  • VERY IMPORANT: Go has a required directory structure which the GOPATH needs to point to. Instructions can be found on http://golang.org/doc/code.html or by typing go help gopath in terminal.
  • setup the directory structure in $GOPATH
    • cd $GOPATH; mkdir src pkg bin
    • create the github.com path and compose mkdir -p src/github.com/compose; cd src/github.com/compose
    • clone transporter git clone https://github.com/compose/transporter; cd transporter
    • run go get to get all the dependencies go get -a ./cmd/...
    • now you can build go build -a ./cmd/...

At this point you should be able to run transporter via $GOPATH/bin/transporter, you may need to add $GOPATH to your PATH environment variable. Something along the lines of export PATH="$GOPATH/bin:$PATH" should work.

Windows

See READMEWINDOWS.md

Transporter in the Media

Contributing to Transporter

Want to help out with Transporter? Great! There are instructions to get you started here.

Licensing

Transporter is licensed under the New BSD License. See LICENSE for full license text.

Directories

Path Synopsis
Godeps
_workspace/src/github.com/bitly/go-hostpool
A Go package to intelligently and flexibly pool among multiple hosts from your Go application.
A Go package to intelligently and flexibly pool among multiple hosts from your Go application.
_workspace/src/github.com/cenkalti/backoff
Package backoff implements backoff algorithms for retrying operations.
Package backoff implements backoff algorithms for retrying operations.
_workspace/src/github.com/golang/protobuf/proto
Package proto converts data structures to and from the wire format of protocol buffers.
Package proto converts data structures to and from the wire format of protocol buffers.
_workspace/src/github.com/golang/protobuf/proto/proto3_proto
Package proto3_proto is a generated protocol buffer package.
Package proto3_proto is a generated protocol buffer package.
_workspace/src/github.com/nu7hatch/gouuid
This package provides immutable UUID structs and the functions NewV3, NewV4, NewV5 and Parse() for generating versions 3, 4 and 5 UUIDs as specified in RFC 4122.
This package provides immutable UUID structs and the functions NewV3, NewV4, NewV5 and Parse() for generating versions 3, 4 and 5 UUIDs as specified in RFC 4122.
_workspace/src/github.com/robertkrimen/otto
Package otto is a JavaScript parser and interpreter written natively in Go.
Package otto is a JavaScript parser and interpreter written natively in Go.
_workspace/src/github.com/robertkrimen/otto/ast
Package ast declares types representing a JavaScript AST.
Package ast declares types representing a JavaScript AST.
_workspace/src/github.com/robertkrimen/otto/dbg
Package dbg is a println/printf/log-debugging utility library.
Package dbg is a println/printf/log-debugging utility library.
_workspace/src/github.com/robertkrimen/otto/file
Package file encapsulates the file abstractions used by the ast & parser.
Package file encapsulates the file abstractions used by the ast & parser.
_workspace/src/github.com/robertkrimen/otto/parser
Package parser implements a parser for JavaScript.
Package parser implements a parser for JavaScript.
_workspace/src/github.com/robertkrimen/otto/registry
Package registry is an expirmental package to facillitate altering the otto runtime via import.
Package registry is an expirmental package to facillitate altering the otto runtime via import.
_workspace/src/github.com/robertkrimen/otto/terst
Package terst is a terse (terst = test + terse), easy-to-use testing library for Go.
Package terst is a terse (terst = test + terse), easy-to-use testing library for Go.
_workspace/src/github.com/robertkrimen/otto/token
Package token defines constants representing the lexical tokens of JavaScript (ECMA5).
Package token defines constants representing the lexical tokens of JavaScript (ECMA5).
_workspace/src/github.com/robertkrimen/otto/underscore
Package underscore contains the source for the JavaScript utility-belt library.
Package underscore contains the source for the JavaScript utility-belt library.
_workspace/src/gopkg.in/dancannon/gorethink.v1
Package gorethink implements a Go driver for RethinkDB Current version: v1.0.0 (RethinkDB v2.0) For more in depth information on how to use RethinkDB check out the API docs at http://rethinkdb.com/api
Package gorethink implements a Go driver for RethinkDB Current version: v1.0.0 (RethinkDB v2.0) For more in depth information on how to use RethinkDB check out the API docs at http://rethinkdb.com/api
_workspace/src/gopkg.in/mgo.v2
Package mgo offers a rich MongoDB driver for Go.
Package mgo offers a rich MongoDB driver for Go.
_workspace/src/gopkg.in/mgo.v2/bson
Package bson is an implementation of the BSON specification for Go: http://bsonspec.org It was created as part of the mgo MongoDB driver for Go, but is standalone and may be used on its own without the driver.
Package bson is an implementation of the BSON specification for Go: http://bsonspec.org It was created as part of the mgo MongoDB driver for Go, but is standalone and may be used on its own without the driver.
_workspace/src/gopkg.in/mgo.v2/internal/scram
Pacakage scram implements a SCRAM-{SHA-1,etc} client per RFC5802.
Pacakage scram implements a SCRAM-{SHA-1,etc} client per RFC5802.
_workspace/src/gopkg.in/mgo.v2/testserver
WARNING: This package was replaced by mgo.v2/dbtest.
WARNING: This package was replaced by mgo.v2/dbtest.
_workspace/src/gopkg.in/mgo.v2/txn
The txn package implements support for multi-document transactions.
The txn package implements support for multi-document transactions.
_workspace/src/gopkg.in/yaml.v2
Package yaml implements YAML support for the Go language.
Package yaml implements YAML support for the Go language.
cmd
pkg
message
Package message provides wrapper structs and helper methods to pipe actual database documents throughout transporter.
Package message provides wrapper structs and helper methods to pipe actual database documents throughout transporter.
pipe
Package pipe provides types to help manage transporter communication channels as well as event types.
Package pipe provides types to help manage transporter communication channels as well as event types.
transporter
Package transporter provides all adaptoremented functionality to move data through transporter.
Package transporter provides all adaptoremented functionality to move data through transporter.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL