application-broker

command module
v0.7.4 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 6, 2016 License: Apache-2.0 Imports: 8 Imported by: 0

README

Build Status

Application Broker for Cloud Foundry

A service broker to provision an application stack - set of applications, services and user provided services.

This is a third version, which support copying full application stacks, binding same services to multiple apps in single stack and generating random passwords when copying source ups with e.g. $RANDOM16. Second version used REST API of CF to retrieve and add data. First implementation was based on cf/cli calls which was considered bad solution due to concurrency problems. Now we are using regular REST requests under the hood.

Idea behind

Cloud Foundry introduces two notions when talking about software running within it. Application (pushed and controlled by developers) and Service (spawned by so-called brokers and used by Applications). In general, the first one is a kind of web service. The second one, works as a resource (database for example) to be used on-demand by Application. Initially Cloud Foundry provides a set of Service offerings to choose from (postgresql, mongodb, NATS, etc...). Every single offering in internal marketplace has broker behind it. Broker manages every instance of service and handles its binding to Applications. No service exists without its corresponding broker.

To easily create service offerings without implementing separate broker you may want to use Application Broker. The only thing you need to do is to prepare reference app and register it in our broker. Then you will be able to spawn copies of your reference app and treat them like service instances. Offering will be also visible within CF marketplace. As simple as that.

References: Custom Services

Features

  • Publishing application stack in marketplace

Example stack

  • Cloning existing stacks of apps, services and user provided services if they create a DAG with single root

Example stack

  • Linking apps by adding user provided service as JSON containing url field with value http://.. App mentioned has to be in the same space as source app. Cloned user provided service contains updated url pointing cloned application. Convention for user provided services names linking apps is -ups.
cf cups app1-ups -p url

url> http://app1.<domain>
  • Replacing $RANDOM8, $RANDOM16, $RANDOM24 and $RANDOM32 in user provided services copied with random string of width 8, 16, 24 or 32 from characterset [A-Za-z0-9]. This can be used to generate new password for every stack spawned. Parts for replacement can be on different levels of JSON in UPS. Replacement is done when copying so original user provided service is using phrase with $.
{
    "credentials": {
     "password": "$RANDOM16" -> "hTn8X07zm8KRDKr1"
    },
    "label": "user-provided",
    "name": "random-ups",
    "syslog_drain_url": "",
    "tags": []
}

Usage

Application broker is regular long-living app that has to be pushed to CF. An example of properly filled manifest.yml shall look like this:

---
applications:
- name: application-broker
  memory: 256M
  instances: 1
  path: .
  buildpack: go_buildpack
  services:
    - application-broker-mongodb
  env:
    AUTH_USER: admin            #broker API is secured with these BasicAuth credentials
    AUTH_PASS: password
    CLIENT_ID: client           #broker communicates with CF using OAuth
    CLIENT_SECRET: clientSecret #when asking for token it uses these credentials
    TOKEN_URL: https://uaa.yourserver.com/token_key
    CF_API: http://api.yourserver.com
    VERSION: "0.5.8"

Application broker is using app dependency discoverer for application stack discovery. It needs url of that app in url field of app-dependency-discoverer-ups user provided service, as well as auth_user and auth_pass for basic authentication. For local run http://localhost:9998 is taken. Create user provided service with command:

cf cups app-dependency-discoverer-ups -p "{\"auth_pass\": \"<password>\", \"auth_user\": \"<user>\", \"url\": \"http://<hostname>.<domain>\" }"

When manifest.yml is ready and ups with required name is available in selected space, the following command can be issued:

$ cf push

Application broker serves a catalog of service offerings. It means that it is responsible for multiple entries in marketplace. Informations about services that broker controls (catalog) are held in mongodb database bound to ApplicationBroker. Initially catalog is empty. You must register at least one to benefit from using it. So, push an app that you want to place in marketplace and remember its guid (we will call it referenceAppGuid).

Next, you should register your app within Application Broker catalog using Catalog API. For example (as simple as possible):

curl -sL $APPLICATION_BROKER_ADDRESS/v2/catalog -X POST  \
    -u $AUTH_USER:$AUTH_PASS                             \
    -H "Content-Type: application/json"                  \
    -d '{
            "app" : {"metadata" : {"guid" : "<referenceAppGuid>"}},
            "id" : "<place random guid here>",
            "plans" : [{"id" : "<place random guid here>"}],
            "description" : "<describe your service briefly>",
            "name" : "<service exposed by your broker>"
        }'

or with further possible configuration of dependent components (hdfs service in application stack will only accept parameters hdfs.key1 and hdfs.key2 when provisioned):

curl -sL $APPLICATION_BROKER_ADDRESS/v2/catalog -X POST  \
    -u $AUTH_USER:$AUTH_PASS                             \
    -H "Content-Type: application/json"                  \
    -d '{
            "app" : {"metadata" : {"guid" : "<referenceAppGuid>"}},
            "id" : "<place random guid here>",
            "plans" : [{"id" : "<place random guid here>"}],
            "description" : "<describe your service briefly>",
            "name" : "<service exposed by your broker>",
            "configuration": [
                {
                    "service_name": "hdfs",
                    "parameters": ["key1", "key2"]
                }
            ]
        }'

If parameter name is unique, namespace based on service instance name is optional.

Now Application Broker has one service registered. When asked it responds with non-empty catalog. You can check by firing:

curl -sL $APPLICATION_BROKER_ADDRESS/v2/catalog -X GET -u $AUTH_USER:$AUTH_PASS

Next we need to inform CF that your Application Broker instance is in fact broker.

$ cf create-service-broker <brokerName> admin admin http://address.of.pushed.application.broker
$ cf enable-service-access <service exposed by your broker>

While running cf create-service-broker Cloud Controller make request to provided URL and saves catalog that Application Broker exposes.

Once the service broker is running as an app, registered as a service broker, and enabled for access by users, a user can use like any other service broker:

cf create-service <service exposed by your broker> Simple <instanceName>
cf bind-service my-app <instanceName>
cf restage my-app

The client application my-app would have a $VCAP_SERVICE service for <instanceName>. The credentials would include the url to access the backend service application.

Behind the scenes, when cf create-service was invoked the CLI asked the Cloud Controller for the new service instance. The Cloud Controller asked the Application Broker for a new service instance. The Application Broker deploys a new backend application as a copy of referenceApp.

If the backend application requires any services for itself, then those too are created and bound to the new backend application.

The backend application and its own dependency services are created into the same organization and space being used by the end user.

NATS

Application Broker uses NATS messagebus to emit events. For now, events are being sent on every service instance provisioning. Events are meant to inform users about correct or erroneous results of operation. To enable NATS for your broker use the environment variable named NATS_URL pointing to address your NATS is listening on. Additionally, you can specify topic Application Broker should talk on using NATS_SERVICE_CREATION_SUBJECT. By default it is service-creation.

Development

Prerequisites

Application broker is using [app-dependency-discoverer] (https://github.com/intel-data/app-dependency-discoverer) so you need to run it locally before using application-broker. You also need to specify url and credentials to it in app-dependency-discoverer-ups with content described earlier.

To locally develop this service broker, we encourage you to use lightweight reference stack that will push and start fast. Testing won't take too much time.

Additionally you will need mongodb instance. Install it by using package-manager your distro provides. For Ubuntu/Debian it will be: sudo apt-get install mongodb. Local Application Broker will connect to it on default port so no additional configuration is needed.

Running locally

Install project dependencies:

godep restore ./...

You can run the Application Broker locally via gin, which will automatically reload any file changes during development. Our application needs several environment variables to work properly. Ensure that they are exported before starting. Running following commands is sufficient to run the Application Broker correctly:

go get github.com/codegangsta/gin
export CF_API=http://api.yourserver.com
export TOKEN_URL=https://uaa.yourserver.com/oauth/token
export CLIENT_ID=client
export CLIENT_SECRET=clientSecret
export AUTH_USER=admin
export AUTH_PASS=password
gin -a 9999 -i run main.go

The broker, via gin, will be running on port 3000 by default.

Unit Testing

To write self-describing unit tests easily we adopted ginkgo framework. Running these is simple as:

go get github.com/onsi/ginkgo/ginkgo
export PATH=$PATH:$GOPATH/bin
#commands above need to be executed just once
ginkgo -r
IDE

We recommend using IntelliJ IDEA as IDE with golang plugin. To apply formatting automatically on every save you may use go-fmt with File Watcher plugin.

Tips

Golang tips

Developing golang apps requires you store all dependencies (Godeps) in separate directory. They shall be placed in source control.

godep save ./...

Command above places all dependencies from $GOPATH, your app uses, in Godeps and writes its versions to Godeps/Godeps.json file.

Limitations

Additionally, in special circumstances, some problems may occur when spawning new instance with dependencies. Imagine referenceApp with dependencyServiceInstance bound to it. It is possible to spawn copy of referenceApp to space that dependencyService is not enabled in. In such situation provision operation will end up with failure.

Documentation

Overview

Package main application-broker API

The purpose of this application is to easily expose Cloud Foundry applications to marketplace as service offerings without implementing separate brokers for each of them.

Version: 0.6.5

swagger:meta

Directories

Path Synopsis
Godeps
_workspace/src/github.com/boj/redistore
Package redistore is a session store backend for gorilla/sessions
Package redistore is a session store backend for gorilla/sessions
_workspace/src/github.com/cihub/seelog
Package seelog implements logging functionality with flexible dispatching, filtering, and formatting.
Package seelog implements logging functionality with flexible dispatching, filtering, and formatting.
_workspace/src/github.com/cloudfoundry-community/go-cfenv
Package cfenv provides information about the current app deployed on Cloud Foundry, including any bound service(s).
Package cfenv provides information about the current app deployed on Cloud Foundry, including any bound service(s).
_workspace/src/github.com/codegangsta/inject
Package inject provides utilities for mapping and injecting dependencies in various ways.
Package inject provides utilities for mapping and injecting dependencies in various ways.
_workspace/src/github.com/davecgh/go-spew/spew
Package spew implements a deep pretty printer for Go data structures to aid in debugging.
Package spew implements a deep pretty printer for Go data structures to aid in debugging.
_workspace/src/github.com/garyburd/redigo/internal/redistest
Package redistest contains utilities for writing Redigo tests.
Package redistest contains utilities for writing Redigo tests.
_workspace/src/github.com/garyburd/redigo/redis
Package redis is a client for the Redis database.
Package redis is a client for the Redis database.
_workspace/src/github.com/go-martini/martini
Package martini is a powerful package for quickly writing modular web applications/services in Golang.
Package martini is a powerful package for quickly writing modular web applications/services in Golang.
_workspace/src/github.com/gorilla/context
Package context stores values shared during a request lifetime.
Package context stores values shared during a request lifetime.
_workspace/src/github.com/gorilla/securecookie
Package gorilla/securecookie encodes and decodes authenticated and optionally encrypted cookie values.
Package gorilla/securecookie encodes and decodes authenticated and optionally encrypted cookie values.
_workspace/src/github.com/gorilla/sessions
Package gorilla/sessions provides cookie and filesystem sessions and infrastructure for custom session backends.
Package gorilla/sessions provides cookie and filesystem sessions and infrastructure for custom session backends.
_workspace/src/github.com/jarcoal/httpmock
HTTPmock provides tools for mocking HTTP responses.
HTTPmock provides tools for mocking HTTP responses.
_workspace/src/github.com/martini-contrib/sessions
Package sessions contains middleware for easy session management in Martini.
Package sessions contains middleware for easy session management in Martini.
_workspace/src/github.com/mitchellh/mapstructure
The mapstructure package exposes functionality to convert an abitrary map[string]interface{} into a native Go structure.
The mapstructure package exposes functionality to convert an abitrary map[string]interface{} into a native Go structure.
_workspace/src/github.com/nats-io/nats
A Go client for the NATS messaging system (https://nats.io).
A Go client for the NATS messaging system (https://nats.io).
_workspace/src/github.com/nu7hatch/gouuid
This package provides immutable UUID structs and the functions NewV3, NewV4, NewV5 and Parse() for generating versions 3, 4 and 5 UUIDs as specified in RFC 4122.
This package provides immutable UUID structs and the functions NewV3, NewV4, NewV5 and Parse() for generating versions 3, 4 and 5 UUIDs as specified in RFC 4122.
_workspace/src/github.com/onsi/ginkgo
Ginkgo is a BDD-style testing framework for Golang
Ginkgo is a BDD-style testing framework for Golang
_workspace/src/github.com/onsi/ginkgo/config
Ginkgo accepts a number of configuration options.
Ginkgo accepts a number of configuration options.
The Ginkgo CLI
_workspace/src/github.com/onsi/ginkgo/internal/remote
Aggregator is a reporter used by the Ginkgo CLI to aggregate and present parallel test output coherently as tests complete.
Aggregator is a reporter used by the Ginkgo CLI to aggregate and present parallel test output coherently as tests complete.
Ginkgo's Default Reporter
_workspace/src/github.com/onsi/gomega
Gomega is the Ginkgo BDD-style testing framework's preferred matcher library.
Gomega is the Ginkgo BDD-style testing framework's preferred matcher library.
_workspace/src/github.com/onsi/gomega/format
Gomega's format package pretty-prints objects.
Gomega's format package pretty-prints objects.
_workspace/src/github.com/onsi/gomega/gbytes
Package gbytes provides a buffer that supports incrementally detecting input.
Package gbytes provides a buffer that supports incrementally detecting input.
_workspace/src/github.com/onsi/gomega/gexec
Package gexec provides support for testing external processes.
Package gexec provides support for testing external processes.
_workspace/src/github.com/onsi/gomega/ghttp
Package ghttp supports testing HTTP clients by providing a test server (simply a thin wrapper around httptest's server) that supports registering multiple handlers.
Package ghttp supports testing HTTP clients by providing a test server (simply a thin wrapper around httptest's server) that supports registering multiple handlers.
_workspace/src/github.com/onsi/gomega/ghttp/protobuf
Package protobuf is a generated protocol buffer package.
Package protobuf is a generated protocol buffer package.
Gomega matchers
_workspace/src/github.com/pmezard/go-difflib/difflib
Package difflib is a partial port of Python difflib module.
Package difflib is a partial port of Python difflib module.
_workspace/src/github.com/stretchr/objx
objx - Go package for dealing with maps, slices, JSON and other data.
objx - Go package for dealing with maps, slices, JSON and other data.
_workspace/src/github.com/stretchr/testify/assert
Package assert provides a set of comprehensive testing tools for use with the normal Go testing system.
Package assert provides a set of comprehensive testing tools for use with the normal Go testing system.
_workspace/src/github.com/stretchr/testify/mock
Package mock provides a system by which it is possible to mock your objects and verify calls are happening as expected.
Package mock provides a system by which it is possible to mock your objects and verify calls are happening as expected.
_workspace/src/golang.org/x/net/context
Package context defines the Context type, which carries deadlines, cancelation signals, and other request-scoped values across API boundaries and between processes.
Package context defines the Context type, which carries deadlines, cancelation signals, and other request-scoped values across API boundaries and between processes.
_workspace/src/golang.org/x/net/context/ctxhttp
Package ctxhttp provides helper functions for performing context-aware HTTP requests.
Package ctxhttp provides helper functions for performing context-aware HTTP requests.
_workspace/src/golang.org/x/oauth2
Package oauth2 provides support for making OAuth2 authorized and authenticated HTTP requests.
Package oauth2 provides support for making OAuth2 authorized and authenticated HTTP requests.
_workspace/src/golang.org/x/oauth2/bitbucket
Package bitbucket provides constants for using OAuth2 to access Bitbucket.
Package bitbucket provides constants for using OAuth2 to access Bitbucket.
_workspace/src/golang.org/x/oauth2/clientcredentials
Package clientcredentials implements the OAuth2.0 "client credentials" token flow, also known as the "two-legged OAuth 2.0".
Package clientcredentials implements the OAuth2.0 "client credentials" token flow, also known as the "two-legged OAuth 2.0".
_workspace/src/golang.org/x/oauth2/facebook
Package facebook provides constants for using OAuth2 to access Facebook.
Package facebook provides constants for using OAuth2 to access Facebook.
_workspace/src/golang.org/x/oauth2/github
Package github provides constants for using OAuth2 to access Github.
Package github provides constants for using OAuth2 to access Github.
_workspace/src/golang.org/x/oauth2/google
Package google provides support for making OAuth2 authorized and authenticated HTTP requests to Google APIs.
Package google provides support for making OAuth2 authorized and authenticated HTTP requests to Google APIs.
_workspace/src/golang.org/x/oauth2/internal
Package internal contains support packages for oauth2 package.
Package internal contains support packages for oauth2 package.
_workspace/src/golang.org/x/oauth2/jws
Package jws provides encoding and decoding utilities for signed JWS messages.
Package jws provides encoding and decoding utilities for signed JWS messages.
_workspace/src/golang.org/x/oauth2/jwt
Package jwt implements the OAuth 2.0 JSON Web Token flow, commonly known as "two-legged OAuth 2.0".
Package jwt implements the OAuth 2.0 JSON Web Token flow, commonly known as "two-legged OAuth 2.0".
_workspace/src/golang.org/x/oauth2/linkedin
Package linkedin provides constants for using OAuth2 to access LinkedIn.
Package linkedin provides constants for using OAuth2 to access LinkedIn.
_workspace/src/golang.org/x/oauth2/odnoklassniki
Package odnoklassniki provides constants for using OAuth2 to access Odnoklassniki.
Package odnoklassniki provides constants for using OAuth2 to access Odnoklassniki.
_workspace/src/golang.org/x/oauth2/paypal
Package paypal provides constants for using OAuth2 to access PayPal.
Package paypal provides constants for using OAuth2 to access PayPal.
_workspace/src/golang.org/x/oauth2/vk
Package vk provides constants for using OAuth2 to access VK.com.
Package vk provides constants for using OAuth2 to access VK.com.
_workspace/src/gopkg.in/mgo.v2
Package mgo offers a rich MongoDB driver for Go.
Package mgo offers a rich MongoDB driver for Go.
_workspace/src/gopkg.in/mgo.v2/bson
Package bson is an implementation of the BSON specification for Go:
Package bson is an implementation of the BSON specification for Go:
_workspace/src/gopkg.in/mgo.v2/internal/scram
Pacakage scram implements a SCRAM-{SHA-1,etc} client per RFC5802.
Pacakage scram implements a SCRAM-{SHA-1,etc} client per RFC5802.
_workspace/src/gopkg.in/mgo.v2/testserver
WARNING: This package was replaced by mgo.v2/dbtest.
WARNING: This package was replaced by mgo.v2/dbtest.
_workspace/src/gopkg.in/mgo.v2/txn
The txn package implements support for multi-document transactions.
The txn package implements support for multi-document transactions.
Package messagebus handles emitting events to buses like NATS
Package messagebus handles emitting events to buses like NATS
*

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL