rovers
rovers is a service to retrieve repository URLs from multiple repository
hosting providers.
Quick start using docker images
Download docker images
Get the latest rovers image:
docker pull srcd/rovers
Start everything using docker-compose
Install docker-compose.
Start Rabbit and Postgres
docker-compose -d up rovers-postgres rovers-rabbitmq
Export as environment variables the API keys(see section).Then, you can execute rovers:
docker-compose up --no-deps rovers
If you need run just some of the availables supported-providers, you can use this command replacing the flag --provider
with those providers you want to use:
docker-compose run --rm --no-deps --service-ports rovers /bin/sh -c "rovers initdb; rovers repos --provider=github --provider=bitbucket"
After that, rovers will generate a lot of 'mentions' (git repositories found on the internet), and sending them to the 'rovers' queue in Rabbit.
To stop and remove all the containers running docker-compose down
Finally, you can use Borges to fetch the content of these repositories.
Supported Providers
All the supported providers are used by default. In case you need run only some of them you must use the --provider
flag:
rovers repos --provider=github --provider=bitbucket
Generate needed API keys for the providers
To be able to fetch github and cgit repositories, you should create several API keys:
GitHub
Uses the GitHub API to get new repositories. Requires a GitHub API token. You can set the token through the environment variable:
$ export CONFIG_GITHUB_TOKEN=github-token
Bitbucket
Uses the Bitbucket API to get new repositories as an anonymous user.
Note: the first time this provider runs it may take a while to find git repositories, as the vast majority of repositories at the beginning of the API results are mercurial repositories.
cgit
cgit is a popular service to create your own self-hosted Git repository provider.
Rovers uses Bing search to discover cgit instances online and tracks them to get
new repositories. Requires a Bing API key. You can set the key through the environment variable:
$ export CONFIG_BING_KEY=bing-api-key
Installation
go get -u github.com/src-d/rovers/...
Usage
Run rovers --help
to get help about the supported commands and their options.
To initialize the database schemas. You need to run this command only once.
rovers initdb
To start collecting repository URLs
rovers repos --provider=github
You can configure rovers by environment variables:
Providers:
CONFIG_GITHUB_TOKEN
to set the github api key.
CONFIG_BING_KEY
to set the cgit api key.
Broker:
CONFIG_BROKER_URL
to set the broker url, by default amqp://guest:guest@localhost:5672
Database:
CONFIG_DBUSER
: database username,by default if not set testing
CONFIG_DBPASS
: database user password,by default if not set testing
CONFIG_DBHOST
: database host,by default if not set 0.0.0.0
CONFIG_DBPORT
: database port,by default if not set 5432
CONFIG_DBNAME
: database name,by default if not set testing
CONFIG_DBSSLMODE
: ssl mode to use,by default if not set disable
CONFIG_DBTIMEOUT
: connection timeout,by default if not set 30s
CONFIG_DBAPPNAME
: application name
Development
Build
rm Makefile.main; rm -rf .ci
to make sure you will have the last Makefile changes.
make dependencies
to download vendor dependencies.
make packages
to generate binaries for several platforms.
You will find the built binaries under ./build
.
Test
This service uses PostgreSQL and RabbitMQ.
To execute test locally you need to run RabbitMQ and PostgreSQL.
docker-compose up -d rovers-postgres rovers-rabbitmq
make test
Running Rovers in Kubernetes
You can use the official Helm chart to deploy Rovers in your kubernetes cluster.