gazebosim-web-backend

Gazebosim Web Backend Server

Development and Code style

See https://github.com/golang/go/wiki/CodeReviewComments

The main() and the package init() functions are in application.go. If you are starting to read this code, probably you will want to start from there.

Install

1. Go version 1.14 or above.

2. Download the server code

   git clone https://github.com/gazebo-web/gazebosim-web-backend
   

3. Build and download dependencies

   cd gazebosim-web-backend
   go build
   

4. Install mysql:

   NOTE: Install a version greater than v5.6.4. In the servers, we are currently using MySQL v5.7.21

   sudo apt-get install default-mysql-server
   

   The installer will ask you to create a root password for mysql.

5. Create the database and a user in mysql. Replace 'newuser' with your username and 'password' with your new password:

    # Xenial
    mysql -u root -p
    # Bionic requires sudo
    sudo mysql -u root -p
   

   CREATE DATABASE gazebo;
   

   Also create a separate database to use with tests:

   CREATE DATABASE gazebo_test;
   

   CREATE USER 'newuser'@'localhost' IDENTIFIED BY 'password';
   

   GRANT ALL PRIVILEGES ON gazebo.* TO 'newuser'@'localhost';
   

   GRANT ALL PRIVILEGES ON gazebo_test.* TO 'newuser'@'localhost';
   

   FLUSH PRIVILEGES;
   

   exit
   

6. Create a .env file in the root of the gazebosim-web-backend directory with the following environment variables.

export GZ_DB_USERNAME=<YOUR_MYSQL_USERNAME>
export GZ_DB_PASSWORD=<YOUR_MYSQL_PASSWORD>
export GZ_DB_ADDRESS=localhost:3306
export GZ_DB_NAME=gazebo
export GZ_DB_MAX_OPEN_CONNECTIONS=44

# This is needed when updating library versions
export GZ_VERSION_PASSWORD=<THE_GAZEBO_VERSION_PASSWORD>

# These are needed only for deployment
export AWS_ACCESS_KEY_ID=<YOUR_AWS_ACCESS_KEY>
export AWS_SECRET_ACCESS_KEY=<YOUR_AWS_SECRET_KEY>
export AWS_INSTANCE_NAME_PREFIX=<AWS_INSTANCE_PREFIX>
export AWS_REGION=us-east-1

# This is for authentication
export AUTH0_RSA256_PUBLIC_KEY=<AUTH0_PUBLIC_KEY>

export GZ_HTTP_PORT=8001
export GZ_SSL_PORT=4431
export GZ_LOGGER_LOG_STDOUT=true

1. Run the server

./gazebosim-web-backend

1. Try the libs route using

curl http://localhost:8000/1.0/libs

Test

1. Create a Test JWT token (this is needed for tests to pass OK -- go test)

   TL;DR: Just copy and paste the following env vars in your system (.env)

    # Test RSA256 Private key WITHOUT the -----BEGIN RSA PRIVATE KEY----- and -----END RSA PRIVATE KEY-----
    # It is used by token-generator to generate the Test JWT Token
    export TOKEN_GENERATOR_PRIVATE_RSA256_KEY=MIICWwIBAAKBgQDdlatRjRjogo3WojgGHFHYLugdUWAY9iR3fy4arWNA1Cos8kVw33cJibXr8bvwUAUparCwlvdbH6dvEOfou0/gCFQsHUfQrSDv+MuSUMAe8jzKE4qW+jK+xQU9a03GUnKHkkle+Q0pX/g6jXZ7r1/xAK5Do2kQ+X5xK9cipRgEKwIDAQABAoGAD+onAtVye4ic7VR7V50DF9bOnwRwNXrARcDhq9LWNRrRGElESYYTQ6EbatXS3MCyjjX2eMhu/aF5YhXBwkppwxg+EOmXeh+MzL7Zh284OuPbkglAaGhV9bb6/5CpuGb1esyPbYW+Ty2PC0GSZfIXkXs76jXAu9TOBvD0ybc2YlkCQQDywg2R/7t3Q2OE2+yo382CLJdrlSLVROWKwb4tb2PjhY4XAwV8d1vy0RenxTB+K5Mu57uVSTHtrMK0GAtFr833AkEA6avx20OHo61Yela/4k5kQDtjEf1N0LfI+BcWZtxsS3jDM3i1Hp0KSu5rsCPb8acJo5RO26gGVrfAsDcIXKC+bQJAZZ2XIpsitLyPpuiMOvBbzPavd4gY6Z8KWrfYzJoI/Q9FuBo6rKwl4BFoToD7WIUS+hpkagwWiz+6zLoX1dbOZwJACmH5fSSjAkLRi54PKJ8TFUeOP15h9sQzydI8zJU+upvDEKZsZc/UhT/SySDOxQ4G/523Y0sz/OZtSWcol/UMgQJALesy++GdvoIDLfJX5GBQpuFgFenRiRDabxrE9MNUZ2aPFaFp+DyAe+b4nDwuJaW2LURbr8AEZga7oQj0uYxcYw==
   
    # JWT Token generated by the token-generator program using the above Test RSA keys
    # This token does not expire.
    export GZ_TEST_JWT=eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ0ZXN0LXVzZXItaWRlbnRpdHkifQ.iV59-kBkZ86XKKsph8fxEeyxDiswY1zvPGi4977cHbbDEkMA3Y3t_zzmwU4JEmjbTeToQZ_qFNJGGNufK2guLy0SAicwjDmv-3dHDfJUH5x1vfi1fZFnmX_b8215BNbCBZU0T2a9DEFypxAQCQyiAQDE9gS8anFLHHlbcWdJdGw
   
    # A Test RSA256 Public key, without the -----BEGIN CERTIFICATE----- and -----END CERTIFICATE-----.
    # It is used to override the AUTH0_RSA256_PUBLIC_KEY when tests are run.
    export TEST_RSA256_PUBLIC_KEY=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDdlatRjRjogo3WojgGHFHYLugdUWAY9iR3fy4arWNA1KoS8kVw33cJibXr8bvwUAUparCwlvdbH6dvEOfou0/gCFQsHUfQrSDv+MuSUMAe8jzKE4qW+jK+xQU9a03GUnKHkkle+Q0pX/g6jXZ7r1/xAK5Do2kQ+X5xK9cipRgEKwIDAQAB
   

   In summary, in order to make go test work with JWT you will need to set the following env vars:

   • TOKEN_GENERATOR_PRIVATE_RSA256_KEY
   • TEST_RSA256_PUBLIC_KEY
   • GZ_TEST_JWT

2. Run the test suite

   First, make sure you have all the required env variables set. Below you can see a list of required env vars. But there could be other required vars too. To be completely sure, see the Env Variables section.

   export GZ_DB_USERNAME=<DB username>
   

   export GZ_DB_PASSWORD=<DB password>
   

   export GZ_DB_ADDRESS=<DB IP and port. Eg: localhost:3306>
   

   export GZ_DB_NAME=gazebo
   

   export GZ_DB_MAX_OPEN_CONNS=66
   

   export GZ_TEST_JWT=<JWT token>
   

   export TEST_RSA256_PUBLIC_KEY=< RSA256 public key without the '-----BEGIN CERTIFICATE-----' and '-----END CERTIFICATE-----'>
   Note: TEST_RSA256_PUBLIC_KEY must be able to decode and validate the GZ_TEST_JWT test token.
   

   export GZ_DB_LOG=false
   

   export GZ_LOGGER_LOG_STDOUT=true
   

   Then, run all tests:

   go test github.com/gazebo-web/gazebosim-web-backend
   

3. Run the backend server

   First, make sure to set the AUTH0_RSA256_PUBLIC_KEY environment variable with the Auth0 RSA256 public key. This env var will be used by the backend to decode and validate any received Auth0 JWT tokens. Note: You can get this key from: https://osrf.auth0.com/.well-known/jwks.json (or from your own auth0 user). Open that url in the browser and copy the value of the x5c field.

   $GOPATH/bin/ign-webserver
   

Environment Variables

You may want to create an .env file to define environment vars. Remember to add it to .gitignore. This .env will be automatically loaded by the server.

Linter

1. Get the linter

   cd ~/go_ws
   

   curl -L https://git.io/vp6lP | sh
   

2. Run the linter

   ./bin/gometalinter $(go list github.com/gazebo-web/gazebosim-web-backend/...)
   

Note you can create this bash script:

#!/bin/bash
curl -L https://git.io/vp6lP | sh -s -- -b $GOPATH/bin
$GOPATH/bin/gometalinter $(go list github.com/gazebo-web/gazebosim-web-backend/...)

Deployment

The staging branch in this repository is used to deploy this website to https://staging-api.gazebosim.org. The production branch in this repository is used to deploy this website to https://api.gazebosim.org.

Github actions will automatically deploy staging on push. The production branch will only deploy when an authorized user approves the deployment on the Github Actions UI.

There is no rule about how a release should be made. A person with sufficient access can choose between direct commits or pull requests.

Development

Debugging inside docker container

If you ever need to debug the application as if it were running in AWS or the pipelines, you need to do it from inside its docker containter. To do that:

Most ideas taken from here: Mysql and Docker https://docs.docker.com/samples/library/mysql/#-via-docker-stack-deploy-or-docker-compose

1. First create the docker image for the ign-webserver server. docker build ign-webserver . Write down its image ID.

2. Then run a dockerized mysql database. docker run --name my-mysql -e MYSQL_ROOT_PASSWORD=<desired-root-pwd> -d mysql:5.7.21 This will create a mysql docker container with an empty mysql in it.

3. Then you need to connect to that mysql container and run some commands: docker exec -it my-mysql bash. From inside the container, connect to mysql using the client (eg. mysql -u root -p) and create databases gazebo and gazebo_test. eg: create database gazebo_test;.

4. Run the ign-webserver docker container and link it to the database. docker run --name ign-webserver --rm --link my-mysql:mysql -ti <ign-webserver-image-id> /bin/bash. This will open a new terminal inside the server container.

5. Then from inside the server container you will need to set the Env Var that points to the linked docker mysql. eg. export GZ_DB_ADDRESS="172.17.0.2:3306"

After that you can run the server (from inside the container) invoking the ign-webserver command, or run tests by doing go test.