cloud-spanner-emulator

Cloud Spanner Emulator

Cloud Spanner Emulator provides application developers with a locally-running, emulated instance of Cloud Spanner to enable local development and testing.

The main focus of the emulator is correctness - an application that runs against the emulator should be able to run against the Cloud Spanner service without any changes. It is not a goal of the emulator to be a performant standalone production database or to provide similar performance to Cloud Spanner. The emulator is specifically intended for local unit testing of applications targeting Cloud Spanner.

Quickstart

There are multiple ways to invoke the emulator.

Via devcontainers

To build, develop and test emulator, devcontainers can be used which installs all the emulator dependencies inside a docker container. Opening this repository in vscode will automatically use the devcontainer defined in .devcontainer directory. This requires docker to be installed on the local machine.

To work directly in the browser without installing docker, a github Codespace can be created and opened in vscode.dev in the browser.

Via gcloud

The emulator is included in the Google Cloud SDK and can be invoked via the gcloud emulators command group:

gcloud components update
gcloud emulators spanner start

Via pre-built docker image

The emulator is also provided as a pre-built docker image. You can run the latest version with:

docker pull gcr.io/cloud-spanner-emulator/emulator
docker run -p 9010:9010 -p 9020:9020 gcr.io/cloud-spanner-emulator/emulator

The first port is the gRPC port and the second port is the REST port. The docker images are also tagged with version numbers, so you can run a specific version with:

VERSION=1.5.6
docker run -p 9010:9010 -p 9020:9020 gcr.io/cloud-spanner-emulator/emulator:$VERSION

Works on x86 and arm64 architectures.

Via pre-built linux binaries

The emulator is also distributed as a standalone linux binary. Note that this binary is not fully static, but has been tested on Ubuntu 18.04+, CentOS 7+, RHEL 8+ and Debian 10+.

Set ARCHITECTURE to arm64 in following command if you are working on arm machine.

VERSION=1.5.6
ARCHITECTURE=amd64
wget https://storage.googleapis.com/cloud-spanner-emulator/releases/${VERSION}/cloud-spanner-emulator_linux_${ARCHITECTURE}-${VERSION}.tar.gz
tar xvf cloud-spanner-emulator_linux_${ARCHITECTURE}-${VERSION}.tar.gz
chmod u+x gateway_main emulator_main

emulator_main contains the gRPC server. If you do not need REST functionality, you can just use this binary. To override the default host/port at which the emulator runs:

./emulator_main --host_port localhost:1234

gateway_main is the REST gateway which will also start the emulator gRPC server as a subprocess. To override the default host/port for the gateway and emulator:

./gateway_main --hostname localhost --grpc_port 1234 --http_port 1235

Via bazel

Production releases of the emulator are built on Ubuntu 18.04 with bazel 5.4.0 and gcc 8.4. You may be able to compile on compatible systems with compatible toolchains. From the emulator source directory, you can build and run the emulator via bazel from the source root with:

bazel run binaries/gateway_main

Works on x86 and arm64 architectures.

Via custom docker image

You can build the emulator docker image from the source root with:

docker build . -t emulator -f build/docker/Dockerfile.ubuntu

You can then invoke the emulator with:

docker run -p 9010:9010 -p 9020:9020 emulator

The first port is the gRPC port and the second port is the REST port.

Works on x86 and arm64 architectures.

Technical Details

The Cloud Spanner Emulator is built using the ZetaSQL reference implementation and is divided into three layers (each in its own directory):

• A REST gateway generated by grpc-gateway
• A gRPC frontend which implements Cloud Spanner's gRPC API
• A database backend which emulates Cloud Spanner's database features

The core emulator codebase is in C++, and the REST wrapper is written in Go. SQL query execution, value/type classes, and SQL functions are provided by the ZetaSQL reference implementation. The API surface, DDL, transactional semantics, constraint enforcement, and in-memory storage are implemented in this codebase.

Features and Limitations

Notable supported features:

• DDL schema changes

• Full SQL/DML query execution (limitations noted below)

• DML sequence numbers

• Non-SQL read and write methods

• Instance and Database admin APIs including long running operations

• Reads with stale timestamps

• Secondary indexes

• Commit timestamps

• Information schema

• Partitioned Read, Partitioned Query and Partitioned DML APIs

• Foreign keys

• NUMERIC type

• JSON type

• Generated columns

• Generated primary keys

• Check constraint

• Dataflow templates

• The Cloud Spanner PostgreSQL interface.

• Connecting with PostgreSQL drivers and tools through PGAdapter.

Notable limitations:

• The gRPC and REST endpoints run on separate ports and serve unencrypted traffic.

• gRPC request deadlines and cancellations are ignored by the emulator.

• IAM apis (SetIamPolicy, GetIamPolicy, SetIamPermissions) and Backup APIs are not supported.

• The emulator only allows one read-write transaction or schema change at a time. Any concurrent transaction will be aborted. Transactions should always be wrapped in a retry loop. This recommendation applies to the Cloud Spanner service as well.

• The emulator does not support persistence - all data is kept in memory and discarded when the emulator terminates.

• Error messages may not be consistent between the emulator and the Cloud Spanner service. Error messages are not part of Cloud Spanner's API contract and application code should not depend on the text of the error message being consistent.

• If multiple constraint violations are found during a transaction commit, the violation reported by the emulator may be different from the one reported by the Cloud Spanner service.

• The SQL query modes PLAN and PROFILE do not return any query plans. The emulator does not guarantee the same query execution plan as the Cloud Spanner service, and hence query plans and statistics reporting are disabled on the emulator. Running a query in PLAN mode will just analyze the SQL string and return the corresponding metadata. Running a query in PROFILE mode executes the query and returns the number of rows that was returned and the execution time. The execution time has no relation with what the execution time on Cloud Spanner will be.

• Some queries that use SQL functionality present in ZetaSQL but not in Cloud Spanner service may succeed instead of being rejected as invalid.

• Certain quotas and limits (such as admin api rate limits and mutation size limits) are not enforced.

• List APIs (ListSessions, ListInstances) do not support filtering by labels.

• Many tables related to runtime introspection in the SPANNER_SYS schema (e.g., query stats tables) are not supported.

• Server-side monitoring and logging functionality such as audit logs, stackdriver logging, and stackdriver monitoring are not supported.

• Foreign key backing index names generated in production Cloud Spanner are usable in emulator query hints. However, index names generated in the emulator cannot be used in the emulator or in production.

Frequently Asked Questions (FAQ)

Which client library versions are supported?

All Cloud Spanner client libraries support the emulator. Install the following version (or higher) to get emulator support:

How do I fix the client library error with "UNAUTHENTICATED: Credentials require .."?

Upgrade to the latest client library versions as listed above.

What is the recommended test setup?

Use a single emulator process and create a Cloud Spanner instance within it. Since creating databases is cheap in the emulator, we recommend that each test bring up and tear down its own database. This ensures hermetic testing and allows the test suite to run tests in parallel if needed.

Why is the order of rows returned by the emulator different across runs?

The emulator intentionally randomizes query results with no ORDER BY clause. You should not depend on ordering done by the Cloud Spanner service in the absence of an ORDER BY clause.

Why does the emulator fail with "Check failed: LoadTimeZone(...)"

The emulator relies on the absl library which in turns uses the tzdata library. The tzdata library must be installed on the system which emulator is running. This will need to be added to the docker build file if you are running through docker.

Contribute

We are currently not accepting external code contributions to this project.

Issues

Please file bugs and feature requests using GitHub's issue tracker or using the existing Cloud Spanner support channels.

Security

For information on reporting security vulnerabilities, see SECURITY.md.

License

Apache License 2.0