sentry-sdk

Sentry SDK Test Harness

This utility allows us to test the completeness/correctness of our errortracking implementation around Sentry SDK - as documented here.

The images built by this utility are used in the errortracking e2e tests.

Architecture

The test harness leverages the testcontainers-go package to build containers for various Sentry SDK clients which then interact with a local HTTP server as a (mock) Sentry server. This allows us to capture the requests, their associated payloads and test (de)serialising them into errortracking abstractions.

Features

• Provides a Sentry-compatible HTTP server to test Sentry SDK clients against
• Provides the ability to (de)serialise received Sentry payloads into errortracking abstractions, e.g. Event, Envelope.
• Provides the ability to visualise and download Sentry payloads locally to use as fixtures of other errortracking tests.

Usage

• First, build the docker images for the supported Sentry SDK clients

➜ make docker-build-apps

• Running test-harness locally which then produces a compatibility matrix for all supported client SDKs with associated item types.

➜ make deploy
(snipped)
+-------------------------------+----------+----------------------+-------------------------------------------------------------+
|            CLIENT             | ENDPOINT | SUPPORTED ITEM TYPES |                           PAYLOAD                           |
+-------------------------------+----------+----------------------+-------------------------------------------------------------+
| sentry-go/0.20.0              | store    | exception            | testdata/payloads/84936b0a-0212-4d27-aa83-56afafcb216b.json |
| sentry-go/0.20.0              | store    | message              | testdata/payloads/2f97c272-b8c2-4ed1-a718-67ceb86b1120.json |
| sentry.java:6.18.1            | envelope | event,message        | testdata/payloads/f76dd877-29b1-4408-a0f6-c3fe940a0745.json |
| sentry.java:6.18.1            | envelope | event,exception      | testdata/payloads/ca463870-d425-45e7-b8c8-00d0391ce9a9.json |
| sentry.javascript.node:7.38.0 | envelope | event,message        | testdata/payloads/dcdb6cb0-be0d-4840-8b5a-21bfef596334.json |
| sentry.javascript.node:7.38.0 | envelope | event,exception      | testdata/payloads/6fa4b4f2-443e-43eb-b2d2-822c2d6a78df.json |
| sentry.javascript.node:7.38.0 | envelope | event,exception      | testdata/payloads/803f1ae0-f9ea-401e-bea3-200901dd726a.json |
| sentry.python/1.21.0          | envelope | event,exception      | testdata/payloads/a1ab7ad4-eb57-49fa-afa0-40b31a3cb461.json |
| sentry.python/1.21.0          | envelope | session              | testdata/payloads/ac53adcb-b874-4e55-9d91-81bf6b66d42b.json |
| sentry.python/1.21.0          | envelope | event,message        | testdata/payloads/5ebe8cd4-7d40-4b7c-9234-781b00e5fcf5.json |
| sentry.ruby:5.9.0             | envelope | event,message        | testdata/payloads/cc1475ab-21f6-420f-9ae7-6a016843f56e.json |
| sentry.ruby:5.9.0             | envelope | event,exception      | testdata/payloads/7aec7f69-fa6f-42ae-a816-3c3659adeaef.json |
| sentry.rust/0.31.0            | envelope | event,message        | testdata/payloads/ca0fe7db-6379-43d3-a998-17d90cb01878.json |
| sentry.rust/0.31.0            | envelope | event,exception      | testdata/payloads/9aaddf35-ddad-4258-bc06-a76e93c10896.json |
| sentry.rust/0.31.0            | envelope | session              | testdata/payloads/5ef07f13-7ee5-47ee-9e72-93c99398477b.json |
| sentry.rust/0.31.0            | envelope | event,message        | testdata/payloads/11fc96de-a05d-4c44-969b-63b95f912f65.json |
| sentry.rust/0.31.0            | envelope | event,exception      | testdata/payloads/5e4c38ca-ee89-45d6-b03b-48c2b0ca063e.json |
+-------------------------------+----------+----------------------+-------------------------------------------------------------+

• You can also use an external Sentry DSN instead of the internal test server

➜ SENTRY_DSN=http://localhost:8888 make deploy

• Limiting make run to test a given set of SDK(s)

➜ TEST_THESE_SDKS_ONLY=rust,java make deploy

Note: The comma-separated list of SDK names specified above should have been configured within internal/harness/sdk_containers.go first, or else the runner errors with...

using TEST_THESE_SDKS_ONLY override to filter testcontainers...
panic: we do not yet have testcontainers configuration for <foo>

To register a new one, read Adding a new client SDK to the test-harness below.

• Inspecting received payloads

➜  cat testdata/payloads/84936b0a-0212-4d27-aa83-56afafcb216b.json | gojq -r .body
{"contexts":{"device":{"arch":"arm64","num_cpu":5},"os":{"name":"linux"},"runtime":{"go_maxprocs":5,"go_numcgocalls":0,"go_numroutines":2,"name":"go","version":"go1.19.9"}},"event_id":"5a575f4b1f4d4158add77e45ab2c8afd","level":"error","platform":"go","sdk":{"name":"sentry.go","version":"0.20.0","integrations":["ContextifyFrames","Environment","IgnoreErrors","Modules"],"packages":[{"name":"sentry-go","version":"0.20.0"}]},"server_name":"3b9d4bc8962b","user":{},"modules":{"":"","github.com/getsentry/sentry-go":"v0.20.0","golang.org/x/sys":"v0.6.0","golang.org/x/text":"v0.8.0"},"exception":[{"type":"*errors.errorString","value":"unsupported protocol scheme \"\""},{"type":"*url.Error","value":"Get \"fake-url\": unsupported protocol scheme \"\"","stacktrace":{"frames":[{"function":"main","module":"main","abs_path":"/go/src/gitlab.com/opstrace/error-tracking-test/go/main.go","lineno":100,"in_app":true}]}}],"timestamp":"2023-05-05T09:57:25.893377795Z"}

• Adding a new client SDK to the test-harness
  • Add your implementation of a new client SDK inside testdata/supported-sdk-clients/<language>
  • Ensure you also have a Dockerfile to package the sample app as a Docker image.
  • Ensure the program uses `SENTRY_DSN`` as the standard env var.
  • Ensure the program uses `INSECURE_OK`` env var to disable TLS verification (if set to "true").
  • Ensure the program can handle SENTRY_USER env var to set the scope username.
  • Ensure there's a corresponding entry in pkg/sdk_containers.go.
  • Once you have your image building, test with make run.

Push New Images

For GitLab team members, use the manual CI job "sentry sdk images" to push new images. These will use the tagging format from the Makefile.go.mk Makefile. latest tagged images are not pushed.

Update the image tag used in the e2e tests accordingly.