spamcheck

Spamcheck

Project Structure

Spamcheck started as an internal project by and for GitLab, over time it became increasingly clear the community could profit from these efforts and the decision was made to strive towards making much of it public. At the moment, management and development (issues, merge requests, etc) happen in the gitlab-com Spamcheck project with the gitlab-org project being a code and registry mirror.

Architecture Diagram

The following diagram gives a high-level overview on how the various components of the anti-spam engine interact with each other:

Development

Development Environment

Install the g go version manager and the correct go version:

curl -sSL https://git.io/g-install | sh -s
# Restart shell session here
g install 1.15

Clone this repo, then build and run the container:

git clone git@gitlab.com:gitlab-com/gl-security/engineering-and-research/automation-team/spam/spamcheck.git
cd spamcheck
cp config/config.toml.example config/config.toml
# Customize config/config.toml if necessary
make run

Dependencies are stored in go.mod.

For generating the Ruby gem that is then used by GitLab you'll need to install bundler.

gem install bundler

Generating gRPC protobuf files

To build the Golang protobuf files when you've made a change: bash make protobuf

To build the Ruby protobuf files bash make proto_ruby

Advanced Configuration

config/config.yml has more configuration options.

Concept overview

Testing

Tests are files with the _test.go suffix. Typically, they sit next to the files they're testing in the directory structure.

Test suite

Run the test suite:

go test ./...

Manually

Run via go run main.go config config/config.toml or just run make.

The REST endpoint accepts JSON that will get translated to protobufs:

curl -H 'Content-Type: application/json' -d "$(cat examples/checkforspamissue.json)" http://localhost:8080/v1/issue_spam

To test the gRPC endpoint, use grpcurl:

grpcurl -plaintext -d "$(cat examples/checkforspamissue.json)" localhost:8001 spamcheck.SpamcheckService/CheckForSpamIssue

By default, it will return an empty object, e.g. {}, because Protobufs don't encode 0-value fields and SpamVerdict.Verdict is an enum whose default value is ALLOW which is 0. This default return value is in app/controllers/spamcheck.go

There is a /healthz endpoint used for Kubernetes probes and uptime checks:

curl localhost:8080/healthz
grpcurl -plaintext localhost:8001 spamcheck.SpamcheckService/Healthz

Publishing a new gem version

1. Build new Ruby protobuf files:

   make proto_ruby
   

2. Make sure to update ruby/spamcheck/version.rb with the new version number. Keep in mind that bundle update uses --major by default, so all minor version updates should not break backward or cross-protocol compatibility.

3. Sign up for a RubyGems account if you already don't have one. You should also be an owner of the spamcheck gem

4. Create a tag for spamcheck with the version number e.g. if the version number is 0.1.0, do:

   git tag v0.1.0
   

5. Check that the tag has been correctly created alongside your latest commit:

   git show v0.1.0
   

6. Push the branch with the tag:

   git push --tags origin <your branch>
   

7. Run bundle exec ruby _support/publish-gem v0.1.0 locally. It should ask you for your rubygems email and password.

8. After a successful push, the new gem version should now be publicly available on RubyGems.org and ready to use.