protoc-gen-multi

protoc-gen-multi

Combine multiple protoc plugin invocations into one with protoc-gen-multi.

Custom multi-plugin for Generated SDKs

Some plugins depend on generated code from other plugins. A common example is protoc-gen-go-grpc and protoc-gen-go. The gRPC code will reference types (messages, enums, etc.) generated by the base plugin.

In this scenario the gRPC generated code assumes the types will be available in the same package. There are limitations when generating code to the same package, however, it's beyond the scope of this document to go into detail.

Due to these limitations, Generated SDKs from the BSR are split in to separate packages and treated as individual modules. This is done by the BSR automatically when generating code for plugins that depend on other plugins.

However, there are some plugins that produce code that cannot be generated to a different package. One such example is the protoc-gen-go-vtproto plugin, which generates optimized marshaling/unmarshaling Go code alongside the base types.

So, what do you do if you want to use Generated SDKs from the BSR, but also want to use a plugin that cannot be generated to a different package?

The answer is to use a custom plugin that wraps the base plugin and any other plugin(s) that cannot be generated to a different package. This custom plugin will output code to the same package and utilize Generated SDKs from the BSR.

In this example, we’ll demonstrate how to package the following plugins into the same Docker image:

• protoc-gen-go-grpc
• protoc-gen-go
• protoc-gen-go-vtproto
• protoc-gen-grpc-gateway

Step 1 - Create an organization to push custom plugin(s)

For this example let’s use https://buf.example.com/custom-plugins

Step 2 - Build a Docker image

Create a docker image with all the protoc plugins desired, plus protoc-gen-multi:

# syntax=docker/dockerfile:1.6
FROM --platform=$BUILDPLATFORM golang:1.21-alpine AS build
ARG TARGETOS TARGETARCH

ENV CGO_ENABLED=0 GOOS=$TARGETOS GOARCH=$TARGETARCH 

# Install protoc-gen-multi, must be installed.
RUN go install -ldflags "-s -w" github.com/bufbuild/tools/cmd/protoc-gen-multi@latest

# Add custom plugins here
RUN go install -ldflags "-s -w" google.golang.org/protobuf/cmd/protoc-gen-go@v1.32 \
    && go install -ldflags "-s -w" google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.3 \
    && go install -ldflags "-s -w" github.com/planetscale/vtprotobuf/cmd/protoc-gen-go-vtproto@v0.6.0 \
    && go install -ldflags "-s -w" github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-grpc-gateway@v2.19

# Move binaries prefixed with GOOS_GOARCH to /go/bin.
RUN mv /go/bin/${TARGETOS}_${TARGETARCH}/* /go/bin || true

# Build final image.
FROM scratch
COPY --from=build --link /etc/passwd /etc/passwd
COPY --from=build /go/bin/ /bin
USER nobody
ENTRYPOINT [ "protoc-gen-multi" ]

Build the image with a tag referencing the organization and plugin name:

docker buildx build --platform linux/amd64 -t buf.example.com/custom-plugins/multi:v0.1.0 .

Step 3 - Create a buf.plugin.yaml file

At the very minimum you’ll need the following fields set:

version: v1
name: buf.example.com/custom-plugins/multi
plugin_version: v0.1.0
output_languages:
  - go
registry:
  # Add the runtime deps required by your plugins for the generated SDK.
  go:
    deps:
      - module: google.golang.org/protobuf
        version: v1.32.0
      - module: google.golang.org/grpc
        version: v1.3.0
      - module: github.com/planetscale/vtprotobuf
        version: v0.6.0
      - module: github.com/grpc-ecosystem/grpc-gateway/v2
        version: v2.19
  # Add the options to invoke each plugin for the generated SDK.
  opts:
    - --go_out=.
    - --go_opt=paths=source_relative
    - --go-grpc_out=.
    - --go-grpc_opt=paths=source_relative
    - --go-vtproto_out=.
    - --go-vtproto_opt=paths=source_relative,features=marshal+unmarshal+size
    - --grpc-gateway_out=.
    - --grpc-gateway_opt=paths=source_relative,generate_unbound_methods=true

Note that typically a buf.plugin.yaml file defines a single plugin. However, in this case, we have three different plugins with different versions. So the version specified here should be whatever makes sense for your setup.

There are additional fields that can be set. Please refer to the buf.plugin.yaml file documentation for more information. Link to the documentation

Step 4 - Push plugin to the BSR

buf beta registry plugin push \
    --visibility public \
    --image buf.example.com/custom-plugins/multi:v0.1.0 \
    --override-remote=buf.example.com

Now, when you install a Generated SDK from the BSR, e.g.

go get buf.example.com/gen/go/acme/petapis/custom-plugins/multi@latest

The BSR will generate a single package containing the output of all three plugins. It will not attempt to build plugin dependencies. This replicates the behavior of manually generating these plugins into the same output directory.