build

package module
v0.0.0-...-80a9e2a Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 16, 2024 License: BSD-3-Clause Imports: 6 Imported by: 30

README

Go Build Tools

Go Reference

This repository holds the source for various packages and tools that support Go's build system and the development of the Go programming language.

Warning: Packages here are internal to Go's build system and its needs. Some may one day be promoted to another golang.org/x repository, or they may be modified arbitrarily or even disappear altogether. In short, code in this repository is not subject to the Go 1 compatibility promise nor the Release Policy.

Report Issues / Send Patches

This repository uses Gerrit for code changes. To contribute, see https://golang.org/doc/contribute.html.

The main issue tracker for the blog is located at https://github.com/golang/go/issues. Prefix your issue with "x/build/DIR: " in the subject line.

Overview

The main components of the Go build system are:

  • The coordinator, in cmd/coordinator/, serves https://farmer.golang.org/ and https://build.golang.org/. It runs on GKE and coordinates the whole build system. It finds work to do (both pre-submit "TryBot" work, and post-submit work) and executes builds, allocating machines to run the builds. It is the owner of all machines. It holds the state for which builds passed or failed, and the build logs.

  • The Go package in buildenv/ contains constants for where the dashboard and coordinator run, for prod, staging, and local development.

  • The buildlet, in cmd/buildlet/, is the HTTP server that runs on each worker machine to execute builds on the coordinator's behalf. This runs on every possible GOOS/GOARCH value. The buildlet binaries are stored on Google Cloud Storage and fetched per-build, so we can update the buildlet binary independently of the underlying machine images. The buildlet is the most insecure server possible: it has HTTP handlers to read & write arbitrary content to disk, and to execute any file on disk. It also has an SSH tunnel handler. The buildlet must never be exposed to the Internet. The coordinator provisions buildlets in one of three ways:

    1. by creating VMs on Google Compute Engine (GCE) with custom images configured to fetch & run the buildlet on boot, listening on port 80 in a private network.

    2. by running Linux containers (on either Google Kubernetes Engine or GCE with the Container-Optimized OS image), with the container images configured to fetch & run the buildlet on start, also listening on port 80 in a private network.

    3. by taking buildlets out of a pool of connected, dedicated machines. The buildlet can run in either listen mode (as on GCE and GKE) or in reverse mode. In reverse mode, the buildlet connects out to https://farmer.golang.org/ and registers itself with the coordinator. The TCP connection is then logically reversed (using revdial and when the coordinator needs to do a build, it makes HTTP requests to the coordinator over the already-open TCP connection.

    These three pools can be viewed at the coordinator's https://farmer.golang.org/#pools.

  • The env/ directory describes build environments. It contains scripts to create VM images, Dockerfiles to create Kubernetes containers, and instructions and tools for dedicated machines.

  • maintner in maintner/ is a library for slurping all of Go's GitHub and Gerrit state into memory. The daemon maintnerd in maintner/maintnerd/ runs on GKE and serves https://maintner.golang.org/. The daemon watches GitHub and Gerrit and appends to a mutation log whenever it sees new activity. The logs are stored on GCS and served to clients.

  • The godata package in maintner/godata/ provides a trivial API to let anybody write programs against Go's maintner corpus (all of our GitHub and Gerrit history), live up to the second. It takes a few seconds to load into memory and a few hundred MB of RAM after it downloads the mutation log from the network.

  • pubsubhelper in cmd/pubsubhelper/ is a dependency of maintnerd. It runs on GKE, is available at https://pubsubhelper.golang.org/, and runs an HTTP server to receive Webhook updates from GitHub on new activity and an SMTP server to receive new activity emails from Gerrit. It then is a pubsub system for maintnerd to subscribe to.

  • The gitmirror server in cmd/gitmirror/ mirrors Gerrit to GitHub, and also serves a mirror of the Gerrit code to the coordinator for builds, so we don't overwhelm Gerrit and blow our quota.

  • The Go gopherbot bot logic runs on GKE. The code is in cmd/gopherbot. It depends on maintner via the godata package.

  • The developer dashboard at https://dev.golang.org/ runs on GKE. Its code is in devapp/. It also depends on maintner via the godata package.

  • cmd/retrybuilds: a Go client program to delete build results from the dashboard

  • The perfdata server, in perfdata/appengine serves https://perfdata.golang.org/. It runs on App Engine and serves the benchmark result storage system.

  • The perf server, in perf/appengine serves https://perf.golang.org/. It runs on App Engine and serves the benchmark result analysis system. See its README for how to start a local testing instance.

Adding a Go Builder

If you wish to run a Go builder, please email golang-dev@googlegroups.com first. There is documentation at https://golang.org/wiki/DashboardBuilders, but depending on the type of builder, we may want to run it ourselves, after you prepare an environment description (resulting in a VM image) of it. See the env directory.

Documentation

Overview

Package build contains constants for the Go continuous build system.

Index

Constants

View Source
const DevCoordinatorCA = `` /* 955-byte string literal not displayed */

DevCoordinatorCA is the cert used by the coordinator and buildlet in development mode. (Not to be confused with the staging "dev" instance under GCE project "go-dashboard-dev")

View Source
const DevCoordinatorKey = `` /* 1674-byte string literal not displayed */

DevCoordinatorKey is the key used by the coordinator and buildlet in development mode. (Not to be confused with the staging "dev" instance under GCE project "go-dashboard-dev")

View Source
const ProdCoordinatorCA = `` /* 1114-byte string literal not displayed */

ProdCoordinatorCA is the production CA cert for farmer.golang.org.

View Source
const StagingCoordinatorCA = `` /* 1073-byte string literal not displayed */

StagingCoordinatorCA is the cert used on GCE for the go-dashboard-dev project.

Variables

This section is empty.

Functions

This section is empty.

Types

type CoordinatorInstance

type CoordinatorInstance string

CoordinatorInstance is either "prod", "staging", or "localhost:<port>".

const (
	ProdCoordinator    CoordinatorInstance = "prod"
	StagingCoordinator CoordinatorInstance = "staging"
)

func (CoordinatorInstance) CACert

func (ci CoordinatorInstance) CACert() string

CACert returns the public certificate of the CA used to sign this coordinator instance's certificate.

func (CoordinatorInstance) TLSDialer

func (ci CoordinatorInstance) TLSDialer() func(network, addr string) (net.Conn, error)

func (CoordinatorInstance) TLSHostPort

func (ci CoordinatorInstance) TLSHostPort() (string, error)

Directories

Path Synopsis
Package autocertcache contains autocert.Cache implementations for golang.org/x/crypto/autocert.
Package autocertcache contains autocert.Cache implementations for golang.org/x/crypto/autocert.
Package buildenv contains definitions for the environments the Go build system can run in.
Package buildenv contains definitions for the environments the Go build system can run in.
Package buildlet contains client tools for working with a buildlet server.
Package buildlet contains client tools for working with a buildlet server.
cloudfns
sendwikidiff
Package sendwikidiff implements a Google Cloud background function that reacts to a pubsub message containing a GitHub webhook change payload.
Package sendwikidiff implements a Google Cloud background function that reacts to a pubsub message containing a GitHub webhook change payload.
wikiwebhook
Package wikiwebhook implements an Google Cloud Function HTTP handler that expects GitHub webhook change events.
Package wikiwebhook implements an Google Cloud Function HTTP handler that expects GitHub webhook change events.
issueswebhook Module
cmd
bootstrapswarm
bootstapswarm will bootstrap the swarming bot depending on the environment that it is run on.
bootstapswarm will bootstrap the swarming bot depending on the environment that it is run on.
buildlet
The buildlet is an HTTP server that untars content to disk and runs commands it has untarred, streaming their output back over HTTP.
The buildlet is an HTTP server that untars content to disk and runs commands it has untarred, streaming their output back over HTTP.
buildlet/stage0
The stage0 command looks up the buildlet's URL from its environment (GCE metadata service, EC2, etc), downloads it, and runs it.
The stage0 command looks up the buildlet's URL from its environment (GCE metadata service, EC2, etc), downloads it, and runs it.
buildlet/testssh
The testssh binary exists to verify that a buildlet container's ssh works, without running the whole coordinator binary in the staging environment.
The testssh binary exists to verify that a buildlet container's ssh works, without running the whole coordinator binary in the staging environment.
buildstats
The buildstats command syncs build logs from Datastore to Bigquery.
The buildstats command syncs build logs from Datastore to Bigquery.
coordinator
The coordinator runs the majority of the Go build system.
The coordinator runs the majority of the Go build system.
coordinator/internal/dashboard
Package dashboard contains the implementation of the build dashboard for the Coordinator.
Package dashboard contains the implementation of the build dashboard for the Coordinator.
coordinator/internal/legacydash
Package legacydash holds the serving code for the build dashboard (build.golang.org) and its remaining HTTP API endpoints.
Package legacydash holds the serving code for the build dashboard (build.golang.org) and its remaining HTTP API endpoints.
coordinator/internal/lucipoll
Package lucipoll implements a simple polling LUCI client for the possibly-short-term needs of the build dashboard.
Package lucipoll implements a simple polling LUCI client for the possibly-short-term needs of the build dashboard.
debugnewvm
The debugnewvm command creates and destroys a VM-based buildlet with lots of logging for debugging.
The debugnewvm command creates and destroys a VM-based buildlet with lots of logging for debugging.
docker2boot
The docker2boot command converts a Docker image into a bootable GCE VM image.
The docker2boot command converts a Docker image into a bootable GCE VM image.
fetchlogs
Fetchlogs downloads build failure logs from the Go dashboard so they can be accessed and searched from the local file system.
Fetchlogs downloads build failure logs from the Go dashboard so they can be accessed and searched from the local file system.
genbootstrap
Genbootstrap prepares GOROOT_BOOTSTRAP tarballs suitable for use on builders.
Genbootstrap prepares GOROOT_BOOTSTRAP tarballs suitable for use on builders.
genbotcert
Command genbotcert can both generate a CSR and private key for a LUCI bot and generate a certificate from a CSR.
Command genbotcert can both generate a CSR and private key for a LUCI bot and generate a certificate from a CSR.
genbuilderkey
The genbuilderkey binary generates a builder key or gomote user key from the build system's master key.
The genbuilderkey binary generates a builder key or gomote user key from the build system's master key.
gerritbot
The gerritbot binary converts GitHub Pull Requests to Gerrit Changes, updating the PR and Gerrit Change as appropriate.
The gerritbot binary converts GitHub Pull Requests to Gerrit Changes, updating the PR and Gerrit Change as appropriate.
gerritbot/internal/rules
Package rules specifies a simple set of rules for checking GitHub PRs or Gerrit CLs for certain common mistakes, like no package in the first line of the commit message or having long lines in the commit message body.
Package rules specifies a simple set of rules for checking GitHub PRs or Gerrit CLs for certain common mistakes, like no package in the first line of the commit message or having long lines in the commit message body.
gitmirror
The gitmirror binary watches the specified Gerrit repositories for new commits and syncs them to mirror repositories.
The gitmirror binary watches the specified Gerrit repositories for new commits and syncs them to mirror repositories.
gomote
The gomote command is a client for the Go builder infrastructure.
The gomote command is a client for the Go builder infrastructure.
gopherbot
The gopherbot command runs Go's gopherbot role account on GitHub and Gerrit.
The gopherbot command runs Go's gopherbot role account on GitHub and Gerrit.
gorebuild
Gorebuild rebuilds and verifies the distribution files posted at https://go.dev/dl/.
Gorebuild rebuilds and verifies the distribution files posted at https://go.dev/dl/.
greplogs
Command greplogs searches Go builder logs.
Command greplogs searches Go builder logs.
greplogs/_embed
Command broken lists the current Go builders with known issues.
Command broken lists the current Go builders with known issues.
greplogs/internal/logparse
Package logparse contains functions for parsing and analyzing build and test logs produced by all.bash.
Package logparse contains functions for parsing and analyzing build and test logs produced by all.bash.
makemac
Command makemac manages MacService instances for LUCI.
Command makemac manages MacService instances for LUCI.
perfrun
perfrun interacts with the buildlet coordinator to run the go1 benchmarks on a buildlet follower for the most recent successful commits according to the build dashboard.
perfrun interacts with the buildlet coordinator to run the go1 benchmarks on a buildlet follower for the most recent successful commits according to the build dashboard.
pubsubhelper
The pubsubhelper is an SMTP server for Gerrit updates and an HTTP server for Github webhook updates.
The pubsubhelper is an SMTP server for Gerrit updates and an HTTP server for Github webhook updates.
pubsubhelper/pubsubtypes
Package pubsubtypes contains types published by pubsubhelper.
Package pubsubtypes contains types published by pubsubhelper.
racebuild
racebuild builds the race runtime (syso files) on all supported OSes using gomote.
racebuild builds the race runtime (syso files) on all supported OSes using gomote.
releaseschedule
Releaseschedule generates the release schedule diagram used on the release schedule wiki.
Releaseschedule generates the release schedule diagram used on the release schedule wiki.
relnote
The relnote command summarizes the Go changes in Gerrit marked with RELNOTE annotations for the release notes.
The relnote command summarizes the Go changes in Gerrit marked with RELNOTE annotations for the release notes.
relui
relui is a web interface for managing the release process of Go.
relui is a web interface for managing the release process of Go.
retrybuilds
The retrybuilds command clears build failures from the build.golang.org dashboard to force them to be rebuilt.
The retrybuilds command clears build failures from the build.golang.org dashboard to force them to be rebuilt.
rundockerbuildlet
The rundockerbuildlet command loops forever and creates and cleans up Docker containers running reverse buildlets.
The rundockerbuildlet command loops forever and creates and cleans up Docker containers running reverse buildlets.
runqemubuildlet
Binary runqemubuildlet runs a single VM-based buildlet in a loop.
Binary runqemubuildlet runs a single VM-based buildlet in a loop.
updatestd
updatestd is an experimental program that has been used to update the standard library modules as part of golang.org/issue/36905 in CL 255860 and CL 266898.
updatestd is an experimental program that has been used to update the standard library modules as part of golang.org/issue/36905 in CL 255860 and CL 266898.
upload
The upload command writes a file to Google Cloud Storage.
The upload command writes a file to Google Cloud Storage.
watchflakes/internal/cache
Package cache implements a simple file-based cache.
Package cache implements a simple file-based cache.
watchflakes/internal/script
Package script implements a simple classification scripting language.
Package script implements a simple classification scripting language.
xb
The xb command wraps GCP deployment commands such as gcloud, kubectl, and docker push and verifies they're interacting with the intended prod-vs-staging environment.
The xb command wraps GCP deployment commands such as gcloud, kubectl, and docker push and verifies they're interacting with the intended prod-vs-staging environment.
crfs module
Package dashboard contains shared configuration and logic used by various pieces of the Go continuous build system.
Package dashboard contains shared configuration and logic used by various pieces of the Go continuous build system.
Devapp is the server running dev.golang.org.
Devapp is the server running dev.golang.org.
env
Package gerrit contains code to interact with Gerrit servers.
Package gerrit contains code to interact with Gerrit servers.
This program runs in the InfluxDB container, performs initial setup of the database, and publishes access secrets to secret manager.
This program runs in the InfluxDB container, performs initial setup of the database, and publishes access secrets to secret manager.
access
Package access provides primitives for implementing authentication and authorization.
Package access provides primitives for implementing authentication and authorization.
basedir
Package basedir finds templates and static files associated with a binary.
Package basedir finds templates and static files associated with a binary.
buildgo
Package buildgo provides tools for pushing and building the Go distribution on buildlets.
Package buildgo provides tools for pushing and building the Go distribution on buildlets.
buildstats
Package buildstats contains code to sync the coordinator's build logs from Datastore to BigQuery.
Package buildstats contains code to sync the coordinator's build logs from Datastore to BigQuery.
coordinator/remote
Package remote provides remote access to resources in the build infrastructure.
Package remote provides remote access to resources in the build infrastructure.
dash
Package dash reads build.golang.org's dashboards.
Package dash reads build.golang.org's dashboards.
datastore/fake
Package fake provides a fake implementation of a Datastore client to use in testing.
Package fake provides a fake implementation of a Datastore client to use in testing.
envutil
Package envutil provides utilities for working with environment variables.
Package envutil provides utilities for working with environment variables.
foreach
Package foreach provides allocation-conscious helpers for iterating over lines of text.
Package foreach provides allocation-conscious helpers for iterating over lines of text.
gcsfs
gcsfs implements io/fs for GCS, adding writability.
gcsfs implements io/fs for GCS, adding writability.
gitauth
Package gitauth writes gitcookies files so git will authenticate to Gerrit as gopherbot for quota purposes.
Package gitauth writes gitcookies files so git will authenticate to Gerrit as gopherbot for quota purposes.
gomote
Package gomote contains all of the necessary components to implement and use the gomote functionality.
Package gomote contains all of the necessary components to implement and use the gomote functionality.
gophers
Package gophers is a list of names, emails, and GitHub usernames of people from the Go git repos and issue trackers.
Package gophers is a list of names, emails, and GitHub usernames of people from the Go git repos and issue trackers.
httpdl
Package httpdl downloads things from HTTP to local disk.
Package httpdl downloads things from HTTP to local disk.
https
Package https contains helpers for starting an HTTP/HTTPS server.
Package https contains helpers for starting an HTTP/HTTPS server.
iapclient
Package iapclient enables programmatic access to IAP-secured services.
Package iapclient enables programmatic access to IAP-secured services.
influx
Package influx provides common constants for setting up and access the performance monitoring InfluxDB instance.
Package influx provides common constants for setting up and access the performance monitoring InfluxDB instance.
installer/darwinpkg
Package darwinpkg encodes the process of building a macOS PKG installer from the given Go toolchain .tar.gz binary archive.
Package darwinpkg encodes the process of building a macOS PKG installer from the given Go toolchain .tar.gz binary archive.
installer/windowsmsi
Package windowsmsi encodes the process of building a Windows MSI installer from the given Go toolchain .tar.gz binary archive.
Package windowsmsi encodes the process of building a Windows MSI installer from the given Go toolchain .tar.gz binary archive.
loghash
Package loghash provides the shared information for computing a log hash (as in https://build.golang.org/log/HASH).
Package loghash provides the shared information for computing a log hash (as in https://build.golang.org/log/HASH).
logparser
Package logparser parses build.golang.org dashboard logs.
Package logparser parses build.golang.org dashboard logs.
lru
Package lru implements an LRU cache.
Package lru implements an LRU cache.
macservice
Package macservice defines the client API for MacService.
Package macservice defines the client API for MacService.
metrics
Package metrics provides a service for reporting metrics to Stackdriver, or locally during development.
Package metrics provides a service for reporting metrics to Stackdriver, or locally during development.
relui/sign
Package sign provides implementations for servers which can sign Go release artifacts.
Package sign provides implementations for servers which can sign Go release artifacts.
secret
Package secret provides a client interface for interacting with the GCP Secret Management service.
Package secret provides a client interface for interacting with the GCP Secret Management service.
singleflight
Package singleflight provides a duplicate function call suppression mechanism.
Package singleflight provides a duplicate function call suppression mechanism.
sourcecache
Package sourcecache provides a cache of code found in Git repositories.
Package sourcecache provides a cache of code found in Git repositories.
spanlog
Package spanlog provides span and event logger interfaces.
Package spanlog provides span and event logger interfaces.
swarmclient
Package swarmclient contains a LUCI swarming client.
Package swarmclient contains a LUCI swarming client.
task
Package task implements tasks involved in making a Go release.
Package task implements tasks involved in making a Go release.
untar
Package untar untars a tarball to disk.
Package untar untars a tarball to disk.
workflow
Package workflow declaratively defines computation graphs that support automatic parallelization, persistence, and monitoring.
Package workflow declaratively defines computation graphs that support automatic parallelization, persistence, and monitoring.
Package kubernetes contains a minimal client for the Kubernetes API.
Package kubernetes contains a minimal client for the Kubernetes API.
api
Package api contains the Kubernetes v1 API types.
Package api contains the Kubernetes v1 API types.
gke
Package gke contains code for interacting with Google Container Engine (GKE), the hosted version of Kubernetes on Google Cloud Platform.
Package gke contains code for interacting with Google Container Engine (GKE), the hosted version of Kubernetes on Google Cloud Platform.
Package livelog provides a buffer that can be simultaneously written to by one writer and read from by many readers.
Package livelog provides a buffer that can be simultaneously written to by one writer and read from by many readers.
Package maintner mirrors, searches, syncs, and serves Git, Github, and Gerrit metadata.
Package maintner mirrors, searches, syncs, and serves Git, Github, and Gerrit metadata.
godata
Package godata loads the Go project's corpus of Git, Github, and Gerrit activity into memory to allow easy analysis without worrying about APIs and their pagination, quotas, and other nuisances and limitations.
Package godata loads the Go project's corpus of Git, Github, and Gerrit activity into memory to allow easy analysis without worrying about APIs and their pagination, quotas, and other nuisances and limitations.
internal/robustio
Package robustio wraps I/O functions that are prone to failure on Windows, transparently retrying errors up to an arbitrary timeout.
Package robustio wraps I/O functions that are prone to failure on Windows, transparently retrying errors up to an arbitrary timeout.
maintnerd
The maintnerd command serves project maintainer data from Git, Github, and/or Gerrit.
The maintnerd command serves project maintainer data from Git, Github, and/or Gerrit.
maintnerd/gcslog
Package gcslog is an implementation of maintner.MutationSource and Logger for Google Cloud Storage.
Package gcslog is an implementation of maintner.MutationSource and Logger for Google Cloud Storage.
maintnerd/maintapi
Package maintapi exposes a gRPC maintner service for a given corpus.
Package maintapi exposes a gRPC maintner service for a given corpus.
maintnerd/maintapi/version
Package version implements logic to parse version of Go tags and release branches.
Package version implements logic to parse version of Go tags and release branches.
maintpb
Package maintpb is a generated protocol buffer package.
Package maintpb is a generated protocol buffer package.
maintq
The maintq command queries a maintnerd gRPC server.
The maintq command queries a maintnerd gRPC server.
maintwatch
The maintwatch commands tails the maintner mutation log.
The maintwatch commands tails the maintner mutation log.
reclog
Package reclog contains readers and writers for a record wrapper format used by maintner.
Package reclog contains readers and writers for a record wrapper format used by maintner.
Package pargzip contains a parallel gzip writer implementation.
Package pargzip contains a parallel gzip writer implementation.
perf runs an HTTP server for benchmark analysis.
perf runs an HTTP server for benchmark analysis.
app
Package app implements the performance data analysis server.
Package app implements the performance data analysis server.
Package perfdata contains a client for the performance data storage server.
Package perfdata contains a client for the performance data storage server.
app
Package app implements the performance data storage server.
Package app implements the performance data storage server.
appengine
This binary contains an App Engine app for perfdata.golang.org
This binary contains an App Engine app for perfdata.golang.org
db
Package db provides the high-level database interface for the storage app.
Package db provides the high-level database interface for the storage app.
db/sqlite3
Package sqlite3 provides the sqlite3 driver for x/build/perfdata/db.
Package sqlite3 provides the sqlite3 driver for x/build/perfdata/db.
fs
Package fs provides a backend-agnostic filesystem layer for storing performance results.
Package fs provides a backend-agnostic filesystem layer for storing performance results.
fs/gcs
Package gcs implements the fs.FS interface using Google Cloud Storage.
Package gcs implements the fs.FS interface using Google Cloud Storage.
fs/local
Package local implements the fs.FS interface using local files.
Package local implements the fs.FS interface using local files.
localperfdata
Localperfdata runs an HTTP server for benchmark perfdata.
Localperfdata runs an HTTP server for benchmark perfdata.
query
Package query provides tools for parsing a query.
Package query provides tools for parsing a query.
Package relnote supports working with release notes.
Package relnote supports working with release notes.
Package repos contains information about Go source repositories.
Package repos contains information about Go source repositories.
revdial
v2
Package revdial implements a Dialer and Listener which work together to turn an accepted connection (for instance, a Hijacked HTTP request) into a Dialer which can then create net.Conns connecting back to the original dialer, which then gets a net.Listener accepting those conns.
Package revdial implements a Dialer and Listener which work together to turn an accepted connection (for instance, a Hijacked HTTP request) into a Dialer which can then create net.Conns connecting back to the original dialer, which then gets a net.Listener accepting those conns.
Package tarutil contains utilities for working with tar archives.
Package tarutil contains utilities for working with tar archives.
third_party
bandchart
Package bandchart provides an embedded bandchart.js.
Package bandchart provides an embedded bandchart.js.
Package types contains common types used by the Go continuous build system.
Package types contains common types used by the Go continuous build system.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL