backends

package
v0.0.0-...-7aa2ad8 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: May 7, 2024 License: BSD-3-Clause Imports: 31 Imported by: 0

Documentation

Overview

Package backends provides common logic to provide connections to other backend services that bisection relies on

Index

Constants

View Source
const (
	// RBE CAS isolates expire after 32 days. We use 30 out of caution.
	CasExpiration = 30
	// ChromeProject refers to the "chrome" project.
	ChromeProject = "chrome"
	// ChromiumGitilesURL is the default Gitiles URL for chromium/src.
	ChromiumGitilesURL = "https://chromium.googlesource.com/chromium/src"
	// ChromiumGitilesHost is the default Gitiles host for chromium/src.
	ChromiumGitilesHost = "chromium.googlesource.com"
	// ChromiumGitilesProject is the default project name for chromium/src.
	ChromiumGitilesProject = "chromium/src"
	// ChromiumGitilesRefAtHead is the default ref used for Chromium builds.
	ChromiumGitilesRefAtHead = "refs/heads/main"
	// DefaultBucket is the Pinpoint bucket, equivalent to the "try" builds in Buildbucket.
	DefaultBucket = "try"
	// DefaultBuildsetKey is key tagged on builds for how commit information is tracked in Waterfall (CI) and Pinpoint.
	DefaultBuildsetKey = "buildset"
	// DefaultTagValue is the value format for the key above.
	DefaultBuildsetValue = "commit/gitiles/chromium.googlesource.com/chromium/src/+/%s"
	// DefaultCASInstance is the default CAS instance used by Pinpoint builds.
	//
	// TODO(b/315215756): Support other swarming instances. There are three known
	// swarming instances Pinpoint supports. The majority of Pinpoint builds are
	// this defaultInstance. Buildbucket API does not report the swarming instance
	// so our options are to:
	// - include the expected instance in the build tags
	// - try all 3 known swarming instances and brute force it
	DefaultCASInstance = "projects/chrome-swarming/instances/default_instance"
	// DefaultPerRPCTimeout defines the default time permitted for each RPC call.
	DefaultPerRPCTimeout = 90 * time.Second
	// DefaultRetries is the default number of retries for Backoff logic to Buildbucket.
	DefaultRetries = 10
	// DepsOverrideKey is the key used to find any deps overrides in the input properties from a Buildbucket response.
	DepsOverrideKey = "deps_revision_overrides"
	// SwarmingHashRefKey is the key used to find CAS hashes from successful Pinpoint Buildbucket builds.
	SwarmingHashRefKey = "swarm_hashes_refs"
	// WaterfallBucket is equivalent to the "ci" bucket in Buildbucket.
	WaterfallBucket = "ci"
)
View Source
const (
	DefaultSwarmingServiceAddress = "chrome-swarming.appspot.com:443"
	RunBenchmarkFailure           = "BENCHMARK_FAILURE" // the task completed but benchmark run failed
)
View Source
const (
	ChromiumRepo = "chromium/src"
)
View Source
const (
	DEFAULT_GITILES_SCOPE = "https://www.googleapis.com/auth/gerritcodereview"
)

Variables

View Source
var PinpointWaterfall = map[string]string{
	"Android Compile Perf":                       "android-builder-perf",
	"Android Compile Perf PGO":                   "android-builder-perf-pgo",
	"Android arm64 Compile Perf":                 "android_arm64-builder-perf",
	"Android arm64 Compile Perf PGO":             "android_arm64-builder-perf-pgo",
	"Android arm64 High End Compile Perf":        "android_arm64_high_end-builder-perf",
	"Android arm64 High End Compile Perf PGO":    "android_arm64_high_end-builder-perf-pgo",
	"Chromecast Linux Builder Perf":              "chromecast-linux-builder-perf",
	"Chromeos Amd64 Generic Lacros Builder Perf": "chromeos-amd64-generic-lacros-builder-perf",
	"Fuchsia Builder Perf":                       "fuchsia-builder-perf-arm64",
	"Linux Builder Perf":                         "linux-builder-perf",
	"Linux Builder Perf PGO":                     "linux-builder-perf-pgo",
	"Mac Builder Perf":                           "mac-builder-perf",
	"Mac Builder Perf PGO":                       "mac-builder-perf-pgo",
	"Mac arm Builder Perf":                       "mac-arm-builder-perf",
	"Mac arm Builder Perf PGO":                   "mac-arm-builder-perf-pgo",
	"mac-laptop_high_end-perf":                   "mac-laptop_high_end-perf",
	"Win x64 Builder Perf":                       "win64-builder-perf",
	"Win x64 Builder Perf PGO":                   "win64-builder-perf-pgo",
}

Builds from waterfall can also be recycled for bisection

As part of our anomaly detection system, Waterfall builders will continuously build Chrome near the tip of main. Sometimes, Pinpoint jobs will attempt to build a CL Waterfall has already built i.e. verifying a regression. These builds are automatic. Pinpoint builders will only build Chrome on demand. The Waterfall and Pinpoint builders are maintained in separate pools.

The map is maintained here: https://chromium.googlesource.com/chromium/tools/build/+/986f23767a01508ad1eb39194ffdb5fec4f00d7b/recipes/recipes/pinpoint/builder.py#22 TODO(b/316207255): move this builder map to a more stable config file

Functions

func CreateRepository

func CreateRepository(ctx context.Context, repositoryUrl string) (*gitiles.Repo, error)

CreateRepository returns an authenticated client to communicate with Gitiles.

func NewBuildbucketClient

func NewBuildbucketClient(bc bpb.BuildsClient) *buildbucketClient

func NewIssueTrackerTransport

func NewIssueTrackerTransport(ctx context.Context) (*issueTrackerTransport, error)

NewIssueTrackerTransport returns a issueTrackerTransport object configured with templates.

Types

type BuildbucketClient

type BuildbucketClient interface {
	// CancelBuild sends a cancellation request to Buildbucket. It's expected that
	// Buildbucket will cancel the build, whether that's graceful termination or
	// forced cancellation, as long as the request is received.
	CancelBuild(ctx context.Context, buildID int64, summary string) error

	// GetSingleBuild calls Buildbucket to find existing builds for the
	// given builder, Chromium revision, DEPS overrides and Gerrit patches.
	//
	// TODO(b/315215756): The current mechanism can be updated to utilize
	// tags, so that we aren't operating on O(len(builds) * len(deps_overrides))
	// to find the exact builds. This will require tagging scheduled builds with
	// new tags before it can be utilized.
	GetSingleBuild(ctx context.Context, builderName, bucket, commit string, deps map[string]string, patches []*bpb.GerritChange) (*bpb.Build, error)

	// GetBuildWithDeps search for a build with matching deps.
	// Overloaded method of GetSingleBuild().
	GetBuildWithDeps(ctx context.Context, builderName, bucket, commit string, deps map[string]string) (*bpb.Build, error)

	// GetBuildWithPatches searches for a build with matching patches.
	// Overloaded method of GetSingleBuild().
	GetBuildWithPatches(ctx context.Context, builderName, bucket, commit string, patches []*bpb.GerritChange) (*bpb.Build, error)

	// GetBuildFromWaterfall searches for an existing build using its waterfall
	// (CI) counterpart.
	GetBuildFromWaterfall(ctx context.Context, builderName, commit string) (*bpb.Build, error)

	// GetBuildStatus returns the build status given the ID.
	GetBuildStatus(ctx context.Context, buildID int64) (bpb.Status, error)

	// GetCASReference returns a CAS reference to the output artifacts of a successful build.
	GetCASReference(ctx context.Context, buildID int64, target string) (*swarmingpb.SwarmingRpcsCASReference, error)

	// StartChromeBuild triggers a Chrome build.
	StartChromeBuild(ctx context.Context, pinpointJobID, requestID, builderName, commitHash string, deps map[string]string, patches []*bpb.GerritChange) (*bpb.Build, error)
}

type BuildbucketClientConfig

type BuildbucketClientConfig struct {
	// The buildbucket host to target. See "go.skia.org/infra/go/buildbucket"
	// for the default value.
	Host string

	// Retries, if >= 0, is the number of remaining retries. If <0, no retry
	// count will be applied.
	Retries int

	// Delay is the next generated delay.
	Delay time.Duration

	// MaxDelay is the maximum duration. If <= zero, no maximum will be enforced.
	MaxDelay time.Duration

	// PerRPCTimeout, if > 0, is a timeout that is applied to each call attempt.
	PerRPCTimeout time.Duration
}

BuildbucketClientConfig represents options for the behavior of the Buildbucket client.

Example: bc := DefaultClientConfig().WithClient(c) bc.GetSingleBuild(...)

func DefaultClientConfig

func DefaultClientConfig() BuildbucketClientConfig

DefaultClientConfig returns a BuildbucketClientConfig with defaults:

  • Host: cr-buildbucket.appspot.com
  • Exponential backoff with 10 retries
  • PerRPCTimeout of 90 seconds. Swarming servers have an internal 60-second deadline to respond to requests.

func (BuildbucketClientConfig) WithClient

func (bc BuildbucketClientConfig) WithClient(c *http.Client) *buildbucketClient

WithClient returns a BuildbucketClient as configured by the ClientConfig

type CrrevClient

type CrrevClient interface {
	// GetCommitInfo returns the git hash, project, and repo of a commit
	// Supports commit hashes and positions and can return non-chromium commits.
	GetCommitInfo(ctx context.Context, commit string) (*CrrevResponse, error)
}

CrrevClient creates an API for alert grouping and Pinpoint UI to convert commit positions to git hashes before submitting them as Pinpoint jobs.

type CrrevClientImpl

type CrrevClientImpl struct {
	Client *http.Client
}

CrrevClientImpl implements CrrevClient

func NewCrrevClient

func NewCrrevClient(ctx context.Context) (*CrrevClientImpl, error)

func (*CrrevClientImpl) GetCommitInfo

func (c *CrrevClientImpl) GetCommitInfo(ctx context.Context, commit string) (*CrrevResponse, error)

type CrrevResponse

type CrrevResponse struct {
	GitHash string `json:"git_sha"`
	Project string `json:"project"`
	Repo    string `json:"repo"`
}

CrrevResponse is the response provided by the crrev redirect endpoint

type IssueTracker

type IssueTracker interface {
	ReportCulprit(issueID int64, culprits []*pinpoint_proto.CombinedCommit) error
}

type SwarmingClient

type SwarmingClient interface {
	// CancelTasks tells Swarming to cancel the given tasks.
	CancelTasks(ctx context.Context, taskIDs []string) error

	// GetCASOutput returns the CAS output of a swarming task.
	GetCASOutput(ctx context.Context, taskID string) (*swarmingV1.SwarmingRpcsCASReference, error)

	// GetStates returns the state of each task in a list of tasks.
	GetStates(ctx context.Context, taskIDs []string) ([]string, error)

	// GetStatus gets the current status of a swarming task.
	GetStatus(ctx context.Context, taskID string) (string, error)

	// ListPinpointTasks lists the Pinpoint swarming tasks.
	ListPinpointTasks(ctx context.Context, jobID string, buildArtifact *swarmingV1.SwarmingRpcsCASReference) ([]string, error)

	// TriggerTask is a literal wrapper around swarming.ApiClient TriggerTask
	// TODO(jeffyoon@) remove once run_benchmark is refactored if no longer needed.
	TriggerTask(ctx context.Context, req *swarmingV1.SwarmingRpcsNewTaskRequest) (*swarmingV1.SwarmingRpcsTaskRequestMetadata, error)

	// FetchFreeBots gets a list of available bots per specified builder configuration.
	FetchFreeBots(ctx context.Context, builder string) ([]*swarmingV1.SwarmingRpcsBotInfo, error)
}

SwarmingClient

type SwarmingClientImpl

type SwarmingClientImpl struct {
	swarming.ApiClient
}

SwarmingClientImpl TODO(jeffyoon@) make this private once run_benchmark doesn't rely on this in testing.

func NewSwarmingClient

func NewSwarmingClient(ctx context.Context, server string) (*SwarmingClientImpl, error)

func (*SwarmingClientImpl) CancelTasks

func (s *SwarmingClientImpl) CancelTasks(ctx context.Context, taskIDs []string) error

CancelTasks tells Swarming to cancel the given tasks.

func (*SwarmingClientImpl) FetchFreeBots

func (s *SwarmingClientImpl) FetchFreeBots(ctx context.Context, builder string) ([]*swarmingV1.SwarmingRpcsBotInfo, error)

FetchFreeBots gets a list of available bots per specified builder configuration.

func (*SwarmingClientImpl) GetCASOutput

GetCASOutput returns the CAS output of a swarming task in the form of a RBE CAS hash. This function assumes the task is finished, or it throws an error.

func (*SwarmingClientImpl) GetStatus

func (s *SwarmingClientImpl) GetStatus(ctx context.Context, taskID string) (string, error)

GetStatus gets the current status of a swarming task.

func (*SwarmingClientImpl) ListPinpointTasks

func (s *SwarmingClientImpl) ListPinpointTasks(ctx context.Context, jobID string, buildArtifact *swarmingV1.SwarmingRpcsCASReference) ([]string, error)

ListPinpointTasks lists the Pinpoint swarming tasks of a given job and build identified by Swarming tags.

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL