control

package
v0.0.0-...-e492904 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: May 3, 2024 License: Apache-2.0 Imports: 16 Imported by: 0

Documentation

Overview

Package control provides methods to read and write records used to:

  1. Ensure exactly-once ingestion of test results from builds.
  2. Synchronise build completion and presubmit run completion, so that ingestion only proceeds when both build and presubmit run have completed.

Index

Constants

View Source
const JoinStatsHours = 36

JoinStatsHours is the number of previous hours ReadPresubmitRunJoinStatistics/ReadBuildJoinStatistics reads statistics for.

Variables

This section is empty.

Functions

func BuildID

func BuildID(hostname string, id int64) string

BuildID returns the control record key for a buildbucket build with the given hostname and ID.

func BuildInvocationID

func BuildInvocationID(buildID int64) string

BuildInvocationID returns the invocation ID corresponding to a buildbucket build.

func BuildInvocationName

func BuildInvocationName(buildID int64) string

BuildInvocationName returns the invocation name corresponding to a buildbucket build. The pattern is originally defined here: https://source.chromium.org/chromium/infra/infra/+/main:appengine/cr-buildbucket/resultdb.py;l=75?q=build-%20resultdb&type=cs

func InsertOrUpdate

func InsertOrUpdate(ctx context.Context, e *Entry) error

InsertOrUpdate creates or updates the given ingestion record. This operation is not safe to perform blindly; perform only in a read/write transaction with an attempted read of the corresponding entry.

func ReadBuildToInvocationJoinStatistics

func ReadBuildToInvocationJoinStatistics(ctx context.Context) (map[string]JoinStatistics, error)

ReadBuildToInvocationJoinStatistics measures the performance joining builds to finalized invocations.

The statistics returned uses completed builds with an invocation as the denominator for measuring join performance. The performance joining to finalized invocations is then measured. Data is broken down by the project of the buildbucket build. The last 36 hours of data for each project is returned. Hours are measured since the buildbucket build result was received.

func ReadBuildToPresubmitRunJoinStatistics

func ReadBuildToPresubmitRunJoinStatistics(ctx context.Context) (map[string]JoinStatistics, error)

ReadBuildToPresubmitRunJoinStatistics measures the performance joining builds to presubmit runs.

The statistics returned uses completed builds with a presubmit run as the denominator for measuring join performance. The performance joining to presubmit run results is then measured. Data is broken down by the project of the buildbucket build. The last 36 hours of data for each project is returned. Hours are measured since the buildbucket build result was received.

func ReadInvocationToBuildJoinStatistics

func ReadInvocationToBuildJoinStatistics(ctx context.Context) (map[string]JoinStatistics, error)

ReadInvocationToBuildJoinStatistics measures the performance joining finalized invocations to builds.

The statistics returned uses finalized invocations (for buildbucket builds) as the denominator for measuring join performance. The performance joining to buildbucket build results is then measured. Data is broken down by the project of the ingested invocation (this should be the same as the ingested build, although it comes from a different source). The last 36 hours of data for each project is returned. Hours are measured since the finalized invocation was received.

func ReadPresubmitToBuildJoinStatistics

func ReadPresubmitToBuildJoinStatistics(ctx context.Context) (map[string]JoinStatistics, error)

ReadPresubmitToBuildJoinStatistics measures the performance joining presubmit runs to builds.

The statistics returned uses builds as reported by completed presubmit runs as the denominator for measuring join performance. The performance joining to buildbucket build results is then measured. Data is broken down by the project of the presubmit run. The last 36 hours of data for each project is returned. Hours are measured since the presubmit run result was received.

func SetEntriesForTesting

func SetEntriesForTesting(ctx context.Context, es ...*Entry) (time.Time, error)

SetEntriesForTesting replaces the set of stored entries to match the given set.

func ValidateBuildResult

func ValidateBuildResult(r *ctlpb.BuildResult) error

func ValidatePresubmitResult

func ValidatePresubmitResult(r *ctlpb.PresubmitResult) error

Types

type Entry

type Entry struct {
	// The identity of the build which is being ingested.
	// The scheme is: {buildbucket host name}/{build id}.
	BuildID string

	// Project is the LUCI Project the build belongs to. Used for
	// metrics monitoring join performance.
	BuildProject string

	// BuildResult is the result of the build bucket build, to be passed
	// to the result ingestion task. This is nil if the result is
	// not yet known.
	BuildResult *ctlpb.BuildResult

	// BuildJoinedTime is the Spanner commit time the build result was
	// populated. If the result has not yet been populated, this is the zero time.
	BuildJoinedTime time.Time

	// HasInvocation records wether the build has an associated (ResultDB)
	// invocation.
	// Value only populated once either BuildResult or InvocationResult populated.
	HasInvocation bool

	// Project is the LUCI Project the invocation belongs to. Used for
	// metrics monitoring join performance.
	InvocationProject string

	// InvocationResult is the result of the invocation, to be passed
	// to the result ingestion task. This is nil if the result is
	// not yet known.
	InvocationResult *ctlpb.InvocationResult

	// InvocationJoinedTime is the Spanner commit time the invocation result
	// was populated. If the result has not yet been populated, this is the zero time.
	InvocationJoinedTime time.Time

	// IsPresubmit records whether the build is part of a presubmit run.
	// If true, ingestion should wait for the presubmit result to be
	// populated (in addition to the build result) before commencing
	// ingestion.
	// Value only populated once either BuildResult or PresubmitResult populated.
	IsPresubmit bool

	// PresubmitProject is the LUCI Project the presubmit run belongs to.
	// This may differ from the LUCI Project teh build belongs to. Used for
	// metrics monitoring join performance.
	PresubmitProject string

	// PresubmitResult is result of the presubmit run, to be passed to the
	// result ingestion task. This is nil if the result is not yet known.
	PresubmitResult *ctlpb.PresubmitResult

	// PresubmitJoinedTime is the Spanner commit time the presubmit result was
	// populated. If the result has not yet been populated, this is the zero time.
	PresubmitJoinedTime time.Time

	// LastUpdated is the Spanner commit time the row was last updated.
	LastUpdated time.Time

	// The number of test result ingestion tasks have been created for this
	// invocation.
	// Used to avoid duplicate scheduling of ingestion tasks. If the page_index
	// is the index of the page being processed, an ingestion task for the next
	// page will only be created if (page_index + 1) == TaskCount.
	TaskCount int64
}

Entry is an ingestion control record, used to de-duplicate build ingestions and synchronise them with presubmit results (if required).

func Read

func Read(ctx context.Context, buildIDs []string) ([]*Entry, error)

Read reads ingestion control records for the specified build IDs. Exactly one *Entry is returned for each build ID. The result entry at index i corresponds to the buildIDs[i]. If a record does not exist for the given build ID, an *Entry of nil is returned for that build ID.

type EntryBuilder

type EntryBuilder struct {
	// contains filtered or unexported fields
}

EntryBuilder provides methods to build ingestion control records.

func NewEntry

func NewEntry(uniqifier int) *EntryBuilder

NewEntry starts building a new Entry.

func (*EntryBuilder) Build

func (b *EntryBuilder) Build() *Entry

Build constructs the entry.

func (*EntryBuilder) WithBuildID

func (b *EntryBuilder) WithBuildID(id string) *EntryBuilder

WithBuildID specifies the build ID to use on the ingestion control record.

func (*EntryBuilder) WithBuildJoinedTime

func (b *EntryBuilder) WithBuildJoinedTime(value time.Time) *EntryBuilder

WithBuildJoinedTime specifies the time the build result was populated.

func (*EntryBuilder) WithBuildProject

func (b *EntryBuilder) WithBuildProject(project string) *EntryBuilder

WithBuildProject specifies the build project to use on the ingestion control record.

func (*EntryBuilder) WithBuildResult

func (b *EntryBuilder) WithBuildResult(value *controlpb.BuildResult) *EntryBuilder

WithBuildResult specifies the build result for the entry.

func (*EntryBuilder) WithHasInvocation

func (b *EntryBuilder) WithHasInvocation(hasInvocation bool) *EntryBuilder

WithHasInvocation specifies whether the build that is the subject of the ingestion has a ResultDB invocation.

func (*EntryBuilder) WithInvocationJoinedTime

func (b *EntryBuilder) WithInvocationJoinedTime(value time.Time) *EntryBuilder

WithInvocationJoinedTime specifies the time the invocation result was populated.

func (*EntryBuilder) WithInvocationProject

func (b *EntryBuilder) WithInvocationProject(project string) *EntryBuilder

WithInvocationProject specifies the invocation project to use on the ingestion control record.

func (*EntryBuilder) WithInvocationResult

func (b *EntryBuilder) WithInvocationResult(value *controlpb.InvocationResult) *EntryBuilder

WithInvocationResult specifies the invocation result for the entry.

func (*EntryBuilder) WithIsPresubmit

func (b *EntryBuilder) WithIsPresubmit(isPresubmit bool) *EntryBuilder

WithIsPresubmit specifies whether the ingestion relates to a presubmit run.

func (*EntryBuilder) WithPresubmitJoinedTime

func (b *EntryBuilder) WithPresubmitJoinedTime(lastUpdated time.Time) *EntryBuilder

WithPresubmitJoinedTime specifies the time the presubmit result was populated.

func (*EntryBuilder) WithPresubmitProject

func (b *EntryBuilder) WithPresubmitProject(project string) *EntryBuilder

WithPresubmitProject specifies the presubmit project to use on the ingestion control record.

func (*EntryBuilder) WithPresubmitResult

func (b *EntryBuilder) WithPresubmitResult(value *controlpb.PresubmitResult) *EntryBuilder

WithPresubmitResult specifies the build result for the entry.

func (*EntryBuilder) WithTaskCount

func (b *EntryBuilder) WithTaskCount(taskCount int64) *EntryBuilder

type JoinStatistics

type JoinStatistics struct {
	// TotalByHour captures the number of builds in the ingestions
	// table eligible to be joined (i.e. have the left-hand join input).
	//
	// Data is broken down by by hours since the build became
	// eligible for joining. Index 0 indicates the period
	// from ]-1 hour, now], index 1 indicates [-2 hour, -1 hour] and so on.
	TotalByHour []int64

	// JoinedByHour captures the number of builds in the ingestions
	// table eligible to be joined, which were successfully joined (have
	// results for both join inputs present).
	//
	// Data is broken down by by hours since the build became
	// eligible for joining. Index 0 indicates the period
	// from ]-1 hour, now], index 1 indicates [-2 hour, -1 hour] and so on.
	JoinedByHour []int64
}

JoinStatistics captures indicators of how well two join inputs (e.g. buildbucket build completions and presubmit run completions, or buildbucket build completions and invocation finalizations) are being joined.

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL