Documentation ¶
Overview ¶
Package control provides methods to read and write records used to:
- Ensure exactly-once ingestion of test results from builds.
- Synchronise build completion and presubmit run completion, so that ingestion only proceeds when both build and presubmit run have completed.
Index ¶
- Constants
- func BuildID(hostname string, id int64) string
- func BuildInvocationID(buildID int64) string
- func BuildInvocationName(buildID int64) string
- func InsertOrUpdate(ctx context.Context, e *Entry) error
- func ReadBuildToInvocationJoinStatistics(ctx context.Context) (map[string]JoinStatistics, error)
- func ReadBuildToPresubmitRunJoinStatistics(ctx context.Context) (map[string]JoinStatistics, error)
- func ReadInvocationToBuildJoinStatistics(ctx context.Context) (map[string]JoinStatistics, error)
- func ReadPresubmitToBuildJoinStatistics(ctx context.Context) (map[string]JoinStatistics, error)
- func SetEntriesForTesting(ctx context.Context, es ...*Entry) (time.Time, error)
- func ValidateBuildResult(r *ctlpb.BuildResult) error
- func ValidatePresubmitResult(r *ctlpb.PresubmitResult) error
- type Entry
- type EntryBuilder
- func (b *EntryBuilder) Build() *Entry
- func (b *EntryBuilder) WithBuildID(id string) *EntryBuilder
- func (b *EntryBuilder) WithBuildJoinedTime(value time.Time) *EntryBuilder
- func (b *EntryBuilder) WithBuildProject(project string) *EntryBuilder
- func (b *EntryBuilder) WithBuildResult(value *controlpb.BuildResult) *EntryBuilder
- func (b *EntryBuilder) WithHasInvocation(hasInvocation bool) *EntryBuilder
- func (b *EntryBuilder) WithInvocationJoinedTime(value time.Time) *EntryBuilder
- func (b *EntryBuilder) WithInvocationProject(project string) *EntryBuilder
- func (b *EntryBuilder) WithInvocationResult(value *controlpb.InvocationResult) *EntryBuilder
- func (b *EntryBuilder) WithIsPresubmit(isPresubmit bool) *EntryBuilder
- func (b *EntryBuilder) WithPresubmitJoinedTime(lastUpdated time.Time) *EntryBuilder
- func (b *EntryBuilder) WithPresubmitProject(project string) *EntryBuilder
- func (b *EntryBuilder) WithPresubmitResult(value *controlpb.PresubmitResult) *EntryBuilder
- func (b *EntryBuilder) WithTaskCount(taskCount int64) *EntryBuilder
- type JoinStatistics
Constants ¶
const JoinStatsHours = 36
JoinStatsHours is the number of previous hours ReadPresubmitRunJoinStatistics/ReadBuildJoinStatistics reads statistics for.
Variables ¶
This section is empty.
Functions ¶
func BuildID ¶
BuildID returns the control record key for a buildbucket build with the given hostname and ID.
func BuildInvocationID ¶
BuildInvocationID returns the invocation ID corresponding to a buildbucket build.
func BuildInvocationName ¶
BuildInvocationName returns the invocation name corresponding to a buildbucket build. The pattern is originally defined here: https://source.chromium.org/chromium/infra/infra/+/main:appengine/cr-buildbucket/resultdb.py;l=75?q=build-%20resultdb&type=cs
func InsertOrUpdate ¶
InsertOrUpdate creates or updates the given ingestion record. This operation is not safe to perform blindly; perform only in a read/write transaction with an attempted read of the corresponding entry.
func ReadBuildToInvocationJoinStatistics ¶
func ReadBuildToInvocationJoinStatistics(ctx context.Context) (map[string]JoinStatistics, error)
ReadBuildToInvocationJoinStatistics measures the performance joining builds to finalized invocations.
The statistics returned uses completed builds with an invocation as the denominator for measuring join performance. The performance joining to finalized invocations is then measured. Data is broken down by the project of the buildbucket build. The last 36 hours of data for each project is returned. Hours are measured since the buildbucket build result was received.
func ReadBuildToPresubmitRunJoinStatistics ¶
func ReadBuildToPresubmitRunJoinStatistics(ctx context.Context) (map[string]JoinStatistics, error)
ReadBuildToPresubmitRunJoinStatistics measures the performance joining builds to presubmit runs.
The statistics returned uses completed builds with a presubmit run as the denominator for measuring join performance. The performance joining to presubmit run results is then measured. Data is broken down by the project of the buildbucket build. The last 36 hours of data for each project is returned. Hours are measured since the buildbucket build result was received.
func ReadInvocationToBuildJoinStatistics ¶
func ReadInvocationToBuildJoinStatistics(ctx context.Context) (map[string]JoinStatistics, error)
ReadInvocationToBuildJoinStatistics measures the performance joining finalized invocations to builds.
The statistics returned uses finalized invocations (for buildbucket builds) as the denominator for measuring join performance. The performance joining to buildbucket build results is then measured. Data is broken down by the project of the ingested invocation (this should be the same as the ingested build, although it comes from a different source). The last 36 hours of data for each project is returned. Hours are measured since the finalized invocation was received.
func ReadPresubmitToBuildJoinStatistics ¶
func ReadPresubmitToBuildJoinStatistics(ctx context.Context) (map[string]JoinStatistics, error)
ReadPresubmitToBuildJoinStatistics measures the performance joining presubmit runs to builds.
The statistics returned uses builds as reported by completed presubmit runs as the denominator for measuring join performance. The performance joining to buildbucket build results is then measured. Data is broken down by the project of the presubmit run. The last 36 hours of data for each project is returned. Hours are measured since the presubmit run result was received.
func SetEntriesForTesting ¶
SetEntriesForTesting replaces the set of stored entries to match the given set.
func ValidateBuildResult ¶
func ValidateBuildResult(r *ctlpb.BuildResult) error
func ValidatePresubmitResult ¶
func ValidatePresubmitResult(r *ctlpb.PresubmitResult) error
Types ¶
type Entry ¶
type Entry struct { // The identity of the build which is being ingested. // The scheme is: {buildbucket host name}/{build id}. BuildID string // Project is the LUCI Project the build belongs to. Used for // metrics monitoring join performance. BuildProject string // BuildResult is the result of the build bucket build, to be passed // to the result ingestion task. This is nil if the result is // not yet known. BuildResult *ctlpb.BuildResult // BuildJoinedTime is the Spanner commit time the build result was // populated. If the result has not yet been populated, this is the zero time. BuildJoinedTime time.Time // HasInvocation records wether the build has an associated (ResultDB) // invocation. // Value only populated once either BuildResult or InvocationResult populated. HasInvocation bool // Project is the LUCI Project the invocation belongs to. Used for // metrics monitoring join performance. InvocationProject string // InvocationResult is the result of the invocation, to be passed // to the result ingestion task. This is nil if the result is // not yet known. InvocationResult *ctlpb.InvocationResult // InvocationJoinedTime is the Spanner commit time the invocation result // was populated. If the result has not yet been populated, this is the zero time. InvocationJoinedTime time.Time // IsPresubmit records whether the build is part of a presubmit run. // If true, ingestion should wait for the presubmit result to be // populated (in addition to the build result) before commencing // ingestion. // Value only populated once either BuildResult or PresubmitResult populated. IsPresubmit bool // PresubmitProject is the LUCI Project the presubmit run belongs to. // This may differ from the LUCI Project teh build belongs to. Used for // metrics monitoring join performance. PresubmitProject string // PresubmitResult is result of the presubmit run, to be passed to the // result ingestion task. This is nil if the result is not yet known. PresubmitResult *ctlpb.PresubmitResult // PresubmitJoinedTime is the Spanner commit time the presubmit result was // populated. If the result has not yet been populated, this is the zero time. PresubmitJoinedTime time.Time // LastUpdated is the Spanner commit time the row was last updated. LastUpdated time.Time // The number of test result ingestion tasks have been created for this // invocation. // Used to avoid duplicate scheduling of ingestion tasks. If the page_index // is the index of the page being processed, an ingestion task for the next // page will only be created if (page_index + 1) == TaskCount. TaskCount int64 }
Entry is an ingestion control record, used to de-duplicate build ingestions and synchronise them with presubmit results (if required).
type EntryBuilder ¶
type EntryBuilder struct {
// contains filtered or unexported fields
}
EntryBuilder provides methods to build ingestion control records.
func (*EntryBuilder) WithBuildID ¶
func (b *EntryBuilder) WithBuildID(id string) *EntryBuilder
WithBuildID specifies the build ID to use on the ingestion control record.
func (*EntryBuilder) WithBuildJoinedTime ¶
func (b *EntryBuilder) WithBuildJoinedTime(value time.Time) *EntryBuilder
WithBuildJoinedTime specifies the time the build result was populated.
func (*EntryBuilder) WithBuildProject ¶
func (b *EntryBuilder) WithBuildProject(project string) *EntryBuilder
WithBuildProject specifies the build project to use on the ingestion control record.
func (*EntryBuilder) WithBuildResult ¶
func (b *EntryBuilder) WithBuildResult(value *controlpb.BuildResult) *EntryBuilder
WithBuildResult specifies the build result for the entry.
func (*EntryBuilder) WithHasInvocation ¶
func (b *EntryBuilder) WithHasInvocation(hasInvocation bool) *EntryBuilder
WithHasInvocation specifies whether the build that is the subject of the ingestion has a ResultDB invocation.
func (*EntryBuilder) WithInvocationJoinedTime ¶
func (b *EntryBuilder) WithInvocationJoinedTime(value time.Time) *EntryBuilder
WithInvocationJoinedTime specifies the time the invocation result was populated.
func (*EntryBuilder) WithInvocationProject ¶
func (b *EntryBuilder) WithInvocationProject(project string) *EntryBuilder
WithInvocationProject specifies the invocation project to use on the ingestion control record.
func (*EntryBuilder) WithInvocationResult ¶
func (b *EntryBuilder) WithInvocationResult(value *controlpb.InvocationResult) *EntryBuilder
WithInvocationResult specifies the invocation result for the entry.
func (*EntryBuilder) WithIsPresubmit ¶
func (b *EntryBuilder) WithIsPresubmit(isPresubmit bool) *EntryBuilder
WithIsPresubmit specifies whether the ingestion relates to a presubmit run.
func (*EntryBuilder) WithPresubmitJoinedTime ¶
func (b *EntryBuilder) WithPresubmitJoinedTime(lastUpdated time.Time) *EntryBuilder
WithPresubmitJoinedTime specifies the time the presubmit result was populated.
func (*EntryBuilder) WithPresubmitProject ¶
func (b *EntryBuilder) WithPresubmitProject(project string) *EntryBuilder
WithPresubmitProject specifies the presubmit project to use on the ingestion control record.
func (*EntryBuilder) WithPresubmitResult ¶
func (b *EntryBuilder) WithPresubmitResult(value *controlpb.PresubmitResult) *EntryBuilder
WithPresubmitResult specifies the build result for the entry.
func (*EntryBuilder) WithTaskCount ¶
func (b *EntryBuilder) WithTaskCount(taskCount int64) *EntryBuilder
type JoinStatistics ¶
type JoinStatistics struct { // TotalByHour captures the number of builds in the ingestions // table eligible to be joined (i.e. have the left-hand join input). // // Data is broken down by by hours since the build became // eligible for joining. Index 0 indicates the period // from ]-1 hour, now], index 1 indicates [-2 hour, -1 hour] and so on. TotalByHour []int64 // JoinedByHour captures the number of builds in the ingestions // table eligible to be joined, which were successfully joined (have // results for both join inputs present). // // Data is broken down by by hours since the build became // eligible for joining. Index 0 indicates the period // from ]-1 hour, now], index 1 indicates [-2 hour, -1 hour] and so on. JoinedByHour []int64 }
JoinStatistics captures indicators of how well two join inputs (e.g. buildbucket build completions and presubmit run completions, or buildbucket build completions and invocation finalizations) are being joined.