Documentation
¶
Overview ¶
Package common contains widely used CV utilities & types.
Index ¶
- Constants
- Variables
- func CLIDsAsInt64s(ids []CLID) []int64
- func DifferenceSorted(a, b []int64) []int64
- func DistributeOffset(pollInterval time.Duration, keyParts ...string) time.Duration
- func IsDatastoreContention(err error) bool
- func IsInstantTriggerDogfooder(ctx context.Context, id identity.Identity) bool
- func LogError(ctx context.Context, err error, expectedErrors ...error)
- func MostSevereError(err error) error
- func PB2TimeNillable(pb *timestamppb.Timestamp) time.Time
- func TQifyError(ctx context.Context, err error) error
- func Time2PBNillable(t time.Time) *timestamppb.Timestamp
- func UnionSorted(a, b []int64) []int64
- func UniqueSorted(v []int64) []int64
- type CLID
- type CLIDs
- type CLIDsSet
- func (s CLIDsSet) Add(clid CLID)
- func (s CLIDsSet) AddI64(id int64)
- func (s CLIDsSet) Del(id CLID)
- func (s CLIDsSet) DelAll(ids CLIDs)
- func (s CLIDsSet) DelI64(id int64)
- func (s CLIDsSet) Has(clid CLID) bool
- func (s CLIDsSet) HasI64(id int64) bool
- func (s CLIDsSet) Reset(ids ...CLID)
- func (s CLIDsSet) ResetI64(ids ...int64)
- func (s CLIDsSet) ToCLIDs() CLIDs
- type Env
- type RunID
- type RunIDs
- func (ids RunIDs) ContainsSorted(id RunID) bool
- func (p *RunIDs) DelSorted(id RunID) bool
- func (a RunIDs) DifferenceSorted(b RunIDs) RunIDs
- func (ids RunIDs) Equal(other RunIDs) bool
- func (ids RunIDs) Index(target RunID) int
- func (p *RunIDs) InsertSorted(id RunID)
- func (ids RunIDs) Len() int
- func (ids RunIDs) Less(i, j int) bool
- func (ids RunIDs) Set() map[RunID]struct{}
- func (ids RunIDs) Swap(i, j int)
- func (ids RunIDs) WithoutSorted(exclude RunIDs) RunIDs
- type TQIfy
- type TryjobID
- type TryjobIDSet
- type TryjobIDs
Constants ¶
const ( // MaxTriggerAge limits CV operation only to most recently triggered CLs. // // In particular, this reduces Gerrit querying to CLs modified since // (now-MaxTriggerAge). MaxTriggerAge = 24 * 7 * time.Hour // LaunchRetryClockTag tags the clock to coordinate with the test timer // call back in cvtesting. LaunchRetryClockTag = "retry-launch" )
const ( // builder. Meaning always launch the main builder if possible. FooterNoEquivalentBuilders = "No-Equivalent-Builders" // FooterNoEquivalentBuilders FooterLegacyNoEquivalentBuilders = "NO_EQUIVALENT_BUILDERS" FooterCQDoNotCancelTryjobs = "Cq-Do-Not-Cancel-Tryjobs" FooterNoTreeChecks = "No-Tree-Checks" FooterLegacyNoTreeChecks = "NOTREECHECKS" FooterNoTry = "No-Try" FooterLegacyNoTry = "NOTRY" // // CAVEAT: crbug.com/1292195 - It use `disable_reuse` field to decide // whether a Tryjob is Presubmit which produce false positive. FooterNoPresubmit = "No-Presubmit" FooterLegacyPresubmit = "NOPRESUBMIT" FooterCQIncludeTryjobs = "Cq-Include-Trybots" // FooterCQIncludeTryjobs. FooterLegacyCQIncludeTryjobs = "CQ_INCLUDE_TRYBOTS" // overrides ALL the Tryjobs supposed to be launched according to the // configuration. FooterOverrideTryjobsForAutomation = "Override-Tryjobs-For-Automation" // launched Tryjobs. FooterCQClTag = "Cq-Cl-Tag" )
const InstantTriggerDogfooderGroup = "luci-cv-instant-trigger-dogfooders"
InstantTriggerDogfooderGroup is the CrIA group who signed up for dogfooding cros instant trigger.
const MaxRunTotalDuration = 10 * 24 * time.Hour // 10 days
MaxRunTotalDuration is the max total duration of the Run.
Total duration means end time - create time. Run will be cancelled after the total duration is reached.
const RunKind = "Run"
RunKind is the Datastore entity kind for Run.
Variables ¶
var DSContentionTag = errors.BoolTag{Key: errors.NewTagKey("Datastore Contention")}
DSContentionTag when set indicates Datastore contention.
It's set on errors by parts of CV which are especially prone to DS contention to reduce noise in logs and for more effective retries.
Functions ¶
func CLIDsAsInt64s ¶
CLIDsAsInt64s returns proto representation of CLIDs.
func DifferenceSorted ¶
DifferenceSorted returns all int64s in the first slice and not the second.
Both slices must be sorted. Doesn't modify input slices.
func DistributeOffset ¶
DistributeOffset deterministically chooses an offset across keys in [1..pollInterval) range aimining for uniform distribution across all keys.
A key is simply a concatenation of key parts with '\0' filler in between.
func IsDatastoreContention ¶
IsDatastoreContention is best-effort detection of transactions aborted due to pessimistic concurrency control of Datastore backed by Firestore.
This is fragile, because it relies on undocumented but likely rarely changed English description of an error.
func IsInstantTriggerDogfooder ¶
IsInstantTriggerDogfooder returns true if the given user participate in the cros instant trigger dogfood.
TODO(yiwzhang): remove this function, once cros instant trigger dogfood is done.
func LogError ¶
LogError is errors.Log with CV-specific package filtering.
Logs entire error stack with ERROR severity by default. Logs just error with WARNING severity iff one of error (or its inner error) equal at least one of the given list of `expectedErrors` errors. This is useful if TQ handler is known to frequently fail this way.
expectedErrors must contain only unwrapped errors.
func MostSevereError ¶
MostSevereError returns the most severe error in order of non-transient => transient => nil.
Walks over potentially recursive errors.MultiError errors only.
Returns only singular errors or nil if input was nil.
func PB2TimeNillable ¶
func PB2TimeNillable(pb *timestamppb.Timestamp) time.Time
PB2TimeNillable is the opposite of Time2PBNillable.
ie same as pb.AsTime() but returns zero time if pb is nil.
func TQifyError ¶
TQifyError is shortcut for TQIfy{}.Error.
func Time2PBNillable ¶
func Time2PBNillable(t time.Time) *timestamppb.Timestamp
Time2PBNillable is like timestamppb.New() but returns nil on zero time.
func UnionSorted ¶
UnionSorted returns sorted unique int64s from two slices.
Both slices must be sorted and unique. Doesn't modify input slices.
func UniqueSorted ¶
UniqueSorted sorts & removes duplicates in place.
Returns the potentially shorter slice.
Types ¶
type CLID ¶
type CLID int64
CLID is a unique ID of a CL used internally in CV.
It's just 8 bytes long and is thus much shorter than ExternalID, which reduces CPU & RAM & storage costs of CL graphs for multi-CL Runs.
type CLIDs ¶
type CLIDs []CLID
CLIDs is a convenience type to facilitate handling of a slice of CLID.
func (*CLIDs) Dedupe ¶
func (p *CLIDs) Dedupe()
Dedupe removes duplicates in place and sorts the slice.
Note: Does not preserve original order.
type CLIDsSet ¶
type CLIDsSet map[CLID]struct{}
CLIDsSet is convenience type to reduce the boilerplate.
func MakeCLIDsSet ¶
MakeCLIDsSet returns new CLIDsSet from list of clids in int64.
type Env ¶
type Env struct {
// LogicalHostname is CV hostname referred to in configs.
//
// On GAE, this is something like "luci-change-verifier-dev.appspot.com"
// and it is part of the HTTPAddressBase.
//
// Under local development, this is usually set to GAE-looking hostname, while
// keeping HTTPAddressBase a real localhost URL.
LogicalHostname string
// HTTPAddressBase can be used to generate URLs to this CV service.
//
// Doesn't have a trailing slash.
//
// For example,
// * "https://luci-change-verifier-dev.appspot.com"
// * "http://localhost:8080"
HTTPAddressBase string
// IsGAEDev is true if this is a -dev GAE environment.
//
// Deprecated. Do not use in new code. It should only be used during migration
// from CQDaemon which doesn't have equivalent -dev environment.
IsGAEDev bool
// GAEInfo is populated if LUCI CV runs on GAE.
GAEInfo struct {
// CloudProject is the name of the Google Cloud Project LUCI CV runs in.
CloudProject string
// ServiceName is the name of the micro-service in the GAE app.
ServiceName string
// InstanceID is the ID of the instance that runs LUCI CV.
InstanceID string
}
}
Env describes where CV runs at.
type RunID ¶
type RunID string
RunID is an unique RunID to identify a Run in CV.
RunID is string like `luciProject/inverseTS-1-hexHashDigest` consisting of 7 parts:
- The LUCI Project that this Run belongs to. Purpose: separates load on Datastore from different projects.
- `/` separator.
- InverseTS, defined as (`endOfTheWorld` - CreateTime) in ms precision, left-padded with zeros to 13 digits. See `Run.CreateTime` Doc. Purpose: ensures queries by default orders runs of the same project by most recent first.
- `-` separator.
- Digest version (see part 7).
- `-` separator.
- A hex digest string uniquely identifying the set of CLs involved in this Run. Purpose: ensures two simultaneously started Runs in the same project won't have the same RunID.
func FromPublicRunID ¶
FromPublicRunID is the inverse of RunID.PublicID().
func (RunID) AttemptKey ¶
AttemptKey returns CQDaemon attempt key.
func (RunID) PublicID ¶
PublicID returns the public representation of the RunID.
The format of a public ID is `projects/$luci-project/runs/$id`, where - luci-project is the name of the LUCI project the Run belongs to - id is an opaque key unique in the LUCI project.
func (RunID) Validate ¶
Validate returns an error if Run ID is not valid.
If validate returns nil,
- it means all other methods on RunID will work fine instead of panicking,
- it doesn't mean Run ID is possible to generate using the MakeRunID. This is especially relevant in CV tests, where specifying short Run IDs is useful.
type RunIDs ¶
type RunIDs []RunID
RunIDs is a convenience type to facilitate handling of run RunIDs.
func MakeRunIDs ¶
MakeRunIDs returns RunIDs from list of strings.
func (RunIDs) ContainsSorted ¶
ContainsSorted returns true if ids contain the given one.
func (*RunIDs) DelSorted ¶
DelSorted removes the given ID if it exists.
DelSorted is a pointer receiver method, because it modifies slice itself.
func (RunIDs) DifferenceSorted ¶
DifferenceSorted returns all IDs in this slice and not the other one.
Both slices must be sorted. Doesn't modify input slices.
func (RunIDs) Index ¶
Index returns the index of the first instance of the provided id.
Returns -1 if the provided id isn't present.
func (*RunIDs) InsertSorted ¶
InsertSorted adds given ID if not yet exists to the list keeping list sorted.
InsertSorted is a pointer receiver method, because it modifies slice itself.
func (RunIDs) WithoutSorted ¶
WithoutSorted returns a subsequence of IDs without excluded IDs.
Both this and the excluded slices must be sorted.
If this and excluded IDs are disjoint, return this slice. Otherwise, returns a copy without excluded IDs.
type TQIfy ¶
type TQIfy struct {
// KnownRetry are expected errors which will result in HTTP 429 and retries.
//
// Retries may not happen if task queue configuration prevents it, e.g.
// because task has exhausted its retry quota.
//
// KnownRetry and KnownIgnore should not match the same error, but if this
// happens, Retry takes effect and KnownIgnore is ignored to avoid accidental
// loss of tasks.
//
// Must contain only leaf errors, i.e. no annotated or MultiError objects.
KnownRetry []error
// KnownRetryTags are similar to `KnowRetry`, but are the expected tags that
// the CV error should be tagged with.
//
// Must not contain `transient.Tag`.
KnownRetryTags []errors.BoolTag
// NeverRetry instructs TQ not to retry on any unexpected error.
//
// Transient error will be tagged with `tq.Ignore` while non-transient error
// will be tagged with `tq.Fatal`. See the struct doc for what each tag means.
//
// Recommend to use this flag when tasks are executed periodically in short
// interval (e.g. refresh config task) where as retrying failed task is not
// necessary.
//
// Mutually exclusive with `KnownRetry` and `KnownRetryTags`.
NeverRetry bool
// KnownIgnore are expected errors which will result in HTTP 204 and no
// retries.
//
// Must contain only leaf errors, i.e. no annotated or MultiError objects.
KnownIgnore []error
// KnownIgnoreTags are similar to `KnownIgnore`, but are the expected tags
// that the CV error should be tagged with.
//
// Must not contain `transient.Tag`.
KnownIgnoreTags []errors.BoolTag
}
TQIfy converts CV error semantics to server/TQ, and logs error if necessary.
Usage:
func tqHandler(ctx ..., payload...) error {
err := doStuff(ctx, ...)
return TQIfy{}.Error(ctx, err)
}
Given that:
- TQ lib recognizes these error kinds:
- tq.Ignore => HTTP 204, no retries
- tq.Fatal => HTTP 202, no retries, but treated with alertable in our monitoring configuration;
- transient.Tag => HTTP 500, will be retried;
- else => HTTP 429, will be retried.
OTOH, CV uses
- transient.Tag to treat all _transient_ situations, where retry should help
- else => permanent errors, where retries aren't helpful.
Most _transient_ situations in CV are due to expected issues such as Gerrit giving stale data. Getting HTTP 500s in this case is an unfortunate noise, which obscures other infrequent situations which are worth looking at.
type TryjobID ¶
type TryjobID int64
TryjobID is a unique ID of a Tryjob used internally in CV.
This ID is not a Buildbucket Build ID. See also tryjob.Tryjob type.
type TryjobIDSet ¶
type TryjobIDSet map[TryjobID]struct{}
TryjobIDSet is convenience type to reduce the boilerplate.
func (TryjobIDSet) Add ¶
func (s TryjobIDSet) Add(tjID TryjobID)
Add adds the provided Tryjob ID to the set.
func (TryjobIDSet) Has ¶
func (s TryjobIDSet) Has(tjID TryjobID) bool
Has returns true if the provided Tryjob ID is in the set.
Otherwise, returns false.
type TryjobIDs ¶
type TryjobIDs []TryjobID
TryjobIDs is a convenience type to facilitate handling of a slice of TryjobID.
func MakeTryjobIDs ¶
MakeTryjobIDs returns TryjobIDs from list of TryjobID in int64.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Package bq handles sending rows to BigQuery.
|
Package bq handles sending rows to BigQuery. |
|
Package eventbox batches incoming events for a single Datastore entity for processing.
|
Package eventbox batches incoming events for a single Datastore entity for processing. |
|
dsset
Package dsset implements a particular flavor of Datastore-on-Firestore backed set.
|
Package dsset implements a particular flavor of Datastore-on-Firestore backed set. |
|
Package lease provides a way to "lock" an external resource with expiration time so that concurrent processes/task executions can achieve exclusive privilege to make mutations (generally long-running and non-idempotent) on that resource.
|
Package lease provides a way to "lock" an external resource with expiration time so that concurrent processes/task executions can achieve exclusive privilege to make mutations (generally long-running and non-idempotent) on that resource. |
|
Package pubsub provides a generic way to batch pubsub pull notifications.
|
Package pubsub provides a generic way to batch pubsub pull notifications. |
|
Package tree implements fetching tree status from Tree Status App.
|
Package tree implements fetching tree status from Tree Status App. |
|
treetest
Package treetest implements fake Tree for testing in CV.
|
Package treetest implements fake Tree for testing in CV. |