luci: go.chromium.org/luci/dm/appengine/model Index | Files

package model

import "go.chromium.org/luci/dm/appengine/model"

Package model contains all of DM's datastore models.

Index

Package Files

attempt.go attempt_result.go backdep.go doc.go edge.go execution.go fwddep.go keys.go quest.go template_info.go

func ActivateExecution Uses

func ActivateExecution(c context.Context, auth *dm.Execution_Auth, actToken []byte) (a *Attempt, e *Execution, err error)

ActivateExecution validates that the execution is unactivated and that the activation token matches and then sets the token to the new value.

It's OK to retry this. Subsequent invocations with the same Token will recognize this case and not return an error.

func AttemptIDFromKey Uses

func AttemptIDFromKey(k *ds.Key) *dm.Attempt_ID

AttemptIDFromKey makes a AttemptID from the given datastore.Key. It panics if the Key does not point to a Attempt.

func AttemptKeyFromID Uses

func AttemptKeyFromID(c context.Context, aid *dm.Attempt_ID) *ds.Key

AttemptKeyFromID makes a datastore.Key given the AttemptID.

func AuthenticateExecution Uses

func AuthenticateExecution(c context.Context, auth *dm.Execution_Auth) (a *Attempt, e *Execution, err error)

AuthenticateExecution verifies that the Attempt is executing, and that evkey matches the execution key of the current Execution for this Attempt.

As a bonus, it will return the loaded Attempt and Execution.

func ExecutionIDFromKey Uses

func ExecutionIDFromKey(k *ds.Key) *dm.Execution_ID

ExecutionIDFromKey makes a ExecutionID from the given datastore.Key. It panics if the Key does not point to a Execution.

func ExecutionKeyFromID Uses

func ExecutionKeyFromID(c context.Context, eid *dm.Execution_ID) *ds.Key

ExecutionKeyFromID makes a datastore.Key given the ExecutionID.

func FwdDepKeysFromList Uses

func FwdDepKeysFromList(c context.Context, base *dm.Attempt_ID, list *dm.AttemptList) []*ds.Key

FwdDepKeysFromList makes a list of datastore.Key's that correspond to all of the FwdDeps expressed by the <base, list> pair.

func InvalidateExecution Uses

func InvalidateExecution(c context.Context, auth *dm.Execution_Auth) (a *Attempt, e *Execution, err error)

InvalidateExecution verifies that the execution key is valid, and then revokes the execution key.

As a bonus, it will return the loaded Attempt and Execution.

func MakeRandomToken Uses

func MakeRandomToken(c context.Context, l uint32) []byte

MakeRandomToken creates a cryptographically random byte slice of the specified length. It panics if the specified length cannot be read in full.

func QueryAttemptsForQuest Uses

func QueryAttemptsForQuest(c context.Context, qid string) *ds.Query

QueryAttemptsForQuest returns all Attempt objects that exist for this Quest.

func QuestIDFromKey Uses

func QuestIDFromKey(k *ds.Key) string

QuestIDFromKey makes a QuestID from the given datastore.Key. It panics if the Key does not point to a Quest.

func QuestKeyFromID Uses

func QuestKeyFromID(c context.Context, qid string) *ds.Key

QuestKeyFromID makes a datastore.Key given the QuestID.

type Attempt Uses

type Attempt struct {
    ID  dm.Attempt_ID `gae:"$id"`

    Created  time.Time
    Modified time.Time

    State      dm.Attempt_State
    RetryState AttemptRetryState

    // IsAbnormal is true iff State==ABNORMAL_FINISHED, used for walk_graph.
    IsAbnormal bool

    // A lazily-updated boolean to reflect that this Attempt is expired for
    // queries.
    IsExpired bool

    // Contains either data (State==FINISHED) or abnormal_finish (State==ABNORMAL_FINISHED)
    //
    // Does not contain the `data.object` field (which is in the AttemptResult,1 object)
    Result dm.Result `gae:",noindex"`

    // CurExecution is the maximum Execution ID for this Attempt so far. Execution
    // IDs are contiguous from [1, CurExecution]. If the State is not currently
    // Executing, then CurExecution represents the execution that JUST finished
    // (or 0 if no Executions have been made yet).
    CurExecution uint32

    // LastSuccessfulExecution is the execution ID of the last successful
    // execution, or 0 if no such execution occured yet.
    LastSuccessfulExecution uint32

    // DepMap is valid only while Attempt is in a State of EXECUTING or WAITING.
    //
    // The size of this field is inspected to deteremine what the next state after
    // EXECUTING is. If the size == 0, it means the Attempt should move to the
    // FINISHED state. Otherwise it means that the Attempt should move to the
    // WAITING state.
    //
    // A bit field value of 0 means that the dep is currently waiting, and a bit
    // value of 1 means that the coresponding dep is satisfined. The Attempt can
    // be unblocked from WAITING back to SCHEDULING when all bits are set to 1.
    DepMap bf.BitField `gae:",noindex" json:"-"`
}

Attempt is the datastore model for a DM Attempt. It has no parent key, but it may have the following children entities:

* FwdDep
* AttemptResult

Additionally, every Attempt has an associated BackDepGroup whose ID equals the ID of this Attempt.

func AttemptFromID Uses

func AttemptFromID(aid *dm.Attempt_ID) *Attempt

AttemptFromID produces an empty Attempt model from the AttemptID.

func MakeAttempt Uses

func MakeAttempt(c context.Context, aid *dm.Attempt_ID) *Attempt

MakeAttempt is a convenience function to create a new Attempt model in the NeedsExecution state.

func (*Attempt) DataProto Uses

func (a *Attempt) DataProto() (ret *dm.Attempt_Data)

DataProto returns an Attempt.Data message for this Attempt.

func (*Attempt) ModifyState Uses

func (a *Attempt) ModifyState(c context.Context, newState dm.Attempt_State) error

ModifyState changes the current state of this Attempt and updates its Modified timestamp.

func (*Attempt) ToProto Uses

func (a *Attempt) ToProto(withData bool) *dm.Attempt

ToProto returns a dm proto version of this Attempt.

type AttemptResult Uses

type AttemptResult struct {
    Attempt *datastore.Key `gae:"$parent"`

    // The sizes and expirations are denormalized across Attempt and
    // AttemptResult.
    Data dm.JsonResult `gae:",noindex"`
    // contains filtered or unexported fields
}

AttemptResult holds the raw, compressed json blob returned from the execution.

type AttemptRetryState Uses

type AttemptRetryState struct {
    Failed   uint32
    Expired  uint32
    TimedOut uint32
    Crashed  uint32
}

AttemptRetryState indicates the current state of the Attempt's retry counters.

func (*AttemptRetryState) Reset Uses

func (a *AttemptRetryState) Reset()

Reset resets all of the AttemptRetryState counters.

type BackDep Uses

type BackDep struct {
    // The attempt id of the attempt that's depending on this dependee.
    Depender dm.Attempt_ID `gae:"$id"`

    // The BackdepGroup for the attempt that is being depended on.
    DependeeGroup *datastore.Key `gae:"$parent"`

    // Propagated is true if the BackDepGroup has AttemptFinished, and this
    // BackDep has been processed by the mutate.RecordCompletion tumble
    // mutation. So if with two attempts A and B, A depends on B, the
    // BackDep{DependeeGroup: B, Depender: A} has Propagated as true when B is
    // finished, and a tumble Mutation has been launched to inform A of that fact.
    Propagated bool
}

BackDep represents a single backwards dependency. Its ID is the same as the Attempt that's depending on this one. See BackDepGroup for more context.

func (*BackDep) Edge Uses

func (b *BackDep) Edge() *FwdEdge

Edge produces a fwdedge object which points from the depending attempt to the depended-on attempt.

type BackDepGroup Uses

type BackDepGroup struct {
    // Dependee is the "<AttemptID>" that the deps in this group point
    // back FROM.
    Dependee dm.Attempt_ID `gae:"$id"`

    // This is a denormalized version of Attempt.State, used to allow
    // transactional additions to the BackDepGroup to stay within this Entity
    // Group when adding new back deps.
    AttemptFinished bool
}

BackDepGroup describes a group of reverse dependencies ('depended-by') between Attempts. Its ID is the same as the id of the Attempt that's being depended-on by other attempts, and it serves as the parent entity for the BackDep model. So:

Attempt(OTHER_QUEST|2)
  FwdDep(QUEST|1)

Attempt(QUEST|1)

BackDepGroup(QUEST|1)
  BackDep(OTHER_QUEST|2)

Represents the OTHER_QUEST|2 depending on QUEST|1.

type Execution Uses

type Execution struct {
    ID      invertedHexUint32 `gae:"$id"`
    Attempt *ds.Key           `gae:"$parent"`

    Created  time.Time
    Modified time.Time

    // DistributorConfigName is redundant with the Quest definition, but this
    // helps avoid extra unnecessary datastore round-trips to load the Quest.
    DistributorConfigName    string
    DistributorConfigVersion string
    DistributorToken         string

    State dm.Execution_State

    // IsAbnormal is true iff State==ABNORMAL_FINISHED. Used for walk_graph.
    IsAbnormal bool

    // A lazily-updated boolean to reflect that this Execution is expired for
    // queries.
    IsExpired bool

    // Contains either data (State==FINISHED) or abnormal_finish (State==ABNORMAL_FINISHED)
    Result dm.Result `gae:",noindex"`

    // These are DM's internal mechanism for performing timeout actions on
    // Executions.
    //
    // The TimeTo* variables are copied from the quest description.
    //
    // The Timeout is only active when the Execution is in a non-terminal state.
    TimeToStart time.Duration `gae:",noindex"` // timeouts.start
    TimeToRun   time.Duration `gae:",noindex"` // timeouts.run
    TimeToStop  time.Duration `gae:",noindex"` // pollTimeout || timeouts.stop

    // Token is a randomized nonce that's used to verify that RPCs verify from the
    // expected client (the client that's currently running the Execution). The
    // Token has 2 modes.
    //
    // When the Execution is handed to the distributor, the Token is randomly
    // generated by DM and passed to the distributor. The State of the Execution
    // starts as SCHEDULED. This token may be used by the client to "activate" the
    // Execution with the ActivateExecution rpc. At that point, the client
    // provides a new random token, the Execution State moves from SCHEDULED to
    // RUNNING, and Token assumes the new value. As long as the Execution State is
    // RUNNING, the client may continue to use that new Token value to
    // authenticate other rpc's like AddDeps and FinishAttempt.
    //
    // As soon as the Execution is in the STOPPING, ABNORMAL_FINISHED or FINISHED
    // state, this will be nil'd out.
    Token []byte `gae:",noindex"`
}

Execution represents either an ongoing execution on the Quest's specified distributor, or is a placeholder for an already-completed Execution.

func ExecutionFromID Uses

func ExecutionFromID(c context.Context, eid *dm.Execution_ID) *Execution

ExecutionFromID produces an empty Execution model from the ExecutionID.

func MakeExecution Uses

func MakeExecution(c context.Context, e *dm.Execution_ID, cfgName, cfgVers string) *Execution

MakeExecution makes a new Execution in the SCHEDULING state, with a new random Token.

func (*Execution) DataProto Uses

func (e *Execution) DataProto() (ret *dm.Execution_Data)

DataProto returns an Execution.Data message for this Execution.

This omits the DistributorInfo.Url portion, which must be filled in elsewhere for package cyclical import reasons.

func (*Execution) GetEID Uses

func (e *Execution) GetEID() *dm.Execution_ID

GetEID gets an Execution_ID for this Execution. It panics if the Execution is in an invalid state.

func (*Execution) ModifyState Uses

func (e *Execution) ModifyState(c context.Context, newState dm.Execution_State) error

ModifyState changes the current state of this Execution and updates its Modified timestamp.

func (*Execution) Revoke Uses

func (e *Execution) Revoke(c context.Context) error

Revoke will clear the Token and Put this Execution to the datastore. This action requires the Execution to be in the RUNNING state, and causes it to enter the STOPPING state.

func (*Execution) ToProto Uses

func (e *Execution) ToProto(includeID bool) *dm.Execution

ToProto returns a dm proto version of this Execution.

type FwdDep Uses

type FwdDep struct {
    // Attempt that this points from.
    Depender *ds.Key `gae:"$parent"`

    // A FwdDep's ID is the Attempt ID that it points to.
    Dependee dm.Attempt_ID `gae:"$id"`

    // This will be used to set a bit in the Attempt (WaitingDepBitmap) when the
    // Dep completes.
    BitIndex uint32

    // ForExecution indicates which Execution added this dependency. This is used
    // for validation of AckFwdDep mutations to ensure that they're operating
    // on an Attempt in the correct state, but can also be used for historical
    // analysis/display.
    ForExecution uint32
}

FwdDep describes a 'depends-on' relation between two Attempts. It has a reciprocal BackDep as well, which notes the depended-on-by relationship. So:

Attempt(OTHER_QUEST|2)
  FwdDep(QUEST|1)

Attempt(QUEST|1)

BackDepGroup(QUEST|1)
  BackDep(OTHER_QUEST|2)

Represents the OTHER_QUEST|2 depending on QUEST|1.

func FwdDepsFromList Uses

func FwdDepsFromList(c context.Context, base *dm.Attempt_ID, list *dm.AttemptList) []*FwdDep

FwdDepsFromList creates a slice of *FwdDep given an originating base Attempt_ID, and a list of dependency Attempts.

func (*FwdDep) Edge Uses

func (f *FwdDep) Edge() *FwdEdge

Edge produces a edge object which points 'forwards' from the depending attempt to the depended-on attempt.

type FwdEdge Uses

type FwdEdge struct {
    From *dm.Attempt_ID
    To   *dm.Attempt_ID
}

FwdEdge represents a forward-dependency from one attempt to another. The From attempt will block until the To attempt completes.

func (*FwdEdge) Back Uses

func (e *FwdEdge) Back(c context.Context) (*BackDepGroup, *BackDep)

Back returns the BackDepGroup (To) and the BackDep (From) models that represent the reverse dependency for this FwdEdge.

func (*FwdEdge) Fwd Uses

func (e *FwdEdge) Fwd(c context.Context) (*Attempt, *FwdDep)

Fwd returns the Attempt (From) and FwdDep (To) models that this FwdEdge represents.

type Quest Uses

type Quest struct {
    // ID is the base64 sha256 of questPayload
    ID  string `gae:"$id"`

    Desc    dm.Quest_Desc `gae:",noindex"`
    BuiltBy TemplateInfo  `gae:",noindex"`

    Created time.Time `gae:",noindex"`
}

Quest is the model for a job-to-run. Its questPayload should fully describe the job to be done.

func NewQuest Uses

func NewQuest(c context.Context, desc *dm.Quest_Desc) *Quest

NewQuest builds a new Quest object with a correct ID given the current contents of the Quest_Desc. It returns an error if the Desc is invalid.

Desc must already be Normalize()'d

func QuestFromID Uses

func QuestFromID(qid string) *Quest

QuestFromID produces an empty Quest model from the QuestID.

func (*Quest) DataProto Uses

func (q *Quest) DataProto() *dm.Quest_Data

DataProto gets the Quest.Data proto message for this Quest.

func (*Quest) Equals Uses

func (q *Quest) Equals(a *Quest) bool

Equals is true if q and a and equal.

func (*Quest) ToProto Uses

func (q *Quest) ToProto() *dm.Quest

ToProto converts this Quest into its display equivalent.

type TemplateInfo Uses

type TemplateInfo []dm.Quest_TemplateSpec

TemplateInfo is an ordered list of dm.Quest_TemplateSpec's

func (*TemplateInfo) Add Uses

func (ti *TemplateInfo) Add(ts ...dm.Quest_TemplateSpec)

Add adds ts to the TemplateInfo uniq'ly.

func (TemplateInfo) Equals Uses

func (ti TemplateInfo) Equals(other TemplateInfo) bool

Equals returns true iff this TemplateInfo is exactly same as `other`.

func (TemplateInfo) EqualsData Uses

func (ti TemplateInfo) EqualsData(other []*dm.Quest_TemplateSpec) bool

EqualsData returns true iff this TemplateInfo has the same content as the proto-style TemplateInfo. This assumes that `other` is sorted.

func (TemplateInfo) Len Uses

func (ti TemplateInfo) Len() int

func (TemplateInfo) Less Uses

func (ti TemplateInfo) Less(i, j int) bool

func (TemplateInfo) Swap Uses

func (ti TemplateInfo) Swap(i, j int)

Package model imports 20 packages (graph) and is imported by 8 packages. Updated 2018-10-21. Refresh now. Tools for package owners.