luci: go.chromium.org/luci/cipd/appengine/impl/model Index | Files

package model

import "go.chromium.org/luci/cipd/appengine/impl/model"

Package model contains core CIPD datastore entities.

Index

Package Files

deletion.go doc.go events.go instance.go package.go processing.go ref.go search.go tag.go txn.go version.go

Constants

const (
    // Hidden can be used in place of 'true' when working with Package.Hidden flag.
    Hidden = true
    // Visible can be used in place of 'false' when working with Package.Hidden flag.
    Visible = false
)
const NoInstance = "NONE"

NoInstance is used in place of empty Instance field in Event entity, so that we can query for all events that are not associated with instances and only them (these are package-level events).

Variables

var EventsEpoch = time.Date(2018, time.January, 1, 0, 0, 0, 0, time.UTC)

EventsEpoch is used to calculate timestamps used to order entities in the datastore.

time.Time is stored with millisecond precision in the datastore. This is not enough to preserve the order of cipd.Event protos (that use nanoseconds for ordering events within a transactional group).

So instead we store (event.When-EventsEpoch).Nanos() as int64. The original timestamp is still available in cipd.Event proto. This is only for datastore index.

func AttachTags Uses

func AttachTags(c context.Context, inst *Instance, tags []*api.Tag) error

AttachTags transactionally attaches a bunch of tags to an instance.

Assumes inputs are already validated. Launches a transaction inside (and thus can't be a part of a transaction itself). Updates 'inst' in-place with the most recent instance state.

Returns gRPC-tagged errors:

NotFound if there's no such instance or package.
FailedPrecondition if some processors are still running.
Aborted if some processors have failed.
Internal on tag ID collision.

func CheckInstanceExists Uses

func CheckInstanceExists(c context.Context, inst *Instance) error

CheckInstanceExists fetches the instance and verifies it exists.

Can be called as part of a transaction. Updates 'inst' in place.

Returns gRPC-tagged NotFound error if there's no such instance or package.

func CheckInstanceReady Uses

func CheckInstanceReady(c context.Context, inst *Instance) error

CheckInstanceReady fetches the instance and verifies it exists and has zero pending or failed processors.

Can be called as part of a transaction. Updates 'inst' in place.

Returns gRPC-tagged errors:

NotFound if there's no such instance or package.
FailedPrecondition if some processors are still running.
Aborted if some processors have failed.

func CheckPackageExists Uses

func CheckPackageExists(c context.Context, pkg string) error

CheckPackageExists verifies the package exists.

Returns gRPC-tagged NotFound error if there's no such package.

func CheckPackages Uses

func CheckPackages(c context.Context, names []string, includeHidden bool) ([]string, error)

CheckPackages given a list of package names returns packages that exist, in the order they are listed in the list.

If includeHidden is false, omits hidden packages from the result.

Returns only transient errors.

func DeletePackage Uses

func DeletePackage(c context.Context, pkg string) error

DeletePackage deletes all entities associated with a package.

Returns grpc-tagged errors (in particular NotFound if there's no such package).

func DeleteRef Uses

func DeleteRef(c context.Context, pkg, ref string) error

DeleteRef removes the ref if it exists.

Does nothing if there's no such ref or package.

func DetachTags Uses

func DetachTags(c context.Context, inst *Instance, tags []*api.Tag) error

DetachTags detaches a bunch of tags from an instance.

Assumes inputs are already validated. Launches a transaction inside (and thus can't be a part of a transaction itself).

'inst' is used only for its key.

func EmitEvent Uses

func EmitEvent(c context.Context, e *api.Event) error

EmitEvent adds a single event to the event log.

Prefer using Events to add multiple events at once.

func EmitMetadataEvents Uses

func EmitMetadataEvents(c context.Context, before, after *api.PrefixMetadata) error

EmitMetadataEvents compares two metadatum of a prefix and emits events for fields that have changed.

func FetchProcessors Uses

func FetchProcessors(c context.Context, inst *Instance) ([]*api.Processor, error)

FetchProcessors fetches results of all processors assigned to the instance and returns them as cipd.Processor proto messages (sorted by processor ID).

func FlushEventsToBQ Uses

func FlushEventsToBQ(c context.Context) error

FlushEventsToBQ sends all buffered events to BigQuery.

It is fine to call FlushEventsToBQ concurrently from multiple request handlers, if necessary (it will effectively parallelize the flush).

func ListPackages Uses

func ListPackages(c context.Context, prefix string, includeHidden bool) (out []string, err error)

ListPackages returns a list of names of packages under the given prefix.

Lists all packages recursively. If there's package named as 'prefix' it is NOT included in the result. Only packaged under the prefix are included.

The result is sorted by the package name. Returns only transient errors.

func NewEventsQuery Uses

func NewEventsQuery() *datastore.Query

NewEventsQuery returns an unfiltered query for Event entities.

func PackageKey Uses

func PackageKey(c context.Context, pkg string) *datastore.Key

PackageKey returns a datastore key of some package, given its name.

func QueryEvents Uses

func QueryEvents(c context.Context, q *datastore.Query) ([]*api.Event, error)

QueryEvents fetches and deserializes all events matching the query, sorting them by timestamp (most recent first).

Start the query with NewEventsQuery.

func ResolveTag Uses

func ResolveTag(c context.Context, pkg string, tag *api.Tag) (string, error)

ResolveTag searches for a given tag among all instances of the package and returns an ID of the instance the tag is attached to.

Assumes inputs are already validated. Doesn't double check the instance exists.

Returns gRPC-tagged errors:

NotFound if there's no such tag at all.
FailedPrecondition if the tag resolves to multiple instances.

func SetPackageHidden Uses

func SetPackageHidden(c context.Context, pkg string, hidden bool) error

SetPackageHidden updates Hidden field of the package.

If the package is missing returns datastore.ErrNoSuchEntity. All other errors are transient.

func SetRef Uses

func SetRef(c context.Context, ref string, inst *Instance) error

SetRef moves or creates a ref.

Assumes inputs are already validated. Launches a transaction inside (and thus can't be a part of a transaction itself). Updates 'inst' in-place with the most recent instance state.

Returns gRPC-tagged errors:

NotFound if there's no such instance or package.
FailedPrecondition if some processors are still running.
Aborted if some processors have failed.

func TagID Uses

func TagID(t *api.Tag) string

TagID calculates Tag entity ID (SHA1 digest) from the given tag.

Panics if the tag is invalid.

func Txn Uses

func Txn(c context.Context, name string, cb func(context.Context) error) error

Txn runs the callback in a datastore transaction, marking commit errors with transient tag.

The given name will be used for logging and error messages.

type Event Uses

type Event struct {
    ID      int64          `gae:"$id"`      // auto-generated
    Package *datastore.Key `gae:"$parent"`  // see PackageKey()
    Event   []byte         `gae:",noindex"` // serialized cipd.Event proto

    // Fields extracted from cipd.Event for indexing.
    Kind     api.EventKind
    Who      string
    When     int64  // nanoseconds since EventsEpoch
    Instance string // literal 'NONE' for package events, to be able to query them
    Ref      string
    Tag      string
    // contains filtered or unexported fields
}

Event in a global structured event log.

It exists in both BigQuery (for adhoc queries) and in Datastore (for showing in web UI, e.g. for "recent tags" feature).

ID as auto-generated. The parent entity is the corresponding Package entity (so that events can be committed transactionally with actual changes).

func (*Event) FromProto Uses

func (e *Event) FromProto(c context.Context, p *api.Event) *Event

FromProto fills in the entity based on the proto message.

Panics if the proto can't be serialized. This should never happen.

Returns the entity itself for easier chaining.

type Events Uses

type Events struct {
    // contains filtered or unexported fields
}

Events collects events emitted in some transaction and then flushes them.

func (*Events) Emit Uses

func (t *Events) Emit(e *api.Event)

Emit adds an event to be flushed later in Flush.

'Who' and 'When' fields are populated in Flush using values from the context.

func (*Events) Flush Uses

func (t *Events) Flush(c context.Context) error

Flush sends all pending events to the datastore/bigquery.

Returned errors are tagged as transient. On success, clears the pending events queue.

type Instance Uses

type Instance struct {
    InstanceID string         `gae:"$id"`     // see common.ObjectRefToInstanceID()
    Package    *datastore.Key `gae:"$parent"` // see PackageKey()

    RegisteredBy string    `gae:"registered_by"` // who registered it
    RegisteredTs time.Time `gae:"registered_ts"` // when it was registered

    // Names of currently running or scheduled processors.
    ProcessorsPending []string `gae:"processors_pending"`
    // Names of processors that successfully finished the processing.
    ProcessorsSuccess []string `gae:"processors_success"`
    // Names of processors that returned fatal errors.
    ProcessorsFailure []string `gae:"processors_failure"`
    // contains filtered or unexported fields
}

Instance represents a package instance as it is stored in the datastore.

Contains some instance metadata, in particular a list of processors that scanned the instance (see below).

The parent entity is the corresponding package entity. ID is derived from package instance file digest, see common.ObjectRefToInstanceID().

Compatible with the python version of the backend.

func ListInstances Uses

func ListInstances(c context.Context, pkg string, pageSize int32, cursor datastore.Cursor) (out []*Instance, nextCur datastore.Cursor, err error)

ListInstances lists instances of a package, more recent first.

Only does a query over Instances entities. Doesn't check whether the Package entity exists. Returns up to pageSize entities, plus non-nil cursor (if there are more results). pageSize must be positive.

func RegisterInstance Uses

func RegisterInstance(c context.Context, inst *Instance, cb func(context.Context, *Instance) error) (reg bool, out *Instance, err error)

RegisterInstance transactionally registers an instance (and the corresponding package), if it isn't registered already.

Calls the given callback (inside the transaction) if it is indeed registering a new instance. The callback may mutate the instance entity before it is stored. The callback may be called multiple times in case of retries (each time it will be given a fresh instance to be mutated).

Returns (true, entity, nil) if the instance was registered just now or (false, entity, nil) if it existed before.

In either case, it returns the entity that is stored now in the datastore. It is either the new instance, or something that existed there before.

func ResolveVersion Uses

func ResolveVersion(c context.Context, pkg, version string) (*Instance, error)

ResolveVersion takes a version identifier (an instance ID, a ref or a tag) and resolves it into a concrete package instance, ensuring it exists.

Returns gRPC-tagged errors:

InvalidArgument if the version string format is invalid.
NotFound if there's no such package or such version.
FailedPrecondition if the tag resolves to multiple instances.

func SearchInstances Uses

func SearchInstances(c context.Context, pkg string, tags []*api.Tag, pageSize int32, cursor datastore.Cursor) (out []*Instance, nextCur datastore.Cursor, err error)

SearchInstances lists instances of a package with all given tags attached.

Only does a query over Instances entities. Doesn't check whether the Package entity exists. Returns up to pageSize entities, plus non-nil cursor (if there are more results). 'pageSize' must be positive.

func (*Instance) FromProto Uses

func (e *Instance) FromProto(c context.Context, p *api.Instance) *Instance

FromProto fills in the entity based on the proto message.

Returns the entity itself for easier chaining.

Doesn't touch output-only fields at all.

func (*Instance) Proto Uses

func (e *Instance) Proto() *api.Instance

Proto returns cipd.Instance proto with information from this entity.

type Package Uses

type Package struct {
    Name string `gae:"$id"` // e.g. "a/b/c"

    RegisteredBy string    `gae:"registered_by"` // who registered it
    RegisteredTs time.Time `gae:"registered_ts"` // when it was registered

    Hidden bool `gae:"hidden"` // if true, hide from the listings
    // contains filtered or unexported fields
}

Package represents a package as it is stored in the datastore.

It is mostly a marker that the package exists plus some minimal metadata about this specific package. Metadata for the package prefix is stored separately elsewhere (see 'metadata' package). Package instances, tags and refs are stored as child entities (see below).

Root entity. ID is the package name.

Compatible with the python version of the backend.

type ProcessingResult Uses

type ProcessingResult struct {
    ProcID   string         `gae:"$id"`     // processor that generated the result
    Instance *datastore.Key `gae:"$parent"` // instance it was generated from

    CreatedTs time.Time `gae:"created_ts"`     // when it was generated
    Success   bool      `gae:"success"`        // mostly for for indexing
    Error     string    `gae:"error,noindex"`  // for Success == false
    ResultRaw []byte    `gae:"result,noindex"` // for Success == true
    // contains filtered or unexported fields
}

ProcessingResult holds information extracted from the package instance file.

It is obtained during an asynchronous post processing step triggered after the instance is uploaded. Immutable.

Entity ID is a processor name used to extract it. Parent entity is PackageInstance the information was extracted from.

func (*ProcessingResult) Proto Uses

func (p *ProcessingResult) Proto() (*api.Processor, error)

Proto returns cipd.Processor proto with information from this entity.

func (*ProcessingResult) ReadResult Uses

func (p *ProcessingResult) ReadResult(r interface{}) error

ReadResult deserializes the result into the given variable.

Does nothing if there's no results stored.

func (*ProcessingResult) ReadResultIntoStruct Uses

func (p *ProcessingResult) ReadResultIntoStruct(s *structpb.Struct) error

ReadResultIntoStruct deserializes the result into the protobuf.Struct.

Does nothing if there's no results stored.

func (*ProcessingResult) WriteResult Uses

func (p *ProcessingResult) WriteResult(r interface{}) error

WriteResult overwrites ResultRaw field with compressed JSON-serialized 'r'.

'r' should serialize into a JSON object, e.g '{...}'.

type Ref Uses

type Ref struct {
    Name    string         `gae:"$id"`     // e.g. "latest"
    Package *datastore.Key `gae:"$parent"` // see PackageKey()

    InstanceID string    `gae:"instance_id"` // see common.ObjectRefToInstanceID()
    ModifiedBy string    `gae:"modified_by"` // who moved it the last time
    ModifiedTs time.Time `gae:"modified_ts"` // when it was moved the last time
    // contains filtered or unexported fields
}

Ref represents a named pointer to some package instance.

ID is a ref name, the parent entity is the corresponding Package.

Compatible with the python version of the backend.

func GetRef Uses

func GetRef(c context.Context, pkg, ref string) (*Ref, error)

GetRef fetches the given ref.

Returns gRPC-tagged NotFound error if there's no such ref.

func ListInstanceRefs Uses

func ListInstanceRefs(c context.Context, inst *Instance) (out []*Ref, err error)

ListInstanceRefs returns all refs that point to a particular instance, most recently modified first.

This is a subset of the output of ListPackageRefs for the corresponding package.

Assumes 'inst' is a valid Instance, panics otherwise.

Returns an empty list if there's no such instance at all.

func ListPackageRefs Uses

func ListPackageRefs(c context.Context, pkg string) (out []*Ref, err error)

ListPackageRefs returns all refs in a package, most recently modified first.

Returns an empty list if there's no such package at all.

func (*Ref) Proto Uses

func (e *Ref) Proto() *api.Ref

Proto returns cipd.Ref proto with information from this entity.

type Tag Uses

type Tag struct {
    ID       string         `gae:"$id"`     // see TagID()
    Instance *datastore.Key `gae:"$parent"` // key of corresponding Instance entity

    Tag          string    `gae:"tag"`           // the tag itself, as "k:v" pair
    RegisteredBy string    `gae:"registered_by"` // who added this tag
    RegisteredTs time.Time `gae:"registered_ts"` // when it was added
    // contains filtered or unexported fields
}

Tag represents a "key:value" pair attached to some package instance.

Tags exist as separate entities (as opposed to being a repeated property of Instance entity) to allow essentially unlimited number of them. Also we often do not want to fetch all tags when fetching Instance entity.

ID is hex-encoded SHA1 of the tag. The parent entity is the corresponding Instance. The tag string can't be made a part of the key because the total key length (including entity kind names and all parent keys) is limited to 500 bytes and tags are allowed to be pretty long (up to 400 bytes).

There's a possibility someone tries to abuse a collision in SHA1 hash used by TagID() to attach or detach some wrong tag. The chance is minuscule, and it's not a big deal if it happens now, but it may become important in the future once tags have their own ACLs. For the sake of paranoia, we double check 'Tag' field in all entities we touch.

Compatible with the python version of the backend.

func ListInstanceTags Uses

func ListInstanceTags(c context.Context, inst *Instance) (out []*Tag, err error)

ListInstanceTags returns all tags attached to an instance, sorting them by the tag key first, and then by the timestamp (most recent first).

Returns an empty list if there's no such instance at all.

func (*Tag) Proto Uses

func (t *Tag) Proto() *api.Tag

Proto returns cipd.Tag proto with information from this entity.

Assumes the tag is valid.

Package model imports 32 packages (graph) and is imported by 4 packages. Updated 2018-12-12. Refresh now. Tools for package owners.