luci: Index | Files | Directories

package config

import ""

Package config contains a client to access LUCI configuration service.

WARNING: Large chunks of this package tree are deprecated and will be removed at some point. In particular 'appengine/*' and 'server/cfgclient/*' will not be made compatible with GAE v2 and will be deleted.

The existing API (exposed via server/cfgclient package) is "pull-based": users "pull" configs via cfgclient.Get(...) calls whenever they want. In particular they may do it from the data plane (which generally handles a lot of QPS). This requires cfgclient.Get(...) to be very fast and very reliable. "Very fast and very reliable" doesn't apply to LUCI Config service itself at all (nor should it). As a result cfgclient.Get(...) implementation had to incorporate some pretty complicated multi-layered cache (that no one fully understands anymore).

But turns out the majority of users (but not all) use cfgclient.Get(...) from their control plane, i.e. they run some cron job that periodically pulls most recent configs, processes them in some way, and stores them somewhere where the data plane can reliably reach them. This also gives them flexibility to react to diffs in the configs.

For such users the complexity of cfgclient.Get(...) implementation is completely unnecessary. It just adds more work to them to configure the caching layer.

We are migrating to GAE v2 and GKE, and we no longer assume Cloud Datastore and Memcache are ubiquitously present. The existing complicated cfgclient code will not work in such environments.

Instead of refactoring it we are planning to remove it completely and change the public package API to be "push-based", i.e. we want to *force* users to register some callback which receives configs whenever they change. Then users may store them whereever they want (like most of them already do in the cron jobs).

This approach would result in a vastly simpler config client code and will guarantee LUCI Config service is not a dependency of the data plane (so it's not a big deal if it goes down).


Package Files

doc.go interface.go project_name.go set.go


var ErrNoConfig = errors.New("no such config")

ErrNoConfig is returned if requested config does not exist.

func ValidateProjectName Uses

func ValidateProjectName(p string) error

ValidateProjectName returns an error if the supplied string is not a valid project name.

A valid project name may only include:

- Lowercase letters [a-z]
- Numbers [0-9]
- Hyphen (-)
- Underscore (_)

It also must begin with a letter.


type Config Uses

type Config struct {

    // FormatSpec, if non-empty, qualifies the format of the Content.
    // It is set only if config was obtained through cfgclient guts that does
    // reformatting (e.g. formats text protos into binary protos).
    FormatSpec FormatSpec `json:"formatSpec,omitempty"`

    // Error is not nil if there where troubles fetching this config. Used only
    // by functions that operate with multiple configs at once, such as
    // GetProjectConfigs and GetRefConfigs.
    Error error `json:"error,omitempty"`

    // Content is the actual body of the config file.
    Content string `json:"content,omitempty"`

Config is a configuration file along with its metadata.

type FormatSpec Uses

type FormatSpec struct {
    // Formatter is the name of the formatter that produces the content data.
    // An empty string means the original config service format.
    Formatter string

    // Data is additional format data describing the type. It may be empty.
    Data string

FormatSpec is a specification for formatted data.


func (*FormatSpec) Unformatted Uses

func (fs *FormatSpec) Unformatted() bool

Unformatted returns true if fs does not specify a format.

type Interface Uses

type Interface interface {
    // GetConfig returns a config at a path in a config set or ErrNoConfig
    // if missing. If metaOnly is true, returned Config struct has only Meta set
    // (and the call is faster).
    GetConfig(ctx context.Context, configSet Set, path string, metaOnly bool) (*Config, error)

    // GetConfigByHash returns the contents of a config, as identified by its
    // content hash, or ErrNoConfig if missing.
    GetConfigByHash(ctx context.Context, contentHash string) (string, error)

    // GetConfigSetLocation returns the URL location of a config set.
    GetConfigSetLocation(ctx context.Context, configSet Set) (*url.URL, error)

    // GetProjectConfigs returns all the configs at the given path in all
    // projects that have such config. If metaOnly is true, returned Config
    // structs have only Meta set (and the call is faster).
    GetProjectConfigs(ctx context.Context, path string, metaOnly bool) ([]Config, error)

    // GetProjects returns all the registered projects in the configuration
    // service.
    GetProjects(ctx context.Context) ([]Project, error)

    // GetRefConfigs returns the config at the given path in all refs of all
    // projects that have such config. If metaOnly is true, returned Config
    // structs have only Meta set (and the call is faster).
    GetRefConfigs(ctx context.Context, path string, metaOnly bool) ([]Config, error)

    // GetRefs returns the list of refs for a project.
    GetRefs(ctx context.Context, projectID string) ([]string, error)

    // ListFiles returns the list of files for a config set.
    ListFiles(ctx context.Context, configSet Set) ([]string, error)

Interface represents low-level pull-based LUCI Config API.

This is roughly a wrapper over LUCI Config RPC interface, and all methods are generally slow and depend on available of LUCI Config service. They *must not* be used in a hot path of requests.

Transient errors are tagged with transient.Tag.

type Meta Uses

type Meta struct {
    // ConfigSet is the config set name (e.g. "projects/<id>") this config
    // belongs to.
    ConfigSet Set `json:"configSet,omitempty"`

    // Path is the filename relative to the root of the config set,
    // without leading slash, e.g. "luci-scheduler.cfg".
    Path string `json:"path,omitempty"`

    // ContentHash can be used to quickly check that content didn't change.
    ContentHash string `json:"contentHash,omitempty"`

    // Revision is git SHA1 of a repository the config was fetched from.
    Revision string `json:"revision,omitempty"`

    // ViewURL is the URL surfaced for viewing the config.
    ViewURL string `json:"view_url,omitempty"`

Meta is metadata about a single configuration file.

type Project Uses

type Project struct {
    // ID is unique project identifier.
    ID  string

    // Name is a short friendly display name of the project.
    Name string

    // RepoType specifies in what kind of storage projects configs are stored.
    RepoType RepoType

    // RepoUrl is the location of this project code repository. May be nil if
    // unknown or cannot be parsed.
    RepoURL *url.URL

Project is a project registered in the luci-config service.

type RepoType Uses

type RepoType string

RepoType is the type of the repo the Project is stored in.

const (
    // GitilesRepo means a repo is backed by the Gitiles service.
    GitilesRepo RepoType = "GITILES"

    // UnknownRepo means a repo is backed by an unknown service.
    // It may be an invalid repo.
    UnknownRepo RepoType = "UNKNOWN"

type Set Uses

type Set string

Set is a name of a configuration set: a bunch of config files versioned and stored as a single unit in a same repository.

A config set name consists of a domain and a series of path components: domain/target[/ref].

- Service config sets are config sets in the "services" domain, with the
  service name as the target.
- Project config sets are config sets in the "projects" domain. The target
  is the project name.

func ProjectSet Uses

func ProjectSet(project string) Set

ProjectSet returns the config set for the specified project.

func RefSet Uses

func RefSet(project string, ref string) Set

RefSet returns the config set for the specified project and ref. If ref is empty, this will equal the ProjectSet value.

func ServiceSet Uses

func ServiceSet(service string) Set

ServiceSet returns the name of a config set for the specified service.

func (Set) Project Uses

func (cs Set) Project() string

Project returns a project name for a project-rooted config set or empty string for all other sets.

Use ProjectAndRef if you need to get both the project name and the ref.

func (Set) ProjectAndRef Uses

func (cs Set) ProjectAndRef() (project, ref string)

ProjectAndRef splits a project-rooted config set (projects/<name>[/...]) into its project name and ref components.

For example, "projects/foo/bar/baz" is a config set that belongs to project "foo". It will be parsed into ("foo", "bar/baz").

If configSet is not a project config set, empty strings will be returned.

func (Set) Ref Uses

func (cs Set) Ref() string

Ref returns a ref component of a project-rooted config set or empty string for all other sets.

Use ProjectAndRef if you need to get both the project name and the ref.

func (Set) Service Uses

func (cs Set) Service() string

Service returns a service name for a service-rooted config set or empty string for all other sets.

func (Set) Split Uses

func (cs Set) Split() (domain, target, ref string)

Split splits a Set into its domain, target, and ref components.


appengine/backend/memcachePackage memcache implements a caching config client backend backed by AppEngine's memcache service.
appengine/gaeconfigPackage gaeconfig implements LUCI-config service bindings backed by AppEngine storage and caching.
impl/erroringPackage erroring implements a backend that always returns an error for all of its calls.
impl/filesystemPackage filesystem implements a file system backend for the config client.
impl/memoryPackage memory implements in-memory backend for the config client.
server/cfgclientPackage cfgclient contains service implementations for the LUCI configuration service defined in
server/cfgclient/accessPackage access implements a config service access check against a project config client.
server/cfgclient/backendPackage backend implements configuration client backend interface and associated data types.
server/cfgclient/backend/cachingPackage caching implements a config.Interface that uses a caching layer to store its configuration values.
server/cfgclient/backend/clientPackage client implements a config client backend for a configuration client.
server/cfgclient/backend/erroringPackage erroring implements config.Backend that simply returns an error.
server/cfgclient/backend/formatPackage format implements a config client Backend that performs formatting on items.
server/cfgclient/textprotoPackage textproto implements a textproto config service Resolver.
server/cfgmodulePackage cfgmodule provides a server module that registers config validation hooks.
validationPackage validation provides helpers for performing and setting up handlers for config validation related requests from luci-config.

Package config imports 5 packages (graph) and is imported by 64 packages. Updated 2020-05-31. Refresh now. Tools for package owners.