luci: go.chromium.org/luci/luciexe/build Index | Files

package build

import "go.chromium.org/luci/luciexe/build"

Package build implements a minimal, safe, easy-to-use, hard-to-use-wrong, 'luciexe binary' implementation in Go.

See `Start` for the entrypoint to this API.

Features:

* Can be used in library code which is used in both luciexe and non-luciexe
  contexts. You won't need "two versions" of your library just to give it
  the appearance of steps in a LUCI build context, or to have it read or
  write build properties.
* Goroutine-safe API.
* Handles all interactions with the log sink (e.g. LogDog), and
  automatically reroutes the logging library if a log sink is configured
  (so all log messages will be preserved in a build context).
* Handles coalescing all build mutations from multiple goroutines.
  Efficient throttling for outgoing updates.
* Automatically upholds all luciexe protocol invariants; Avoids invalid
  Build states by construction.
* Automatically maps errors and panics to their obvious visual
  representation in the build (and allows manual control in case you want
  something else).
* Fully decoupled from LUCI service implementation; You can use this API
  with no external services (e.g. for a CLI application), backed with the
  real LUCI implementation (e.g. when running under BuildBucket), and in
  tests (you can observe all outputs from this API without any live
  services or heavy mocks).

No-Op Mode

When no Build is in use in the context, the library behaves in 'no-op' mode. This should enable libraries to add `build` features which gracefully degrade into pure terminal output via logging.

* MakePropertyReader functions will return empty messages. Well-behaved
  libraries should handle having no configuration in the context if this is
  possible.
* There will be no *State object, because there is no Start call.
* StartStep/ScheduleStep will return a *Step which is detached. Step
  namespacing will still work in context (but name deduplication will not).
* The result of State.Modify/Step.Modify (and Set) calls will be
  logged at DEBUG.
* Step scheduled/started/ended messages will be logged at INFO.
  Ended log messages will include the final summary markdown as well.
* Text logs will be logged line-by-line at INFO with fields set indicating
  which step and log they were emitted from. Debug text logs (those whose
  log names start with "$") will be logged at DEBUG level.
* Non-text logs will be dropped with a WARNING indicating that they're
  being dropped.
* MakePropertyReader property readers will return empty message objects.
* MakePropertyModifier property manipulators will log their emitted
  properties at INFO.

Index

Package Files

doc.go errors.go logging.go main.go name_tracker.go properties.go start_options.go state.go state_view.go step.go step_view.go

func AttachStatus Uses

func AttachStatus(err error, status bbpb.Status, details *bbpb.StatusDetails) error

AttachStatus attaches a buildbucket status (and details) to a given error.

If such an error is handled by Step.End or State.End, that step/build will have its status updated to match.

AttachStatus allows overriding the attached status if the error already has one.

This panics if the status is a non-terminal status like SCHEDULED or STARTED.

This is a no-op if the error is nil.

func ExtractStatus Uses

func ExtractStatus(err error) (bbpb.Status, *bbpb.StatusDetails)

ExtractStatus retrieves the Buildbucket status (and details) from a given error.

This returns:

* (SUCCESS, nil) on nil error
* Any values attached with AttachStatus.
* (CANCELED, nil) on context.Canceled
* (INFRA_FAILURE, &bbpb.StatusDetails{Timeout: {}}) on
  context.DeadlineExceeded
* (FAILURE, nil) otherwise

This function is used internally by Step.End and State.End, but is provided publically for completeness.

func Main Uses

func Main(inputMsg proto.Message, writeFnptr, mergeFnptr interface{}, cb func(context.Context, []string, *State) error)

Main implements all the 'command-line' behaviors of the luciexe 'exe' protocol, including:

* parsing command line for "--output", "--help", etc.
* parsing stdin (as appropriate) for the incoming Build message
* creating and configuring a logdog client to send State evolutions.
* Configuring a logdog client from the environment.
* Writing Build state updates to the logdog "build.proto" stream.
* Start'ing the build in this process.
* End'ing the build with the returned error from your function

If `inputMsg` is nil, the top-level properties will be ignored.

If `writeFnptr` and `mergeFnptr` are nil, they're ignored. Otherwise they work as they would for MakePropertyModifier.

CLI Arguments parsed:

* -h / --help : Print help for this binary (including input/output
  property type info)
* --strict : Enable strict property parsing (see OptStrictInputProperties)
* --output : luciexe "output" flag; See
  https://pkg.go.dev/go.chromium.org/luci/luciexe#hdr-Recursive_Invocation
* -- : Any extra arguments after a "--" token are passed to your callback
  as-is.

Example:

func main() {
  input := *MyInputProps{}
  var writeOutputProps func(context.Context, *MyOutputProps)
  var mergeOutputProps func(context.Context, *MyOutputProps)

  Main(input, &writeOutputProps, &mergeOutputProps, func(ctx context.Context, args []string, st *build.State) error {
    // actual build code here, build is already Start'd
    // input was parsed from build.Input.Properties
    writeOutputProps(ctx, &MyOutputProps{...})
    return nil // will mark the Build as SUCCESS
  })
}

NOTE: These types are pretty bad; There's significant opportunity to improve them with Go2 generics.

func MakePropertyModifier Uses

func MakePropertyModifier(ns string, writeFnptr, mergeFnptr interface{})

MakePropertyModifier allows your library/module to reserve a section of the output properties for itself.

You can use this to obtain a write function (replace contents at namespace) and/or a merge function (do proto.Merge on the current contents of that namespace). If one of the function pointers is nil, it will be skipped (at least one must be non-nil). If both function pointers are provided, their types must exactly agree.

Attempting to reserve duplicate namespaces will panic. The namespace refers to the top-level property key. It is recommended that:

* The `ns` begins with '$'.
* The value after the '$' is the canonical Go package name for your
  library.

You should call this at init()-time like:

var propWriter func(context.Context, *MyMessage)
var propMerger func(context.Context, *MyMessage)

func init() {
  // one of the two function pointers may be nil
  MakePropertyModifier("$some/namespace", &propWriter, &propMerger)
}

In Go2 this will be less weird:

type PropertyModifier[T proto.Message] interface {
  Write(context.Context, value T) // assigns 'value'
  Merge(context.Context, value T) // does proto.Merge(current, value)
}
func MakePropertyModifier[T proto.Message](ns string) PropertyModifier[T]

func MakePropertyReader Uses

func MakePropertyReader(ns string, fnptr interface{})

MakePropertyReader allows your library/module to reserve a section of the input properties for itself.

Attempting to reserve duplicate namespaces will panic. The namespace refers to the top-level property key. It is recommended that:

* The `ns` begins with '$'.
* The value after the '$' is the canonical Go package name for your
  library.

Using the generated function will parse the relevant input property namespace as JSONPB, returning the parsed message (and an error, if any).

var myPropertyReader func(context.Context) *MyPropertyMsg
func init() {
  MakePropertyReader("$some/namespace", &myPropertyReader)
}

In Go2 this will be less weird:

MakePropertyReader[T proto.Message](ns string) func(context.Context) T

type Loggable Uses

type Loggable interface {
    // Log creates a new log stream (by default, line-oriented text) with the
    // given name.
    //
    // To uphold the requirements of the Build proto message, duplicate log names
    // will be deduplicated with the same algorithm used for deduplicating step
    // names.
    //
    // To create a binary stream, pass streamclient.Binary() as one of the
    // options.
    //
    // The stream will close when the associated object (step or build) is End'd.
    Log(name string, opts ...streamclient.Option) io.Writer

    // Log creates a new datagram-oriented log stream with the given name.
    //
    // To uphold the requirements of the Build proto message, duplicate log names
    // will be deduplicated with the same algorithm used for deduplicating step
    // names.
    //
    // The stream will close when the associated object (step or build) is End'd.
    LogDatagram(name string, opts ...streamclient.Option) streamclient.DatagramWriter
}

Loggable is the common interface for build entities which have log data associated with them.

Implemented by State and Step.

Logs all have a name which is an arbitrary bit of text to identify the log to human users (it will appear as the link on the build UI page). In particular it does NOT need to conform to the LogDog stream name alphabet.

The log name "log" is reserved, and will automatically capture all logging outputs generated with the "go.chromium.org/luci/common/logging" API.

type StartOption Uses

type StartOption func(*State)

StartOption is an object which can be passed to the Start function, and modifies the behavior of the luciexe/build library.

StartOptions are exclusively constructed from the Opt* functions in this package.

StartOptions are all unique per Start (i.e. you can only pass one of a kind per option to Start).

func OptLogsink Uses

func OptLogsink(c *streamclient.Client) StartOption

OptLogsink allows you to associate a streamclient with the started build.

See `streamclient.New` and `streamclient.NewFake` for how to create a client suitable to your needs (note that this includes a local filesystem option).

If a logsink is configured, it will be used as the output destination for the go.chromium.org/luci/common/logging library, and will recieve all data written via the Loggable interface.

If no logsink is configured, the go.chromium.org/luci/common/logging library will be unaffected, and data written to the Loggable interface will go to an ioutil.NopWriteCloser.

func OptOutputProperties Uses

func OptOutputProperties(writeFnptr, mergeFnptr interface{}) StartOption

OptOutputProperties allows you to register a property writer for the top-level output properties of the build.

The registered message must not have any fields which conflict with a namespace reserved with MakePropertyModifier, or this will panic.

This works like MakePropertyModifier, except that it works at the top level (i.e. no namespace).

Usage:

var writer func(context.Context, *MyMessage)
var merger func(context.Context, *MyMessage)

// one function may be nil and will be skipped
... = Start(, ..., OptOutputProperties(&writer, &merger))

in go2 this will be:

type PropertyManipulator[T proto.Message] interface {
  Write func(context.Context, *T)
  Merge func(context.Context, *T)
}
func OptOutputProperties[T proto.Message]() (StartOption, PropertyManipulator[T])

func OptParseProperties Uses

func OptParseProperties(msg proto.Message) StartOption

OptParseProperties allows you to parse the build's Input.Properties field as JSONPB into the given protobuf message.

Message fields which overlap with property namespaces reserved by MakePropertyReader will not be populated (i.e. all property namespaces reserved with MakePropertyReader will be removed before parsing into this message).

Type mismatches (i.e. parsing a non-numeric string into an int field) will report an error and quit the build.

Example:

msg := &MyOutputMessage{}
state, ctx := Start(ctx, inputBuild, OptParseProperties(msg))
# `msg` has been populated from inputBuild.InputProperties

func OptSend Uses

func OptSend(lim rate.Limit, callback func(int64, *bbpb.Build)) StartOption

OptSend allows you to get a callback when the state of the underlying Build changes.

This callback will be called at most as frequently as `rate` allows, up to once per Build change, and is called with the version number and a copy of Build. Only one outstanding invocation of this callback can occur at once.

If new updates come in while this callback is blocking, they will apply silently in the background, and as soon as the callback returns (and rate allows), it will be invoked again with the current Build state.

Every modification of the Build state increments the version number by one, even if it doesn't result in an invocation of the callback. If your program modifies the build state from multiple threads, then the version assignment is arbitrary, but if you make 10 parallel changes, you'll see the version number jump by 10 (and you may, or may not, observe versions in between).

func OptStrictInputProperties Uses

func OptStrictInputProperties() StartOption

OptStrictInputProperties will cause the build to report an error if data is passed via Input.Properties which wasn't parsed into OptParseProperties or MakePropertyReader.

type State Uses

type State struct {
    // contains filtered or unexported fields
}

State is the state of the current Build.

This is properly initialized with the Start function, and as long as it isn't "End"ed, you can manipulate it with the State's various methods.

The State is preserved in the context.Context for use with the ScheduleStep and StartStep functions. These will add a new manipulatable step to the build State.

All manipulations to the build State will result in an invocation of the configured Send function (see OptSend).

func Start Uses

func Start(ctx context.Context, initial *bbpb.Build, opts ...StartOption) (*State, context.Context, error)

Start is the 'inner' entrypoint to this library.

If you're writing a standalone luciexe binary, see `Main` and `MainWithOutput`.

This function clones `initial` as the basis of all state updates (see OptSend) and MakePropertyReader declarations. This also initializes the build State in `ctx` and returns the manipulable State object.

You must End the returned State. To automatically map errors and panics to their correct visual representation, End the State like:

var err error
state, ctx := build.Start(ctx, initialBuild, ...)
defer func() { state.End(err) }()

err = opThatErrsOrPanics(ctx)

NOTE: A panic will still crash the program as usual. This does NOT `recover()` the panic. Please use conventional Go error handling and control flow mechanisms.

func (*State) End Uses

func (s *State) End(err error)

End sets the build's final status, according to `err` (See ExtractStatus).

End will also be able to set INFRA_FAILURE status and log additional information if the program is panic'ing.

End must be invoked like:

var err error
state, ctx := build.Start(ctx, initialBuild, ...)
defer func() { state.End(err) }()

err = opThatErrsOrPanics(ctx)

NOTE: A panic will still crash the program as usual. This does NOT `recover()` the panic. Please use conventional Go error handling and control flow mechanisms.

func (*State) Log Uses

func (s *State) Log(name string, opts ...streamclient.Option) io.Writer

Log creates a new build-level line-oriented text log stream with the given name.

You must close the stream when you're done with it.

func (*State) LogDatagram Uses

func (s *State) LogDatagram(name string, opts ...streamclient.Option) streamclient.DatagramWriter

LogDatagram creates a new build-level datagram log stream with the given name. Each call to WriteDatagram will produce a single datagram message in the stream.

You must close the stream when you're done with it.

func (*State) Modify Uses

func (s *State) Modify(cb func(*View))

Modify allows you to atomically manipulate the View on the build State.

Blocking in Modify will block other callers of Modify and Set*, as well as the ability for the build State to be sent (with the function set by OptSend).

The Set* methods should be preferred unless you need to read/modify/write View items.

func (*State) SetCritical Uses

func (s *State) SetCritical(critical bbpb.Trinary)

SetCritical atomically sets the build's Critical field.

func (*State) SetGitilesCommit Uses

func (s *State) SetGitilesCommit(gc *bbpb.GitilesCommit)

SetGitilesCommit atomically sets the GitilesCommit.

This will make a copy of the GitilesCommit object to store in the build State.

func (*State) SetSummaryMarkdown Uses

func (s *State) SetSummaryMarkdown(summaryMarkdown string)

SetSummaryMarkdown atomically sets the build's SummaryMarkdown field.

type Step Uses

type Step struct {
    // contains filtered or unexported fields
}

Step represents the state of a single step.

This is properly initialized by the StartStep and ScheduleStep functions.

func ScheduleStep Uses

func ScheduleStep(ctx context.Context, name string) (*Step, context.Context)

ScheduleStep is like StartStep, except that it leaves the new step in the SCHEDULED status, and does not set a StartTime.

The step will move to STARTED when calling any other methods on the Step, when creating a sub-Step, or if you explicitly call Step.Start().

func StartStep Uses

func StartStep(ctx context.Context, name string) (*Step, context.Context)

StartStep adds a new step to the build.

The step will have a "STARTED" status with a StartTime.

The returned context is updated so that calling StartStep/ScheduleStep on it will create sub-steps.

If `name` contains `|` this function will panic, since this is a reserved character for delimiting hierarchy in steps.

Duplicate step names will be disambiguated by appending " (N)" for the 2nd, 3rd, etc. duplicate.

The returned context will have `name` embedded in it; Calling StartStep or ScheduleStep with this context will generate a sub-step.

The returned context also has an updated `environ.FromCtx` containing a unique $LOGDOG_NAMESPACE value. If you launch a subprocess, you should use this environment to correctly namespace any logdog log streams your subprocess attempts to open. Using `go.chromium.org/luci/luciexe/build/exec` does this automatically.

You MUST call Step.End. To automatically map errors and panics to their correct visual representation, End the Step like:

var err error
step, ctx := build.StartStep(ctx, "Step name")
defer func() { step.End(err) }()

err = opThatErrsOrPanics(ctx)

NOTE: A panic will still crash the program as usual. This does NOT `recover()` the panic. Please use conventional Go error handling and control flow mechanisms.

func (*Step) End Uses

func (s *Step) End(err error)

End sets the step's final status, according to `err` (See ExtractStatus).

End will also be able to set INFRA_FAILURE status and log additional information if the program is panic'ing.

End'ing a Step will Cancel the context associated with this step (returned from StartStep or ScheduleStep).

End must be invoked like:

var err error
step, ctx := build.StartStep(ctx, ...)  // or build.ScheduleStep
defer func() { step.End(err) }()

err = opThatErrsOrPanics()

NOTE: A panic will still crash the program as usual. This does NOT `recover()` the panic. Please use conventional Go error handling and control flow mechanisms.

func (*Step) Log Uses

func (s *Step) Log(name string, opts ...streamclient.Option) io.Writer

Log creates a new step-level line-oriented text log stream with the given name.

The stream will close when the step is End'd.

func (*Step) LogDatagram Uses

func (s *Step) LogDatagram(name string, opts ...streamclient.Option) streamclient.DatagramWriter

LogDatagram creates a new step-level datagram log stream with the given name. Each call to WriteDatagram will produce a single datagram message in the stream.

The stream will close when the step is End'd.

func (*Step) Modify Uses

func (s *Step) Modify(cb func(*StepView))

Modify allows you to atomically manipulate the StepView for this Step.

Blocking in Modify will block other callers of Modify and Set*, as well as the ability for the build State to be sent (with the function set by OptSend).

The Set* methods should be preferred unless you need to read/modify/write View items.

This starts the step if it's still SCHEDULED.

func (*Step) SetSummaryMarkdown Uses

func (s *Step) SetSummaryMarkdown(summaryMarkdown string)

SetSummaryMarkdown atomically sets the step's SummaryMarkdown field.

func (*Step) Start Uses

func (s *Step) Start()

Start will change the status of this Step from SCHEDULED to STARTED and initializes StartTime.

This must only be called for ScheduleStep invocations. If the step is already started (e.g. it was produced via StartStep() or Start() was already called), this does nothing.

type StepView Uses

type StepView struct {
    SummaryMarkdown string
}

StepView is a window into the build State.

You can obtain/manipulate this with the Step.Modify method.

type View Uses

type View struct {
    SummaryMarkdown string
    Critical        bbpb.Trinary
    GitilesCommit   *bbpb.GitilesCommit
}

View is a window into the build State.

You can obtain/manipulate this with the State.Modify method.

Bugs

When OptLogsink is used and `logging` output is redirected to a Step log entitled "log", the current log format is reset to `gologger.StdFormat` instead of preserving the current log format from the context.

Package build imports 29 packages (graph). Updated 2021-01-21. Refresh now. Tools for package owners.