build

package
v0.0.0-...-df660c4 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 23, 2024 License: Apache-2.0 Imports: 42 Imported by: 4

Documentation

Overview

Package build implements a minimal, safe, easy-to-use, hard-to-use-wrong, 'luciexe binary' implementation in Go.

See `Start` for the entrypoint to this API.

Features:

  • Can be used in library code which is used in both luciexe and non-luciexe contexts. You won't need "two versions" of your library just to give it the appearance of steps in a LUCI build context, or to have it read or write build properties.
  • Goroutine-safe API.
  • Handles all interactions with the log sink (e.g. LogDog), and automatically reroutes the logging library if a log sink is configured (so all log messages will be preserved in a build context).
  • Handles coalescing all build mutations from multiple goroutines. Efficient throttling for outgoing updates.
  • Automatically upholds all luciexe protocol invariants; Avoids invalid Build states by construction.
  • Automatically maps errors and panics to their obvious visual representation in the build (and allows manual control in case you want something else).
  • Fully decoupled from LUCI service implementation; You can use this API with no external services (e.g. for a CLI application), backed with the real LUCI implementation (e.g. when running under BuildBucket), and in tests (you can observe all outputs from this API without any live services or heavy mocks).

No-Op Mode

When no Build is in use in the context, the library behaves in 'no-op' mode. This should enable libraries to add `build` features which gracefully degrade into pure terminal output via logging.

  • MakePropertyReader functions will return empty messages. Well-behaved libraries should handle having no configuration in the context if this is possible.
  • There will be no *State object, because there is no Start call.
  • StartStep/ScheduleStep will return a *Step which is detached. Step namespacing will still work in context (but name deduplication will not).
  • The result of State.Modify/Step.Modify (and Set) calls will be logged at DEBUG.
  • Step scheduled/started/ended messages will be logged at INFO. Ended log messages will include the final summary markdown as well.
  • Text logs will be logged line-by-line at INFO with fields set indicating which step and log they were emitted from. Debug text logs (those whose log names start with "$") will be logged at DEBUG level.
  • Non-text logs will be dropped with a WARNING indicating that they're being dropped.
  • MakePropertyReader property readers will return empty message objects.
  • MakePropertyModifier property manipulators will log their emitted properties at INFO.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func AttachStatus

func AttachStatus(err error, status bbpb.Status, details *bbpb.StatusDetails) error

AttachStatus attaches a buildbucket status (and details) to a given error.

If such an error is handled by Step.End or State.End, that step/build will have its status updated to match.

AttachStatus allows overriding the attached status if the error already has one.

This panics if the status is a non-terminal status like SCHEDULED or STARTED.

This is a no-op if the error is nil.

func ExtractStatus

func ExtractStatus(err error) (bbpb.Status, *bbpb.StatusDetails)

ExtractStatus retrieves the Buildbucket status (and details) from a given error.

This returns:

  • (SUCCESS, nil) on nil error
  • Any values attached with AttachStatus.
  • (CANCELED, nil) on context.Canceled
  • (INFRA_FAILURE, &bbpb.StatusDetails{Timeout: {}}) on context.DeadlineExceeded
  • (FAILURE, nil) otherwise

This function is used internally by Step.End and State.End, but is provided publically for completeness.

func Main

func Main(inputMsg proto.Message, writeFnptr, mergeFnptr any, cb func(context.Context, []string, *State) error)

Main implements all the 'command-line' behaviors of the luciexe 'exe' protocol, including:

  • parsing command line for "--output", "--help", etc.
  • parsing stdin (as appropriate) for the incoming Build message
  • creating and configuring a logdog client to send State evolutions.
  • Configuring a logdog client from the environment.
  • Writing Build state updates to the logdog "build.proto" stream.
  • Start'ing the build in this process.
  • End'ing the build with the returned error from your function

If `inputMsg` is nil, the top-level properties will be ignored.

If `writeFnptr` and `mergeFnptr` are nil, they're ignored. Otherwise they work as they would for MakePropertyModifier.

CLI Arguments parsed:

  • -h / --help : Print help for this binary (including input/output property type info)
  • --strict-input : Enable strict property parsing (see OptStrictInputProperties)
  • --output : luciexe "output" flag; See https://pkg.go.dev/go.chromium.org/luci/luciexe#hdr-Recursive_Invocation
  • --working-dir : The working directory to run from; Default is $PWD unless LUCIEXE_FAKEBUILD is set, in which case a temp dir is used and cleaned up after.
  • -- : Any extra arguments after a "--" token are passed to your callback as-is.

Example:

func main() {
  input := *MyInputProps{}
  var writeOutputProps func(*MyOutputProps)
  var mergeOutputProps func(*MyOutputProps)

  Main(input, &writeOutputProps, &mergeOutputProps, func(ctx context.Context, args []string, st *build.State) error {
    // actual build code here, build is already Start'd
    // input was parsed from build.Input.Properties
    writeOutputProps(&MyOutputProps{...})
    return nil // will mark the Build as SUCCESS
  })
}

Main also supports running a luciexe build locally by specifying the LUCIEXE_FAKEBUILD environment variable. The value of the variable should be a path to a file containing a JSON-encoded Build proto.

TODO(iannucci): LUCIEXE_FAKEBUILD does not support nested invocations. It should set up bbagent and butler in order to aggregate logs.

NOTE: These types are pretty bad; There's significant opportunity to improve them with Go2 generics.

func MakePropertyModifier

func MakePropertyModifier(ns string, writeFnptr, mergeFnptr any)

MakePropertyModifier allows your library/module to reserve a section of the output properties for itself.

You can use this to obtain a write function (replace contents at namespace) and/or a merge function (do proto.Merge on the current contents of that namespace). If one of the function pointers is nil, it will be skipped (at least one must be non-nil). If both function pointers are provided, their types must exactly agree.

Attempting to reserve duplicate namespaces will panic. The namespace refers to the top-level property key. It is recommended that:

  • The `ns` begins with '$'.
  • The value after the '$' is the canonical Go package name for your library.

You should call this at init()-time like:

var propWriter func(context.Context, *MyMessage)
var propMerger func(context.Context, *MyMessage)

func init() {
  // one of the two function pointers may be nil
  MakePropertyModifier("$some/namespace", &propWriter, &propMerger)
}

Note that all MakePropertyModifier invocations must happen BEFORE the build is Started. Otherwise invoking the returned writer/merger functions will panic.

In Go2 this will be less weird:

type PropertyModifier[T proto.Message] interface {
  Write(context.Context, value T) // assigns 'value'
  Merge(context.Context, value T) // does proto.Merge(current, value)
}
func MakePropertyModifier[T proto.Message](ns string) PropertyModifier[T]

func MakePropertyReader

func MakePropertyReader(ns string, fnptr any)

MakePropertyReader allows your library/module to reserve a section of the input properties for itself.

Attempting to reserve duplicate namespaces will panic. The namespace refers to the top-level property key. It is recommended that:

  • The `ns` begins with '$'.
  • The value after the '$' is the canonical Go package name for your library.

Using the generated function will parse the relevant input property namespace as JSONPB, returning the parsed message (and an error, if any).

var myPropertyReader func(context.Context) *MyPropertyMsg
func init() {
  MakePropertyReader("$some/namespace", &myPropertyReader)
}

In Go2 this will be less weird:

MakePropertyReader[T proto.Message](ns string) func(context.Context) T

Types

type Log

type Log struct {
	io.Writer
	// contains filtered or unexported fields
}

Log represents a step or build log. It can be written to directly, and also provides additional information about the log itself.

The creator of the Log is responsible for cleaning up any resources associated with it (e.g. the Step or State this was created from).

func (l *Log) UILink() string

UILink returns a URL to this log fit for surfacing in the LUCI UI.

This may return an empty string if there's no available LogDog infra being logged to, for instance in testing or during local execution where logdog streams are not sunk to the actual logdog service.

type Loggable

type Loggable interface {
	// Log creates a new log stream (by default, line-oriented text) with the
	// given name.
	//
	// To uphold the requirements of the Build proto message, duplicate log names
	// will be deduplicated with the same algorithm used for deduplicating step
	// names.
	//
	// To create a binary stream, pass streamclient.Binary() as one of the
	// options.
	//
	// The stream will close when the associated object (step or build) is End'd.
	Log(name string, opts ...streamclient.Option) *Log

	// LogDatagram creates a new datagram-oriented log stream with the given name.
	//
	// To uphold the requirements of the Build proto message, duplicate log names
	// will be deduplicated with the same algorithm used for deduplicating step
	// names.
	//
	// The stream will close when the associated object (step or build) is End'd.
	LogDatagram(name string, opts ...streamclient.Option) streamclient.DatagramWriter
}

Loggable is the common interface for build entities which have log data associated with them.

Implemented by State and Step.

Logs all have a name which is an arbitrary bit of text to identify the log to human users (it will appear as the link on the build UI page). In particular it does NOT need to conform to the LogDog stream name alphabet.

The log name "log" is reserved, and will automatically capture all logging outputs generated with the "go.chromium.org/luci/common/logging" API.

type StartOption

type StartOption func(*State)

StartOption is an object which can be passed to the Start function, and modifies the behavior of the luciexe/build library.

StartOptions are exclusively constructed from the Opt* functions in this package.

StartOptions are all unique per Start (i.e. you can only pass one of a kind per option to Start).

func OptLogsink

func OptLogsink(c *streamclient.Client) StartOption

OptLogsink allows you to associate a streamclient with the started build.

See `streamclient.New` and `streamclient.NewFake` for how to create a client suitable to your needs (note that this includes a local filesystem option).

If a logsink is configured, it will be used as the output destination for the go.chromium.org/luci/common/logging library, and will recieve all data written via the Loggable interface.

If no logsink is configured, the go.chromium.org/luci/common/logging library will be unaffected, and data written to the Loggable interface will go to an ioutil.NopWriteCloser.

func OptOutputProperties

func OptOutputProperties(writeFnptr, mergeFnptr any) StartOption

OptOutputProperties allows you to register a property writer for the top-level output properties of the build.

The registered message must not have any fields which conflict with a namespace reserved with MakePropertyModifier, or this panics.

This works like MakePropertyModifier, except that it works at the top level (i.e. no namespace) and the functions operate directly on the State (i.e. they do not take a context).

Usage

var writer func(*MyMessage)
var merger func(*MyMessage)

// one function may be nil and will be skipped
... = Start(, ..., OptOutputProperties(&writer, &merger))

in go2 this can be improved (possibly by making State a generic type):

func OptParseProperties

func OptParseProperties(msg proto.Message) StartOption

OptParseProperties allows you to parse the build's Input.Properties field as JSONPB into the given protobuf message.

Message fields which overlap with property namespaces reserved by MakePropertyReader will not be populated (i.e. all property namespaces reserved with MakePropertyReader will be removed before parsing into this message).

Type mismatches (i.e. parsing a non-numeric string into an int field) will report an error and quit the build.

Example:

msg := &MyOutputMessage{}
state, ctx := Start(ctx, inputBuild, OptParseProperties(msg))
# `msg` has been populated from inputBuild.InputProperties

func OptSend

func OptSend(lim rate.Limit, callback func(int64, *bbpb.Build)) StartOption

OptSend allows you to get a callback when the state of the underlying Build changes.

This callback will be called at most as frequently as `rate` allows, up to once per Build change, and is called with the version number and a copy of Build. Only one outstanding invocation of this callback can occur at once.

If new updates come in while this callback is blocking, they will apply silently in the background, and as soon as the callback returns (and rate allows), it will be invoked again with the current Build state.

Every modification of the Build state increments the version number by one, even if it doesn't result in an invocation of the callback. If your program modifies the build state from multiple threads, then the version assignment is arbitrary, but if you make 10 parallel changes, you'll see the version number jump by 10 (and you may, or may not, observe versions in between).

func OptStrictInputProperties

func OptStrictInputProperties() StartOption

OptStrictInputProperties will cause the build to report an error if data is passed via Input.Properties which wasn't parsed into OptParseProperties or MakePropertyReader.

type State

type State struct {
	// contains filtered or unexported fields
}

State is the state of the current Build.

This is properly initialized with the Start function, and as long as it isn't "End"ed, you can manipulate it with the State's various methods.

The State is preserved in the context.Context for use with the ScheduleStep and StartStep functions. These will add a new manipulatable step to the build State.

All manipulations to the build State will result in an invocation of the configured Send function (see OptSend).

func Start

func Start(ctx context.Context, initial *bbpb.Build, opts ...StartOption) (*State, context.Context, error)

Start is the 'inner' entrypoint to this library.

If you're writing a standalone luciexe binary, see `Main` and `MainWithOutput`.

This function clones `initial` as the basis of all state updates (see OptSend) and MakePropertyReader declarations. This also initializes the build State in `ctx` and returns the manipulable State object.

You must End the returned State. To automatically map errors and panics to their correct visual representation, End the State like:

var err error
state, ctx := build.Start(ctx, initialBuild, ...)
defer func() { state.End(err) }()

err = opThatErrsOrPanics(ctx)

NOTE: A panic will still crash the program as usual. This does NOT `recover()` the panic. Please use conventional Go error handling and control flow mechanisms.

func (*State) Build

func (s *State) Build() *bbpb.Build

Build returns a copy of the initial Build state.

This is useful to access fields such as Infra, Tags, Ancestor ids etc.

Changes to this copy will not reflect anywhere in the live Build state and not affect other calls to Build().

NOTE: It is recommended to use the PropertyModifier/PropertyReader functionality of this package to interact with Build Input Properties; They are encoded as Struct proto messages, which are extremely cumbersome to work with directly.

func (*State) End

func (s *State) End(err error)

End sets the build's final status, according to `err` (See ExtractStatus).

End will also be able to set INFRA_FAILURE status and log additional information if the program is panic'ing.

End must be invoked like:

var err error
state, ctx := build.Start(ctx, initialBuild, ...)
defer func() { state.End(err) }()

err = opThatErrsOrPanics(ctx)

NOTE: A panic will still crash the program as usual. This does NOT `recover()` the panic. Please use conventional Go error handling and control flow mechanisms.

func (*State) Log

func (s *State) Log(name string, opts ...streamclient.Option) *Log

Log creates a new step-level line-oriented text log stream with the given name. Returns a Log value which can be written to directly, but also provides additional information about the log itself.

The stream will close when the state is End'd.

func (*State) LogDatagram

func (s *State) LogDatagram(name string, opts ...streamclient.Option) streamclient.DatagramWriter

LogDatagram creates a new build-level datagram log stream with the given name. Each call to WriteDatagram will produce a single datagram message in the stream.

You must close the stream when you're done with it.

func (*State) Modify

func (s *State) Modify(cb func(*View))

Modify allows you to atomically manipulate the View on the build State.

Blocking in Modify will block other callers of Modify and Set*, as well as the ability for the build State to be sent (with the function set by OptSend).

The Set* methods should be preferred unless you need to read/modify/write View items.

func (*State) SetCritical

func (s *State) SetCritical(critical bbpb.Trinary)

SetCritical atomically sets the build's Critical field.

func (*State) SetGitilesCommit

func (s *State) SetGitilesCommit(gc *bbpb.GitilesCommit)

SetGitilesCommit atomically sets the GitilesCommit.

This will make a copy of the GitilesCommit object to store in the build State.

func (*State) SetSummaryMarkdown

func (s *State) SetSummaryMarkdown(summaryMarkdown string)

SetSummaryMarkdown atomically sets the build's SummaryMarkdown field.

func (*State) SynthesizeIOProto

func (s *State) SynthesizeIOProto(o io.Writer) error

SynthesizeIOProto synthesizes a `.proto` file from the input and ouptut property messages declared at Start() time.

type Step

type Step struct {
	// contains filtered or unexported fields
}

Step represents the state of a single step.

This is properly initialized by the StartStep and ScheduleStep functions.

func ScheduleStep

func ScheduleStep(ctx context.Context, name string) (*Step, context.Context)

ScheduleStep is like StartStep, except that it leaves the new step in the SCHEDULED status, and does not set a StartTime.

The step will move to STARTED when calling any other methods on the Step, when creating a sub-Step, or if you explicitly call Step.Start().

func StartStep

func StartStep(ctx context.Context, name string) (*Step, context.Context)

StartStep adds a new step to the build.

The step will have a "STARTED" status with a StartTime.

If `name` contains `|` this function will panic, since this is a reserved character for delimiting hierarchy in steps.

Duplicate step names will be disambiguated by appending " (N)" for the 2nd, 3rd, etc. duplicate.

The returned context has the following changes:

  1. It contains the returned *Step as the current step, which means that calling the package-level StartStep/ScheduleStep on it will create sub-steps of this one.
  2. The returned context also has an updated `environ.FromCtx` containing a unique $LOGDOG_NAMESPACE value. If you launch a subprocess, you should use this environment to correctly namespace any logdog log streams your subprocess attempts to open. Using `go.chromium.org/luci/luciexe/build/exec` does this automatically.
  3. `go.chromium.org/luci/common/logging` is wired up to a new step log stream called "log".

You MUST call Step.End. To automatically map errors and panics to their correct visual representation, End the Step like:

var err error
step, ctx := build.StartStep(ctx, "Step name")
defer func() { step.End(err) }()

err = opThatErrsOrPanics(ctx)

NOTE: A panic will still crash the program as usual. This does NOT `recover()` the panic. Please use conventional Go error handling and control flow mechanisms.

func (*Step) AddTagValue

func (s *Step) AddTagValue(key, value string)

AddTagValue sets the step's tag field by appending the new tag to the existing list.

func (*Step) End

func (s *Step) End(err error)

End sets the step's final status, according to `err` (See ExtractStatus).

End will also be able to set INFRA_FAILURE status and log additional information if the program is panic'ing.

End'ing a Step will Cancel the context associated with this step (returned from StartStep or ScheduleStep).

End must be invoked like:

var err error
step, ctx := build.StartStep(ctx, ...)  // or build.ScheduleStep
defer func() { step.End(err) }()

err = opThatErrsOrPanics()

NOTE: A panic will still crash the program as usual. This does NOT `recover()` the panic. Please use conventional Go error handling and control flow mechanisms.

func (*Step) Log

func (s *Step) Log(name string, opts ...streamclient.Option) *Log

Log creates a new step-level line-oriented text log stream with the given name. Returns a Log value which can be written to directly, but also provides additional information about the log itself.

The stream will close when the step is End'd.

func (*Step) LogDatagram

func (s *Step) LogDatagram(name string, opts ...streamclient.Option) streamclient.DatagramWriter

LogDatagram creates a new step-level datagram log stream with the given name. Each call to WriteDatagram will produce a single datagram message in the stream.

The stream will close when the step is End'd.

func (*Step) Modify

func (s *Step) Modify(cb func(*StepView))

Modify allows you to atomically manipulate the StepView for this Step.

Blocking in Modify will block other callers of Modify and Set*, as well as the ability for the build State to be sent (with the function set by OptSend).

The Set* methods should be preferred unless you need to read/modify/write View items.

This starts the step if it's still SCHEDULED.

func (*Step) ScheduleStep

func (s *Step) ScheduleStep(ctx context.Context, name string) (*Step, context.Context)

ScheduleStep will create a child step of this one with `name` in the SCHEDULED status.

This behaves identically to the package level ScheduleStep, except that the 'current step' is `s` and is not pulled from `ctx. This includes all documented behaviors around changes to the returned context.

func (*Step) SetSummaryMarkdown

func (s *Step) SetSummaryMarkdown(summaryMarkdown string)

SetSummaryMarkdown atomically sets the step's SummaryMarkdown field.

func (*Step) Start

func (s *Step) Start()

Start will change the status of this Step from SCHEDULED to STARTED and initializes StartTime.

This must only be called for ScheduleStep invocations. If the step is already started (e.g. it was produced via StartStep() or Start() was already called), this does nothing.

func (*Step) StartStep

func (s *Step) StartStep(ctx context.Context, name string) (*Step, context.Context)

StartStep will create a child step of this one with `name`.

This behaves identically to the package level StartStep, except that the 'current step' is `s` and is not pulled from `ctx. This includes all documented behaviors around changes to the returned context.

type StepView

type StepView struct {
	SummaryMarkdown string
	Tags            map[string][]string
}

StepView is a window into the build State.

You can obtain/manipulate this with the Step.Modify method.

type View

type View struct {
	SummaryMarkdown string
	Critical        bbpb.Trinary
	GitilesCommit   *bbpb.GitilesCommit
}

View is a window into the build State.

You can obtain/manipulate this with the State.Modify method.

Notes

Bugs

  • When OptLogsink is used and `logging` output is redirected to a Step log entitled "log", the current log format is reset to `gologger.StdFormat` instead of preserving the current log format from the context.

Directories

Path Synopsis
Package cv exposes CV properties to luciexe binaries for builds.
Package cv exposes CV properties to luciexe binaries for builds.
internal

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL