build: Index | Files

package buildlet

import ""

Package buildlet contains client tools for working with a buildlet server.


Package Files

buildlet.go buildletclient.go ec2.go gce.go keypair.go kube.go remote.go


const (
    // GomoteCreateStreamVersion is the gomote protocol version at which JSON streamed responses started.
    GomoteCreateStreamVersion = "20191119"

    // GomoteCreateMinVersion is the oldest "gomote create" protocol version that's still supported.
    GomoteCreateMinVersion = "20160922"


var (
    // TODO(evanbrown): resource requirements should be
    // defined per-builder in dashboard/builders.go
    BuildletCPU      = api.MustParse("2")         // 2 Cores
    BuildletCPULimit = api.MustParse("8")         // 8 Cores
    BuildletMemory   = api.MustParse("4000000Ki") // 4,000,000Ki RAM
var ErrClosed = errors.New("buildlet: Client closed")
var ErrTimeout = errors.New("buildlet: timeout waiting for command to complete")
var GCEGate func()

GCEGate optionally specifies a function to run before any GCE API call. It's intended to be used to bound QPS rate to GCE.

var NoKeyPair = KeyPair{}

NoKeyPair is used by the coordinator to speak http directly to buildlets, inside their firewall, without TLS.

func DestroyVM Uses

func DestroyVM(ts oauth2.TokenSource, proj, zone, instance string) error

DestroyVM sends a request to delete a VM. Actual VM description is currently (2015-01-19) very slow for no good reason. This function returns once it's been requested, not when it's done.

func RegisterFlags Uses

func RegisterFlags()

RegisterFlags registers "user" and "staging" flags that control the behavior of NewCoordinatorClientFromFlags. These are used by remote client commands like gomote.

type Client Uses

type Client struct {
    // contains filtered or unexported fields

A Client interacts with a single buildlet.

func NewClient Uses

func NewClient(ipPort string, kp KeyPair) *Client

NewClient returns a *Client that will manipulate ipPort, authenticated using the provided keypair.

This constructor returns immediately without testing the host or auth.

func StartNewVM Uses

func StartNewVM(creds *google.Credentials, buildEnv *buildenv.Environment, instName, hostType string, opts VMOpts) (*Client, error)

StartNewVM boots a new VM on GCE and returns a buildlet client configured to speak to it.

func StartPod Uses

func StartPod(ctx context.Context, kubeClient *kubernetes.Client, podName, hostType string, opts PodOpts) (*Client, error)

StartPod creates a new pod on a Kubernetes cluster and returns a buildlet client configured to speak to it.

func (*Client) Close Uses

func (c *Client) Close() error

Closes destroys and closes down the buildlet, destroying all state immediately.

func (*Client) ConnectSSH Uses

func (c *Client) ConnectSSH(user, authorizedPubKey string) (net.Conn, error)

ConnectSSH opens an SSH connection to the buildlet for the given username. The authorizedPubKey must be a line from an ~/.ssh/authorized_keys file and correspond to the private key to be used to communicate over the net.Conn.

func (*Client) DestroyVM Uses

func (c *Client) DestroyVM(ts oauth2.TokenSource, proj, zone, instance string) error

DestroyVM shuts down the buildlet and destroys the VM instance.

func (*Client) Exec Uses

func (c *Client) Exec(ctx context.Context, cmd string, opts ExecOpts) (remoteErr, execErr error)

Exec runs cmd on the buildlet.

Two errors are returned: one is whether the command succeeded remotely (remoteErr), and the second (execErr) is whether there were system errors preventing the command from being started or seen to completition. If execErr is non-nil, the remoteErr is meaningless.

If the context's deadline is exceeded, the returned execErr is ErrTimeout.

func (*Client) GCEInstanceName Uses

func (c *Client) GCEInstanceName() string

GCEInstanceName gets an instance name for GCE buildlets. This value differs from the buildlet name used in the CLI and web interface. For non-GCE buildlets, this will return an empty string.

func (*Client) GetTar Uses

func (c *Client) GetTar(ctx context.Context, dir string) (io.ReadCloser, error)

GetTar returns a .tar.gz stream of the given directory, relative to the buildlet's work dir. The provided dir may be empty to get everything.

func (*Client) IPPort Uses

func (c *Client) IPPort() string

func (*Client) IsBroken Uses

func (c *Client) IsBroken() bool

IsBroken reports whether this client is broken in some way.

func (*Client) ListDir Uses

func (c *Client) ListDir(ctx context.Context, dir string, opts ListDirOpts, fn func(DirEntry)) error

ListDir lists the contents of a directory. The fn callback is run for each entry. The directory dir itself is not included.

func (*Client) MarkBroken Uses

func (c *Client) MarkBroken()

MarkBroken marks this client as broken in some way.

func (*Client) Name Uses

func (c *Client) Name() string

Name returns the name of this buildlet. It returns the first non-empty string from the name given to SetName, its remote buildlet name, its ip:port, or "(unnamed-buildlet)" in the case where ip:port is empty because there's a custom dialer.

func (*Client) ProxyRoundTripper Uses

func (c *Client) ProxyRoundTripper() http.RoundTripper

ProxyRoundTripper returns a RoundTripper that sends HTTP requests directly through to the underlying buildlet, adding auth and X-Buildlet-Proxy headers as necessary. This is really only intended for use by the coordinator.

func (*Client) ProxyTCP Uses

func (c *Client) ProxyTCP(port int) (io.ReadWriteCloser, error)

ProxyTCP connects to the given port on the remote buildlet. The buildlet client must currently be a gomote client (RemoteName != "") and the target type must be a VM type running on GCE. This was primarily created for RDP to Windows machines, but it might get reused for other purposes in the future.

func (*Client) Put Uses

func (c *Client) Put(ctx context.Context, r io.Reader, path string, mode os.FileMode) error

Put writes the provided file to path (relative to workdir) and sets mode.

func (*Client) PutTar Uses

func (c *Client) PutTar(ctx context.Context, r io.Reader, dir string) error

PutTar writes files to the remote buildlet, rooted at the relative directory dir. If dir is empty, they're placed at the root of the buildlet's work directory. The dir is created if necessary. The Reader must be of a tar.gz file.

func (*Client) PutTarFromURL Uses

func (c *Client) PutTarFromURL(ctx context.Context, tarURL, dir string) error

PutTarFromURL tells the buildlet to download the tar.gz file from tarURL and write it to dir, a relative directory from the workdir. If dir is empty, they're placed at the root of the buildlet's work directory. The dir is created if necessary. The url must be of a tar.gz file.

func (*Client) RemoteName Uses

func (c *Client) RemoteName() string

RemoteName returns the name of this client's buildlet on the coordinator. If this buildlet isn't a remote buildlet created via gomote, this returns the empty string.

func (*Client) RemoveAll Uses

func (c *Client) RemoveAll(ctx context.Context, paths ...string) error

RemoveAll deletes the provided paths, relative to the work directory.

func (*Client) SetDescription Uses

func (c *Client) SetDescription(v string)

SetDescription sets a short description of where the buildlet connection came from. This is used by the build coordinator status page, mostly for debugging.

func (*Client) SetDialer Uses

func (c *Client) SetDialer(dialer func(context.Context) (net.Conn, error))

SetDialer sets the function that creates a new connection to the buildlet. By default, net.Dialer.DialContext is used. SetDialer has effect only when TLS isn't used.

TODO(bradfitz): this is only used for ssh connections to buildlets, which previously required the client to do its own net.Dial + upgrade request. But now that the net/http client supports read/write bodies for protocol upgrades, we could change how ssh works and delete this.

func (*Client) SetGCEInstanceName Uses

func (c *Client) SetGCEInstanceName(v string)

SetGCEInstanceName sets an instance name for GCE buildlets. This value differs from the buildlet name used in the CLI and web interface.

func (*Client) SetHTTPClient Uses

func (c *Client) SetHTTPClient(httpClient *http.Client)

SetHTTPClient replaces the underlying HTTP client. It should only be called before the Client is used.

func (*Client) SetName Uses

func (c *Client) SetName(name string)

func (*Client) SetOnHeartbeatFailure Uses

func (c *Client) SetOnHeartbeatFailure(fn func())

SetOnHeartbeatFailure sets a function to be called when heartbeats against this builder fail, or when the client is destroyed with Close. The function fn is never called more than once. SetOnHeartbeatFailure must be set before any use of the buildlet.

func (*Client) SetReleaseMode Uses

func (c *Client) SetReleaseMode(v bool)

SetReleaseMode sets whether this client is being used in "release mode", to prepare the final binaries to be shipped to users. All this mode does for now is disable pargzip multi-stream gzip files. See

SetReleaseMode must be set before using the client.

func (*Client) Status Uses

func (c *Client) Status(ctx context.Context) (Status, error)

Status returns an Status value describing this buildlet.

func (*Client) String Uses

func (c *Client) String() string

func (*Client) URL Uses

func (c *Client) URL() string

URL returns the buildlet's URL prefix, without a trailing slash.

func (*Client) WorkDir Uses

func (c *Client) WorkDir(ctx context.Context) (string, error)

WorkDir returns the absolute path to the buildlet work directory.

type CoordinatorClient Uses

type CoordinatorClient struct {
    // Auth specifies how to authenticate to the coordinator.
    Auth UserPass

    // Instance optionally specifies the build coordinator to connect
    // to. If zero, the production coordinator is used.
    Instance build.CoordinatorInstance
    // contains filtered or unexported fields

A CoordinatorClient makes calls to the build coordinator.

func NewCoordinatorClientFromFlags Uses

func NewCoordinatorClientFromFlags() (*CoordinatorClient, error)

NewCoordinatorClientFromFlags constructs a CoordinatorClient for the current user.

func (*CoordinatorClient) CreateBuildlet Uses

func (cc *CoordinatorClient) CreateBuildlet(builderType string) (*Client, error)

CreateBuildlet creates a new buildlet of the given builder type on cc.

This takes a builderType (instead of a hostType), but the returned buildlet can be used as any builder that has the same underlying buildlet type. For instance, a linux-amd64 buildlet can act as either linux-amd64 or linux-386-387.

It may expire at any time. To release it, call Client.Close.

func (*CoordinatorClient) CreateBuildletWithStatus Uses

func (cc *CoordinatorClient) CreateBuildletWithStatus(builderType string, status func(types.BuildletWaitStatus)) (*Client, error)

CreateBuildletWithStatus is like CreateBuildlet but accepts an optional status callback.

func (*CoordinatorClient) NamedBuildlet Uses

func (cc *CoordinatorClient) NamedBuildlet(name string) (*Client, error)

NamedBuildlet returns a buildlet client for the named remote buildlet. Names are not validated. Use Client.Status to check whether the client works.

func (*CoordinatorClient) RemoteBuildlets Uses

func (cc *CoordinatorClient) RemoteBuildlets() ([]RemoteBuildlet, error)

type DirEntry Uses

type DirEntry struct {
    // contains filtered or unexported fields

DirEntry is the information about a file on a buildlet.

func (DirEntry) Digest Uses

func (de DirEntry) Digest() string

Digest returns the SHA-1 digest of the file, such as "da39a3ee5e6b4b0d3255bfef95601890afd80709". It returns the empty string if the digest isn't included.

func (DirEntry) IsDir Uses

func (de DirEntry) IsDir() bool

IsDir reports whether de describes a directory. That is, it tests for the os.ModeDir bit being set in de.Perm().

func (DirEntry) Name Uses

func (de DirEntry) Name() string

Name returns the relative path to the file, such as "src/net/http/" or "src/net/http/jar.go".

func (DirEntry) Perm Uses

func (de DirEntry) Perm() string

Perm returns the permission bits in string form, such as "-rw-r--r--" or "drwxr-xr-x".

func (DirEntry) String Uses

func (de DirEntry) String() string

type EC2Client Uses

type EC2Client struct {
    // contains filtered or unexported fields

EC2Client is the client used to create buildlets on EC2.

func NewEC2Client Uses

func NewEC2Client(client *cloud.AWSClient) *EC2Client

NewEC2Client creates a new EC2Client.

func (*EC2Client) StartNewVM Uses

func (c *EC2Client) StartNewVM(ctx context.Context, buildEnv *buildenv.Environment, hconf *dashboard.HostConfig, vmName, hostType string, opts *VMOpts) (*Client, error)

StartNewVM boots a new VM on EC2, waits until the client is accepting connections on the configured port and returns a buildlet client configured communicate with it.

type ExecOpts Uses

type ExecOpts struct {
    // Output is the output of stdout and stderr.
    // If nil, the output is discarded.
    Output io.Writer

    // Dir is the directory from which to execute the command.
    // It is optional. If not specified, it defaults to the directory of
    // the command, or the work directory if SystemLevel is set.
    Dir string

    // Args are the arguments to pass to the cmd given to Client.Exec.
    Args []string

    // ExtraEnv are KEY=VALUE pairs to append to the buildlet
    // process's environment.
    ExtraEnv []string

    // Path, if non-nil, specifies the PATH variable of the executed
    // process's environment. A non-nil empty list clears the path.
    // The following expansions apply:
    //   - the string "$PATH" expands to any existing PATH element(s)
    //   - the substring "$WORKDIR" expands to buildlet's temp workdir
    // After expansions, the list is joined with an OS-specific list
    // separator and supplied to the executed process as its PATH
    // environment variable.
    Path []string

    // SystemLevel controls whether the command is run outside of
    // the buildlet's environment.
    SystemLevel bool

    // Debug, if true, instructs to the buildlet to print extra debug
    // info to the output before the command begins executing.
    Debug bool

    // OnStartExec is an optional hook that runs after the 200 OK
    // response from the buildlet, but before the output begins
    // writing to Output.
    OnStartExec func()

ExecOpts are options for a remote command invocation.

type KeyPair Uses

type KeyPair struct {
    CertPEM string
    KeyPEM  string

KeyPair is the TLS public certificate PEM file and its associated private key PEM file that a builder will use for its HTTPS server. The zero value means no HTTPs, which is used by the coordinator for machines running within a firewall.

func NewKeyPair Uses

func NewKeyPair() (KeyPair, error)

func (KeyPair) IsZero Uses

func (kp KeyPair) IsZero() bool

func (KeyPair) Password Uses

func (kp KeyPair) Password() string

Password returns the SHA1 of the KeyPEM. This is used as the HTTP Basic Auth password.

type ListDirOpts Uses

type ListDirOpts struct {
    // Recursive controls whether the directory is listed
    // recursively.
    Recursive bool

    // Skip are the directories to skip, relative to the directory
    // passed to ListDir. Each item should contain only forward
    // slashes and not start or end in slashes.
    Skip []string

    // Digest controls whether the SHA-1 digests of regular files
    // are returned.
    Digest bool

ListDirOpts are options for Client.ListDir.

type PodOpts Uses

type PodOpts struct {
    // ProjectID is the GCE project ID. Required.
    ProjectID string

    // ImageRegistry specifies the Docker registry Kubernetes
    // will use to create the pod. Required.
    ImageRegistry string

    // TLS optionally specifies the TLS keypair to use.
    // If zero, http without auth is used.
    TLS KeyPair

    // Description optionally describes the pod.
    Description string

    // Labels optionally specify key=value strings that Kubernetes
    // can use to filter and group pods.
    Labels map[string]string

    // DeleteIn optionally specifies a duration at which
    // to delete the pod.
    DeleteIn time.Duration

    // OnPodCreating optionally specifies a hook to run synchronously
    // after the pod create request has been made, but before the create
    // has succeeded.
    OnPodCreating func()

    // OnPodCreated optionally specifies a hook to run synchronously
    // after the pod create request succeeds.
    OnPodCreated func()

    // OnGotPodInfo optionally specifies a hook to run synchronously
    // after the pod Get call.
    OnGotPodInfo func()

PodOpts control how new pods are started.

type RemoteBuildlet Uses

type RemoteBuildlet struct {
    HostType    string // "host-linux-jessie"
    BuilderType string // "linux-386-387"
    Name        string // "buildlet-adg-openbsd-386-2"
    Created     time.Time
    Expires     time.Time

type Status Uses

type Status struct {
    Version int // buildlet version, coordinator rejects any value less than 1.

Status provides status information about the buildlet.

A coordinator can use the provided information to decide what, if anything, to do with a buildlet.

type UserPass Uses

type UserPass struct {
    Username string // "user-$USER"
    Password string // buildlet key

type VM Uses

type VM struct {
    // Name is the name of the GCE VM instance.
    // For example, it's of the form "mote-bradfitz-plan9-386-foo",
    // and not "plan9-386-foo".
    Name   string
    IPPort string
    TLS    KeyPair
    Type   string // buildlet type

func ListVMs Uses

func ListVMs(ts oauth2.TokenSource, proj, zone string) ([]VM, error)

ListVMs lists all VMs.

type VMOpts Uses

type VMOpts struct {
    // Zone is the GCE zone to create the VM in.
    // Optional; defaults to provided build environment's zone.
    Zone string

    // ProjectID is the GCE project ID (e.g. "foo-bar-123", not
    // the numeric ID).
    // Optional; defaults to provided build environment's project ID ("name").
    ProjectID string

    // TLS optionally specifies the TLS keypair to use.
    // If zero, http without auth is used.
    TLS KeyPair

    // Optional description of the VM.
    Description string

    // Optional metadata to put on the instance.
    Meta map[string]string

    // DeleteIn optionally specifies a duration at which
    // to delete the VM.
    // If zero, a reasonable default is used.
    // Negative means no deletion timeout.
    DeleteIn time.Duration

    // OnInstanceRequested optionally specifies a hook to run synchronously
    // after the computeService.Instances.Insert call, but before
    // waiting for its operation to proceed.
    OnInstanceRequested func()

    // OnInstanceCreated optionally specifies a hook to run synchronously
    // after the instance operation succeeds.
    OnInstanceCreated func()

    // OnInstanceCreated optionally specifies a hook to run synchronously
    // after the computeService.Instances.Get call.
    // Only valid for GCE resources.
    OnGotInstanceInfo func(*compute.Instance)

    // OnInstanceCreated optionally specifies a hook to run synchronously
    // after the EC2 instance information is retrieved.
    // Only valid for EC2 resources.
    OnGotEC2InstanceInfo func(*cloud.Instance)

    // OnBeginBuildletProbe optionally specifies a hook to run synchronously
    // before StartNewVM tries to hit buildletURL to see if it's up yet.
    OnBeginBuildletProbe func(buildletURL string)

    // OnEndBuildletProbe optionally specifies a hook to run synchronously
    // after StartNewVM tries to hit the buildlet's URL to see if it's up.
    // The hook parameters are the return values from http.Get.
    OnEndBuildletProbe func(*http.Response, error)

    // SkipEndpointVerification does not verify that the builder is listening
    // on port 80 or 443 before creating a buildlet client.
    SkipEndpointVerification bool

VMOpts control how new VMs are started.

Package buildlet imports 39 packages (graph) and is imported by 18 packages. Updated 2021-01-14. Refresh now. Tools for package owners.