reflow: github.com/grailbio/reflow/ec2cluster Index | Files | Directories

package ec2cluster

import "github.com/grailbio/reflow/ec2cluster"

Package ec2cluster implements support for maintaining elastic clusters of Reflow instances on EC2.

The EC2 instances created launch reflowlet agent processes that are given the user's profile token so that they can set up HTTPS servers that can perform mutual authentication to the reflow driver process and other reflowlets (for transferring objects) and also access external services like caching.

The VM instances are configured to terminate if they are idle on EC2's billing hour boundary. They also terminate on any fatal reflowlet error.

Index

Package Files

cloudconfig.go ec2cluster.go instance.go setup.go

type CloudFile Uses

type CloudFile struct {
    Path        string `yaml:"path,omitempty"`
    Permissions string `yaml:"permissions,omitempty"`
    Owner       string `yaml:"owner,omitempty"`
    Content     string `yaml:"content,omitempty"`
    Encoding    string `yaml:"encoding,omitempty"`
}

CloudFile is a component of the cloudConfig configuration for CoreOS. It represents a file that will be written to the filesystem.

type CloudUnit Uses

type CloudUnit struct {
    Name    string `yaml:"name,omitempty"`
    Command string `yaml:"command,omitempty"`
    Enable  bool   `yaml:"enable,omitempty"`
    Content string `yaml:"content,omitempty"`
}

CloudUnit is a component of the cloudConfig configuration for CoreOS. It represents a CoreOS unit.

type Cluster Uses

type Cluster struct {
    pool.Mux `yaml:"-"`
    // HTTPClient is used to communicate to the reflowlet servers
    // running on the individual instances. In Cluster, this is done for
    // liveness/health checking.
    HTTPClient *http.Client `yaml:"-"`
    // Logger for cluster events.
    Log *log.Logger `yaml:"-"`
    // EC2 is the EC2 API instance through which EC2 calls are made.
    EC2 ec2iface.EC2API `yaml:"-"`
    // Authenticator authenticates the ECR repository that stores the
    // Reflowlet container.
    Authenticator ecrauth.Interface `yaml:"-"`
    // InstanceTags is the set of EC2 tags attached to instances created by this Cluster.
    InstanceTags map[string]string `yaml:"-"`
    // Labels is the set of labels that should be added as EC2 tags (for informational purpose only).
    Labels pool.Labels `yaml:"-"`
    // Spot is set to true when a spot instance is desired.
    Spot bool `yaml:"spot,omitempty"`
    // InstanceProfile is the EC2 instance profile to use for the cluster instances.
    InstanceProfile string `yaml:"instanceprofile,omitempty"`
    // SecurityGroup is the EC2 security group to use for cluster instances.
    SecurityGroup string `yaml:"securitygroup,omitempty"`
    // Subnet is the id of the EC2 subnet to use for cluster instances.
    Subnet string `yaml:"subnet,omitempty"`
    // AvailabilityZone defines which AZ to spawn instances into.
    AvailabilityZone string `yaml:"availabilityzone,omitempty"`
    // Region is the AWS availability region to use for launching new EC2 instances.
    Region string `yaml:"region,omitempty"`
    // InstanceTypesMap stores the set of admissible instance types.
    // If nil, all instance types are permitted.
    InstanceTypesMap map[string]bool `yaml:"-"`
    // BootstrapImage is the URL of the image used for instance bootstrap.
    BootstrapImage string `yaml:"-"`
    // ReflowVersion is the version of reflow binary compatible with this cluster.
    ReflowVersion string `yaml:"-"`
    // MaxInstances is the maximum number of concurrent instances permitted.
    MaxInstances int `yaml:"-"`
    // DiskType is the EBS disk type to use.
    DiskType string `yaml:"disktype"`
    // DiskSpace is the number of GiB of disk space to allocate for each node.
    DiskSpace int `yaml:"diskspace"`
    // DiskSlices is the number of EBS volumes that are used. When DiskSlices > 1,
    // they are arranged in a RAID0 array to increase throughput.
    DiskSlices int `yaml:"diskslices"`
    // AMI is the VM image used to launch new instances.
    AMI string `yaml:"ami"`
    // Configuration for this Reflow instantiation. Used to provide configs to
    // EC2 instances.
    Configuration infra.Config `yaml:"-"`

    // User's public SSH key.
    SshKey string `yaml:"sshkey"`
    // AWS key name for launching instances.
    KeyName string `yaml:"keyname"`
    // Immortal determines whether instances should be made immortal.
    Immortal bool `yaml:"immortal,omitempty"`
    // CloudConfig is merged into the instance's cloudConfig before launching.
    CloudConfig cloudConfig `yaml:"cloudconfig"`
    // SpotProbeDepth is the probing depth for spot instance capacity checks.
    SpotProbeDepth int `yaml:"spotprobedepth,omitempty"`

    // Status is used to report cluster and instance status.
    Status *status.Group `yaml:"-"`

    // InstanceTypesMap defines the set of allowable EC2 instance types for
    // this cluster. If empty, all instance types are permitted.
    InstanceTypes []string `yaml:"instancetypes,omitempty"`
    // Name is the name of the cluster config, which defaults to defaultClusterName.
    // Multiple clusters can be launched/maintained simultaneously by using different names.
    Name string `yaml:"name,omitempty"`
    // contains filtered or unexported fields
}

A Cluster implements a runner.Cluster backed by EC2. The cluster expands with demand. Instances are configured so that they shut down when they are idle on a billing boundary.

No local state is stored; state is inferred from labels managed by EC2. Cluster supports safely sharing state across many processes. In this case, the processes coordinate to maintain a shared cluster, where instances can be used by any of the constituent processes. In the case of Reflow, this means that multiple runs (single or batch) share the same cluster efficiently.

func (*Cluster) Allocate Uses

func (c *Cluster) Allocate(ctx context.Context, req reflow.Requirements, labels pool.Labels) (alloc pool.Alloc, err error)

Allocate reserves an alloc with within the resource requirement boundaries form this cluster. If an existing instance can serve the request, it is returned immediately; otherwise new instance(s) are spun up to handle the allocation.

func (*Cluster) Config Uses

func (c *Cluster) Config() interface{}

Config implements infra.Provider

func (Cluster) Help Uses

func (Cluster) Help() string

Help implements infra.Provider

func (*Cluster) Init Uses

func (c *Cluster) Init(tls tls.Certs, sess *session.Session, labels pool.Labels, bootstrapimage *infra2.BootstrapImage, reflowVersion *infra2.ReflowVersion, id *infra2.User, logger *log.Logger, sshKey *infra2.SshKey) error

Init implements infra.Provider

func (*Cluster) Probe Uses

func (c *Cluster) Probe(ctx context.Context, instanceType string) (time.Duration, error)

Probe attempts to instantiate an EC2 instance of the given type and returns a duration and an error. In case of a nil error the duration represents how long it took (single data point) for a usable Reflowlet to come up on that instance type. A non-nil error means that the reflowlet failed to come up on this instance type. The error could be due to context deadline, in case we gave up waiting for it to come up.

func (*Cluster) QueryTags Uses

func (c *Cluster) QueryTags() map[string]string

QueryTags returns the list of tags to use to query for instances belonging to this cluster. This includes all InstanceTags that are set on any instance brought up by this cluster, and a "reflowlet:version" tag (set on the instance by the reflowlet once it comes up) to match the ReflowVersion of this cluster.

func (*Cluster) Setup Uses

func (c *Cluster) Setup(sess *session.Session) error

Setup sets defaults for any unset ec2 configuration values.

Directories

PathSynopsis
instances
test
volumePackage volume implements support for maintaining (EBS) volumes on an EC2 instance by watching the disk usage of the underlying disk device and resizing the EBS volumes whenever necessary based on provided parameters.

Package ec2cluster imports 44 packages (graph) and is imported by 3 packages. Updated 2020-04-23. Refresh now. Tools for package owners.