lxd: github.com/lxc/lxd/lxd/cluster Index | Files | Directories

package cluster

import "github.com/lxc/lxd/lxd/cluster"


Package Files

config.go connect.go events.go gateway.go heartbeat.go info.go membership.go migrate.go notify.go options.go recover.go resolve.go tls.go upgrade.go


var ConfigSchema = config.Schema{
    "backups.compression_algorithm":  {Default: "gzip", Validator: validateCompression},
    "cluster.offline_threshold":      {Type: config.Int64, Default: offlineThresholdDefault(), Validator: offlineThresholdValidator},
    "cluster.images_minimal_replica": {Type: config.Int64, Default: "3", Validator: imageMinimalReplicaValidator},
    "cluster.max_voters":             {Type: config.Int64, Default: "3", Validator: maxVotersValidator},
    "cluster.max_standby":            {Type: config.Int64, Default: "2", Validator: maxStandByValidator},
    "core.https_allowed_headers":     {},
    "core.https_allowed_methods":     {},
    "core.https_allowed_origin":      {},
    "core.https_allowed_credentials": {Type: config.Bool},
    "core.proxy_http":                {},
    "core.proxy_https":               {},
    "core.proxy_ignore_hosts":        {},
    "core.trust_password":            {Hidden: true, Setter: passwordSetter},
    "core.trust_ca_certificates":     {Type: config.Bool},
    "candid.api.key":                 {},
    "candid.api.url":                 {},
    "candid.domains":                 {},
    "candid.expiry":                  {Type: config.Int64, Default: "3600"},
    "images.auto_update_cached":      {Type: config.Bool, Default: "true"},
    "images.auto_update_interval":    {Type: config.Int64, Default: "6"},
    "images.compression_algorithm":   {Default: "gzip", Validator: validateCompression},
    "images.remote_cache_expiry":     {Type: config.Int64, Default: "10"},
    "maas.api.key":                   {},
    "maas.api.url":                   {},
    "rbac.agent.url":                 {},
    "rbac.agent.username":            {},
    "rbac.agent.private_key":         {},
    "rbac.agent.public_key":          {},
    "rbac.api.expiry":                {Type: config.Int64, Default: "3600"},
    "rbac.api.key":                   {},
    "rbac.api.url":                   {},
    "rbac.expiry":                    {Type: config.Int64, Default: "3600"},

    "storage.lvm_fstype":           {Setter: deprecatedStorage, Default: "ext4"},
    "storage.lvm_mount_options":    {Setter: deprecatedStorage, Default: "discard"},
    "storage.lvm_thinpool_name":    {Setter: deprecatedStorage, Default: "LXDThinPool"},
    "storage.lvm_vg_name":          {Setter: deprecatedStorage},
    "storage.lvm_volume_size":      {Setter: deprecatedStorage, Default: "10GiB"},
    "storage.zfs_pool_name":        {Setter: deprecatedStorage},
    "storage.zfs_remove_snapshots": {Setter: deprecatedStorage, Type: config.Bool},
    "storage.zfs_use_refquota":     {Setter: deprecatedStorage, Type: config.Bool},

ConfigSchema defines available server configuration keys.

var ErrNotLeader = fmt.Errorf("Not leader")

ErrNotLeader signals that a node not the leader.

var SchemaVersion = cluster.SchemaVersion

SchemaVersion holds the version of the cluster database schema.

func Accept Uses

func Accept(state *state.State, gateway *Gateway, name, address string, schema, api, arch int) ([]db.RaftNode, error)

Accept a new node and add it to the cluster.

This instance must already be clustered.

Return an updated list raft database nodes (possibly including the newly accepted node).

func Assign Uses

func Assign(state *state.State, gateway *Gateway, nodes []db.RaftNode) error

Assign a new role to the local dqlite node.

func Bootstrap Uses

func Bootstrap(state *state.State, gateway *Gateway, name string) error

Bootstrap turns a non-clustered LXD instance into the first (and leader) node of a new LXD cluster.

This instance must already have its cluster.https_address set and be listening on the associated network address.

func ConfigGetBool Uses

func ConfigGetBool(cluster *db.Cluster, key string) (bool, error)

ConfigGetBool is a convenience for loading the cluster configuration and returning the value of a particular boolean key.

It's a deprecated API meant to be used by call sites that are not interacting with the database in a transactional way.

func ConfigGetInt64 Uses

func ConfigGetInt64(cluster *db.Cluster, key string) (int64, error)

ConfigGetInt64 is a convenience for loading the cluster configuration and returning the value of a particular key.

It's a deprecated API meant to be used by call sites that are not interacting with the database in a transactional way.

func ConfigGetString Uses

func ConfigGetString(cluster *db.Cluster, key string) (string, error)

ConfigGetString is a convenience for loading the cluster configuration and returning the value of a particular key.

It's a deprecated API meant to be used by call sites that are not interacting with the database in a transactional way.

func Connect Uses

func Connect(address string, cert *shared.CertInfo, notify bool) (lxd.InstanceServer, error)

Connect is a convenience around lxd.ConnectLXD that configures the client with the correct parameters for node-to-node communication.

If 'notify' switch is true, then the user agent will be set to the special value 'lxd-cluster-notifier', which can be used in some cases to distinguish between a regular client request and an internal cluster request.

func ConnectIfContainerIsRemote Uses

func ConnectIfContainerIsRemote(cluster *db.Cluster, project, name string, cert *shared.CertInfo, instanceType instancetype.Type) (lxd.InstanceServer, error)

ConnectIfContainerIsRemote figures out the address of the node which is running the container with the given name. If it's not the local node will connect to it and return the connected client, otherwise it will just return nil.

func ConnectIfVolumeIsRemote Uses

func ConnectIfVolumeIsRemote(cluster *db.Cluster, poolID int64, volumeName string, volumeType int, cert *shared.CertInfo) (lxd.InstanceServer, error)

ConnectIfVolumeIsRemote figures out the address of the node on which the volume with the given name is defined. If it's not the local node will connect to it and return the connected client, otherwise it will just return nil.

If there is more than one node with a matching volume name, an error is returned.

func Count Uses

func Count(state *state.State) (int, error)

Count is a convenience for checking the current number of nodes in the cluster.

func DqliteLog Uses

func DqliteLog(l client.LogLevel, format string, a ...interface{})

DqliteLog redirects dqlite's logs to our own logger

func Enabled Uses

func Enabled(node *db.Node) (bool, error)

Enabled is a convenience that returns true if clustering is enabled on this node.

func Events Uses

func Events(endpoints *endpoints.Endpoints, cluster *db.Cluster, f func(int64, api.Event)) (task.Func, task.Schedule)

Events starts a task that continuously monitors the list of cluster nodes and maintains a pool of websocket connections against all of them, in order to get notified about events.

Whenever an event is received the given callback is invoked.

func Handover Uses

func Handover(state *state.State, gateway *Gateway, address string) (string, []db.RaftNode, error)

Handover looks for a non-voter member that can be promoted to replace a the member with the given address, which is shutting down. It returns the address of such member along with an updated list of nodes, with the ne role set.

It should be called only by the current leader.

func HeartbeatNode Uses

func HeartbeatNode(taskCtx context.Context, address string, cert *shared.CertInfo, heartbeatData *APIHeartbeat) error

HeartbeatNode performs a single heartbeat request against the node with the given address.

func HeartbeatTask Uses

func HeartbeatTask(gateway *Gateway) (task.Func, task.Schedule)

HeartbeatTask returns a task function that performs leader-initiated heartbeat checks against all LXD nodes in the cluster.

It will update the heartbeat timestamp column of the nodes table accordingly, and also notify them of the current list of database nodes.

func Join Uses

func Join(state *state.State, gateway *Gateway, cert *shared.CertInfo, name string, raftNodes []db.RaftNode) error

Join makes a non-clustered LXD node join an existing cluster.

It's assumed that Accept() was previously called against the leader node, which handed the raft server ID.

The cert parameter must contain the keypair/CA material of the cluster being joined.

func Leave Uses

func Leave(state *state.State, gateway *Gateway, name string, force bool) (string, error)

Leave a cluster.

If the force flag is true, the node will leave even if it still has containers and images.

The node will only leave the raft cluster, and won't be removed from the database. That's done by Purge().

Upon success, return the address of the leaving node.

This function must be called by the cluster leader.

func List Uses

func List(state *state.State) ([]api.ClusterMember, error)

List the nodes of the cluster.

func ListDatabaseNodes Uses

func ListDatabaseNodes(database *db.Node) ([]string, error)

ListDatabaseNodes returns a list of database node names.

func MaybeUpdate Uses

func MaybeUpdate(state *state.State) error

MaybeUpdate Check this node's version and possibly run LXD_CLUSTER_UPDATE.

func MigrateToDqlite10 Uses

func MigrateToDqlite10(dir string) error

MigrateToDqlite10 migrates the on-disk global database format to dqlite 1.0.

func NotifyUpgradeCompleted Uses

func NotifyUpgradeCompleted(state *state.State, cert *shared.CertInfo) error

NotifyUpgradeCompleted sends a notification to all other nodes in the cluster that any possible pending database update has been applied, and any nodes which was waiting for this node to be upgraded should re-check if it's okay to move forward.

func Purge Uses

func Purge(cluster *db.Cluster, name string) error

Purge removes a node entirely from the cluster database.

func Rebalance Uses

func Rebalance(state *state.State, gateway *Gateway) (string, []db.RaftNode, error)

Rebalance the raft cluster, trying to see if we have a spare online node that we can promote to voter node if we are below membershipMaxRaftVoters, or to standby if we are below membershipMaxStandBys.

If there's such spare node, return its address as well as the new list of raft nodes.

func Recover Uses

func Recover(database *db.Node) error

Recover attempts data recovery on the cluster database.

func ResolveTarget Uses

func ResolveTarget(cluster *db.Cluster, target string) (string, error)

ResolveTarget is a convenience for handling the value ?targetNode query parameter. It returns the address of the given node, or the empty string if the given node is the local one.

func SetupTrust Uses

func SetupTrust(cert, targetAddress, targetCert, targetPassword string) error

SetupTrust is a convenience around InstanceServer.CreateCertificate that adds the given client certificate to the trusted pool of the cluster at the given address, using the given password.

type APIHeartbeat Uses

type APIHeartbeat struct {
    sync.Mutex // Used to control access to Members maps.
    Members    map[int64]APIHeartbeatMember
    Version    APIHeartbeatVersion
    Time       time.Time

    // Indicates if heartbeat contains a fresh set of node states.
    // This can be used to indicate to the receiving node that the state is fresh enough to
    // trigger node refresh activies (such as forkdns).
    FullStateList bool

APIHeartbeat contains data sent to nodes in heartbeat.

func (*APIHeartbeat) Send Uses

func (hbState *APIHeartbeat) Send(ctx context.Context, cert *shared.CertInfo, localAddress string, nodes []db.NodeInfo, delay bool)

Send sends heartbeat requests to the nodes supplied and updates heartbeat state.

func (*APIHeartbeat) Update Uses

func (hbState *APIHeartbeat) Update(fullStateList bool, raftNodes []db.RaftNode, allNodes []db.NodeInfo, offlineThreshold time.Duration)

Update updates an existing APIHeartbeat struct with the raft and all node states supplied. If allNodes provided is an empty set then this is considered a non-full state list.

type APIHeartbeatMember Uses

type APIHeartbeatMember struct {
    ID            int64     // ID field value in nodes table.
    Address       string    // Host and Port of node.
    RaftID        uint64    // ID field value in raft_nodes table, zero if non-raft node.
    RaftRole      int       // Node role in the raft cluster, from the raft_nodes table
    Raft          bool      // Deprecated, use non-zero RaftID instead to indicate raft node.
    LastHeartbeat time.Time // Last time we received a successful response from node.
    Online        bool      // Calculated from offline threshold and LastHeatbeat time.
    // contains filtered or unexported fields

APIHeartbeatMember contains specific cluster node info.

type APIHeartbeatVersion Uses

type APIHeartbeatVersion struct {
    Schema        int
    APIExtensions int

APIHeartbeatVersion contains max versions for all nodes in cluster.

type Config Uses

type Config struct {
    // contains filtered or unexported fields

Config holds cluster-wide configuration values.

func ConfigLoad Uses

func ConfigLoad(tx *db.ClusterTx) (*Config, error)

ConfigLoad loads a new Config object with the current cluster configuration values fetched from the database.

func (*Config) AutoUpdateInterval Uses

func (c *Config) AutoUpdateInterval() time.Duration

AutoUpdateInterval returns the configured images auto update interval.

func (*Config) CandidServer Uses

func (c *Config) CandidServer() (string, string, int64, string)

CandidServer returns all the Candid settings needed to connect to a server.

func (*Config) Dump Uses

func (c *Config) Dump() map[string]interface{}

Dump current configuration keys and their values. Keys with values matching their defaults are omitted.

func (*Config) HTTPSAllowedCredentials Uses

func (c *Config) HTTPSAllowedCredentials() bool

HTTPSAllowedCredentials returns the relevant CORS setting.

func (*Config) HTTPSAllowedHeaders Uses

func (c *Config) HTTPSAllowedHeaders() string

HTTPSAllowedHeaders returns the relevant CORS setting.

func (*Config) HTTPSAllowedMethods Uses

func (c *Config) HTTPSAllowedMethods() string

HTTPSAllowedMethods returns the relevant CORS setting.

func (*Config) HTTPSAllowedOrigin Uses

func (c *Config) HTTPSAllowedOrigin() string

HTTPSAllowedOrigin returns the relevant CORS setting.

func (*Config) ImagesMinimalReplica Uses

func (c *Config) ImagesMinimalReplica() int64

ImagesMinimalReplica returns the numbers of nodes for cluster images replication

func (*Config) MAASController Uses

func (c *Config) MAASController() (string, string)

MAASController the configured MAAS url and key, if any.

func (*Config) MaxStandBy Uses

func (c *Config) MaxStandBy() int64

MaxStandBy returns the maximum number of standby members in a cluster that will be assigned the stand-by role.

func (*Config) MaxVoters Uses

func (c *Config) MaxVoters() int64

MaxVoters returns the maximum number of members in a cluster that will be assigned the voter role.

func (*Config) OfflineThreshold Uses

func (c *Config) OfflineThreshold() time.Duration

OfflineThreshold returns the configured heartbeat threshold, i.e. the number of seconds before after which an unresponsive node is considered offline..

func (*Config) Patch Uses

func (c *Config) Patch(patch map[string]interface{}) (map[string]string, error)

Patch changes only the configuration keys in the given map.

Return what has actually changed.

func (*Config) ProxyHTTP Uses

func (c *Config) ProxyHTTP() string

ProxyHTTP returns the configured HTTP proxy, if any.

func (*Config) ProxyHTTPS Uses

func (c *Config) ProxyHTTPS() string

ProxyHTTPS returns the configured HTTPS proxy, if any.

func (*Config) ProxyIgnoreHosts Uses

func (c *Config) ProxyIgnoreHosts() string

ProxyIgnoreHosts returns the configured ignore-hosts proxy setting, if any.

func (*Config) RBACServer Uses

func (c *Config) RBACServer() (string, string, int64, string, string, string, string)

RBACServer returns all the Candid settings needed to connect to a server.

func (*Config) RemoteCacheExpiry Uses

func (c *Config) RemoteCacheExpiry() int64

RemoteCacheExpiry returns the configured expiration value for remote images expiration.

func (*Config) Replace Uses

func (c *Config) Replace(values map[string]interface{}) (map[string]string, error)

Replace the current configuration with the given values.

Return what has actually changed.

func (*Config) TrustCACertificates Uses

func (c *Config) TrustCACertificates() bool

TrustCACertificates returns whether client certificates are checked against a CA.

func (*Config) TrustPassword Uses

func (c *Config) TrustPassword() string

TrustPassword returns the LXD trust password for authenticating clients.

type Gateway Uses

type Gateway struct {

    // Used for the heartbeat handler
    Cluster           *db.Cluster
    HeartbeatNodeHook func(*APIHeartbeat)
    // contains filtered or unexported fields

Gateway mediates access to the dqlite cluster using a gRPC SQL client, and possibly runs a dqlite replica on this LXD node (if we're configured to do so).

func NewGateway Uses

func NewGateway(db *db.Node, cert *shared.CertInfo, options ...Option) (*Gateway, error)

NewGateway creates a new Gateway for managing access to the dqlite cluster.

When a new gateway is created, the node-level database is queried to check what kind of role this node plays and if it's exposed over the network. It will initialize internal data structures accordingly, for example starting a local dqlite server if this node is a database node.

After creation, the Daemon is expected to expose whatever http handlers the HandlerFuncs method returns and to access the dqlite cluster using the dialer returned by the DialFunc method.

func (*Gateway) Context Uses

func (g *Gateway) Context() context.Context

Context returns a cancellation context to pass to dqlite.NewDriver as option.

This context gets cancelled by Gateway.Kill() and at that point any connection failure won't be retried.

func (*Gateway) DialFunc Uses

func (g *Gateway) DialFunc() client.DialFunc

DialFunc returns a dial function that can be used to connect to one of the dqlite nodes.

func (*Gateway) HandlerFuncs Uses

func (g *Gateway) HandlerFuncs(nodeRefreshTask func(*APIHeartbeat)) map[string]http.HandlerFunc

HandlerFuncs returns the HTTP handlers that should be added to the REST API endpoint in order to handle database-related requests.

There are two handlers, one for the /internal/raft endpoint and the other for /internal/db, which handle respectively raft and gRPC-SQL requests.

These handlers might return 404, either because this LXD node is a non-clustered node not available over the network or because it is not a database node part of the dqlite cluster.

func (*Gateway) IsDqliteNode Uses

func (g *Gateway) IsDqliteNode() bool

IsDqliteNode returns true if this gateway is running a dqlite node.

func (*Gateway) Kill Uses

func (g *Gateway) Kill()

Kill is an API that the daemon calls before it actually shuts down and calls Shutdown(). It will abort any ongoing or new attempt to establish a SQL gRPC connection with the dialer (typically for running some pre-shutdown queries).

func (*Gateway) LeaderAddress Uses

func (g *Gateway) LeaderAddress() (string, error)

LeaderAddress returns the address of the current raft leader.

func (*Gateway) NodeStore Uses

func (g *Gateway) NodeStore() client.NodeStore

NodeStore returns a dqlite server store that can be used to lookup the addresses of known database nodes.

func (*Gateway) Reset Uses

func (g *Gateway) Reset(cert *shared.CertInfo) error

Reset the gateway, shutting it down and starting against from scratch using the given certificate.

This is used when disabling clustering on a node.

func (*Gateway) Shutdown Uses

func (g *Gateway) Shutdown() error

Shutdown this gateway, stopping the gRPC server and possibly the raft factory.

func (*Gateway) Snapshot Uses

func (g *Gateway) Snapshot() error

Snapshot can be used to manually trigger a RAFT snapshot

func (*Gateway) Sync Uses

func (g *Gateway) Sync()

Sync dumps the content of the database to disk. This is useful for inspection purposes, and it's also needed by the activateifneeded command so it can inspect the database in order to decide whether to activate the daemon or not.

func (*Gateway) WaitUpgradeNotification Uses

func (g *Gateway) WaitUpgradeNotification()

WaitUpgradeNotification waits for a notification from another node that all nodes in the cluster should now have been upgraded and have matching schema and API versions.

type Notifier Uses

type Notifier func(hook func(lxd.InstanceServer) error) error

Notifier is a function that invokes the given function against each node in the cluster excluding the invoking one.

func NewNotifier Uses

func NewNotifier(state *state.State, cert *shared.CertInfo, policy NotifierPolicy) (Notifier, error)

NewNotifier builds a Notifier that can be used to notify other peers using the given policy.

type NotifierPolicy Uses

type NotifierPolicy int

NotifierPolicy can be used to tweak the behavior of NewNotifier in case of some nodes are down.

const (
    NotifyAll    NotifierPolicy = iota // Requires that all nodes are up.
    NotifyAlive                        // Only notifies nodes that are alive
    NotifyTryAll                       // Attempt to notify all nodes regardless of state.

Possible notifcation policies.

type Option Uses

type Option func(*options)

Option to be passed to NewGateway to customize the resulting instance.

func Latency Uses

func Latency(latency float64) Option

Latency is a coarse grain measure of how fast/reliable network links are. This is used to tweak the various timeouts parameters of the raft algorithm. See the raft.Config structure for more details. A value of 1.0 means use the default values from hashicorp's raft package. Values closer to 0 reduce the values of the various timeouts (useful when running unit tests in-memory).

func LogLevel Uses

func LogLevel(level string) Option

LogLevel sets the logging level for messages emitted by dqlite and raft.


raftPackage raft contains the subset of hashicorp/raft needed to perform data migrations to dqlite 1.0.

Package cluster imports 49 packages (graph) and is imported by 10 packages. Updated 2020-03-30. Refresh now. Tools for package owners.