README
¶
Node Delete Controller
The Node Delete Controller is responsible for removing nodes from the StorageOS cluster when the node has been removed from Kubernetes.
The intention is to allow StorageOS to adapt to dynamic environments where nodes are added and removed regularly, such as when spot instances are used.
Removing nodes when they are no longer required reduces load as StorageOS will no longer check for the node recovering.
Trigger
The controller reconcile will trigger on any Kubernetes Node delete event where the node has the StorageOS CSI driver annotation.
The annotation is added by the CSI node driver registrar when StorageOS starts on the node. Once added, it is not removed.
Reconcile
When a Kubernetes node with the StorageOS CSI driver annotation is deleted, a request is made to the StorageOS API to remove the node. The StorageOS API will only allow the delete to succeed if:
- The node has been marked offline (typically discovered 20-30 seconds after it goes off the network).
- The node lock has expired in etcd. Locks expire after 30 seconds.
- The node no longer holds any master deployments. Normally the master deployment would fail over to another node after it was marked offline, but in the case where no replicas have been configured, blocking the delete allows the data to be recovered. To proceed with the node removal, delete the volume first.
If the delete request failed, it will be requeued and retried after a backoff period.
Typically there will be multiple "node still in use" log entries before the node container is shutdown and StorageOS detects that it is offline. There may also be one or more "node lock has not yet expired" messages before the node is finally removed.
If the node was not found because it has already been deleted, the delete request will be considered successful.
Garbage Collection
In case a node delete event was missed during a restart or outage, a garbage collection runs periodically. It compares the list of nodes known to StorageOS, and removes any that are no longer known to Kubernetes.
Garbage collection is run every hour by default (configurable via the
-node-delete-gc-interval flag). It can be disabled by setting
-node-delete-gc-interval to 0s.
Garbage collection is run on startup after a delay defined by the
-node-delete-gc-delay flag.
Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Controller ¶
type Controller struct {
// contains filtered or unexported fields
}
Controller implements the SyncReconciler contoller interface, deleting nodes in StorageOS when they have been detected as deleted in Kubernetes.
func NewController ¶
func NewController(api NodeDeleter, log logr.Logger) (*Controller, error)
NewController returns a Controller that implements node garbage collection in StorageOS.
func (Controller) Delete ¶
Delete receives a k8s object that's been deleted and calls the StorageOS api to remove it from management.
func (Controller) List ¶
func (c Controller) List(ctx context.Context) ([]types.NamespacedName, error)
List returns a list of nodes known to StorageOS, as NamespacedNames. This is used for garbage collection and can be expensive. The garbage collector is run in a separate goroutine periodically, not affecting the main reconciliation control-loop.
type NodeDeleter ¶
type NodeDeleter interface {
DeleteNode(ctx context.Context, key client.ObjectKey) error
ListNodes(ctx context.Context) ([]client.Object, error)
}
NodeDeleter provides access to removing nodes from StorageOS.
type Predicate ¶
type Predicate struct {
predicate.IgnoreFuncs
// contains filtered or unexported fields
}
Predicate filters events before enqueuing the keys. Ignore all but Delete events, and then filter out events from non-StorageOS nodes.
type Reconciler ¶
type Reconciler struct {
client.Client
objectv1.Reconciler
// contains filtered or unexported fields
}
Reconciler reconciles a Node object by deleting the StorageOS node object when the corresponding Kubernetes node is deleted.
func NewReconciler ¶
func NewReconciler(api NodeDeleter, k8s client.Client, gcDelay time.Duration, gcInterval time.Duration) *Reconciler
NewReconciler returns a new Node delete reconciler.
The gcInterval determines how often the periodic resync operation should be run.
func (*Reconciler) SetupWithManager ¶
func (r *Reconciler) SetupWithManager(mgr ctrl.Manager, workers int) error
SetupWithManager registers the controller with the controller manager.
Source Files
Directories