placement-policy-scheduler-plugins

Placement Policy Scheduler Plugins

A stand alone scheduler that wraps current scheduler and uses scheduler framework with two additional plugins. The scheduler will run side by side with existing kubernetes scheduler and users will have to specifically specify this scheduler to use for their workloads that perform the following:

• A scorer plugin implemented with that will be used in case “best effort” policy enforcement.
  • Extension points implemented: PreScore and Score
• A filter plugin that will be used in case “force” policy enforcement.
  • Extension points implemented: PreFilter and Filter

WARNING: This is experimental code. It is not considered production-grade by its developers, nor is it "supported" software.

Quick Start

Install

The container images for the scheduler plugin is available in the github container registry.

Quick Install

kubectl apply -f https://raw.githubusercontent.com/Azure/placement-policy-scheduler-plugins/main/deploy/kube-scheduler-configuration.yml

customresourcedefinition.apiextensions.k8s.io/placementpolicies.placement-policy.scheduling.x-k8s.io created
configmap/pp-scheduler-config created
clusterrole.rbac.authorization.k8s.io/pp-plugins-scheduler created
clusterrolebinding.rbac.authorization.k8s.io/pp-plugins-scheduler created
rolebinding.rbac.authorization.k8s.io/pp-plugins-scheduler-as-kube-scheduler created
clusterrolebinding.rbac.authorization.k8s.io/pp-plugins-scheduler-as-kube-scheduler created
serviceaccount/pp-plugins-scheduler created
deployment.apps/pp-plugins-scheduler created

Helm

helm repo add placement-policy-scheduler-plugins https://azure.github.io/placement-policy-scheduler-plugins/charts
helm repo update

helm install -n kube-system [RELEASE_NAME] placement-policy-scheduler-plugins/placement-policy-scheduler-plugins

Example config

apiVersion: placement-policy.scheduling.x-k8s.io/v1alpha1
kind: PlacementPolicy
metadata:
  name: besteffort-must
spec:
  weight: 100
  enforcementMode: BestEffort
  podSelector:
    matchLabels:
      app: nginx
  nodeSelector:
    matchLabels:
      node: want
  policy:
    action: Must
    targetSize: 40%

• enforcementMode: specifies how the policy will be enforced during scheduler. Values allowed for this field are:
  • BestEffort (default): the policy will be enforced as best effort (scorer mode).
  • Strict: the policy will be forced during scheduling.
• nodeSelector: selects the nodes where the placement policy will apply on according to action.
• podSelector: identifies which pods this placement policy will apply on
• action: policy placement action that carries the following possible values:
  • Must(default): based on the rule below pods must be placed on nodes selected by node selector MustNot: based on the rule pods
  • MustNot be placed nodes selected by node selector'
• targetSize: the number or percent of pods that can or cannot be placed on the node.
• weight: allows the engine to decide which policy to use when pods match multiple policies.

Demo

• Create a KinD cluster with the following config

kind create cluster --config https://raw.githubusercontent.com/Azure/placement-policy-scheduler-plugins/main/test/e2e/kind-config.yaml

The same node selector node: want will be used as node label for kind-worker and kind-worker2

➜ kubectl get nodes -o=custom-columns="NAME:.metadata.name,LABEL:.metadata.labels['node']"
NAME                 LABEL
kind-control-plane   <none>
kind-worker          want
kind-worker2         want
kind-worker3         unwant

• Deploy a placement policy CRD

kubectl apply -f https://raw.githubusercontent.com/Azure/placement-policy-scheduler-plugins/main/examples/v1alpha1_placementpolicy_strict_must.yml

placementpolicy.placement-policy.scheduling.x-k8s.io/strict-must created

• Deploy a ReplicaSet that will create 10 replicas

kubectl apply -f https://raw.githubusercontent.com/Azure/placement-policy-scheduler-plugins/main/examples/demo_replicaset.yml

replicaset.apps/nginx created

• Get all the pod

kubectl get po -o wide

NAME          READY   STATUS    RESTARTS   AGE   IP           NODE           NOMINATED NODE   READINESS GATES
nginx-8cr58   1/1     Running   0          76s   10.244.3.4   kind-worker3   <none>           <none>
nginx-d7js5   1/1     Running   0          76s   10.244.1.2   kind-worker2   <none>           <none>
nginx-jt527   1/1     Running   0          76s   10.244.3.6   kind-worker3   <none>           <none>
nginx-m5c86   1/1     Running   0          76s   10.244.2.2   kind-worker    <none>           <none>
nginx-qxx6m   1/1     Running   0          76s   10.244.3.2   kind-worker3   <none>           <none>
nginx-rdlzx   1/1     Running   0          76s   10.244.2.3   kind-worker    <none>           <none>
nginx-skk5z   1/1     Running   0          76s   10.244.1.3   kind-worker2   <none>           <none>
nginx-vq598   1/1     Running   0          76s   10.244.3.7   kind-worker3   <none>           <none>
nginx-xzxsb   1/1     Running   0          76s   10.244.3.3   kind-worker3   <none>           <none>
nginx-zwrsk   1/1     Running   0          76s   10.244.3.5   kind-worker3   <none>           <none>

We will find the nodes which carry the same node selector definded in the CRD have been assigned to 40% only from the workload that have been definded in the CRD targetSize

Clean up

• Delete KinD cluster

kind delete cluster

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.