kubelet-csr-approver

kubelet-csr-approver

Kubelet CSR approver is a Kubernetes controller whose sole purpose is to auto-approve kubelet-serving Certificate Signing Request (CSR), provided these CSRs comply with a series of configurable, provider-specific, checks/verifications.

Inspired by existing projects (such as kubelet-rubber-stamp), it implements additional verifications to prevent an attacker from forging Certificates.

Kubelet CSR approver is being kept up-to-date in accordance with the most recent three Kubernetes minor releases.

Quick start

1. deploy kubelet-csr-approver on your k8s cluster using the manifests present in deploy/k8s
2. change the /var/lib/kubelet/config.yaml file and restart your kubelet once having included the following field: yaml serverTLSBootstrap: true
3. at that point, there should be a number of CSRs on your cluster, that the kubelet-csr-approver will approve (or deny) depending on the deployment parameters you have set.

Parameters

The most important parameters (configurable through either flags or environment variables) are:

• --provider-regex or PROVIDER_REGEX lets you decide which hostnames can be approved or not
  e.g. if all your nodes follow a naming convention (say node-randomstr1234.int.company.ch), your regex could look like ^node-\w*\.int\.company\.ch$
• --max-expiration-sec or MAX_EXPIRATION_SEC lets you specify the maximum expirationSeconds the kubelet can ask for.
  Per default it is hardcoded to a maximum of 367 days, and can be reduced with this parameter.
• --bypass-dns-resolution or BYPASS_DNS_RESOLUTION -> permits to bypass DNS resolution check.
  the default value of the boolean is false, and you can enable it by setting it to true (or any other option listed in GoLang's ParseBool function)
• --bypass-hostname-check or BYPASS_HOSTNAME_CHECK: when set to true, it permits having a DNS name that differs (i.e. isn't prefixed) by the hostname
• --provider-ip-prefixes or PROVIDER_IP_PREFIXES permits to specify a comma-separated list of IP (v4 or/and v6) subnets/prefixes, that CSR IP addresses shall fall into. left unspecified, all IP addresses are allowed.
  you can for example set it to 192.168.0.0/16,fc00::/7 if this reflects your local network IP ranges.
• --ignore-non-system-node or IGNORE_NON_SYSTEM_NODE permits ignoring CSRs with a Username different than system:node:.......
  the default value of the boolean is false, and if you want to use this feature you need to set this flag to true
• --allowed-dns-names or ALLOWED_DNS_NAMES permits allowing more than one DNS name in the certificate request. the default value is set to 1.
• --leader-election or LEADER_ELECTION permits enabling leader election when running with multiple replicas

It is important to understand that the node DNS name needs to be resolvable for the kubelet-csr-approver to work properly. If this is an issue for you, you can use the --bypass-dns-resolution flag, which will disable the DNS check altogether.

ℹ have a look below in this README to understand which other validation mechanisms are put in place.

Helm Install

Adjust providerRegex, providerIpPrefixes and maxExpirationSeconds as needed.

helm repo add kubelet-csr-approver https://postfinance.github.io/kubelet-csr-approver
helm install kubelet-csr-approver kubelet-csr-approver/kubelet-csr-approver -n kube-system \
  --set providerRegex='^node-\w*\.int\.company\.ch$' \
  --set providerIpPrefixes='192.168.8.0/22' \
  --set maxExpirationSeconds='86400'
  --set bypassDnsResolution='false'

Attacker model -- what could go wrong ?

Shall our CSR auto-approver not be implemented correctly, it might permit an attacker to get forged CSRs to be approved and later on signed by the K8s certificate controller.

Indeed, while there are some verifications done by the final certificate signer controller (for the details, here and here), nothing prevents a CSR impersonating a DNSName or an IPAddress from getting signed.

Put more concretely, if a CSR requesting the DNS name auth.company.com or control-plane.k8s.local (or both!) was to be approved, it would get signed and the attacker would have a very valid certificate to make use of. (and depending on the ca.key used on your cluster, this could have a measurable impact)

Which verifications do we put in place ?

Taking inspiration from Kubernetes built-in CSR approver, we check the following criteria:

• CSR.Spec.SignerName must be "kubernetes.io/kubelet-serving"
• CSR.Spec.ExpirationSeconds, if specified, must be smaller than MAX_EXPIRATION_SEC
  (the default value and hard-coded maximum for this controller is 367 days)
• CSR.Spec.Username must be prefixed with system:node: (i.e. we only want to treat CSRs originating from the nodes themselves)
• x509 CR CommonName must be equal to the CSR.Spec.Username
• CSR DNS SubjectAlternativeNames (SAN) contains at most one entry
• at least one SAN IP address or SAN DNS Name must be specified
• CSR SAN DNS Name (if specified) must comply with a provider-specific regex.
• CSR SAN DNS Name (if specified) must be prefixed with the node hostname (where the hostname corresponds to CSR.Spec.Username trimmed of the system:node: prefix)
• CSR SAN IP Addresses must all be part of the set of IP addresses resolved from the SAN DNS Name
• the CSR SAN DNS Name (if specified) must resolve to IP address(es) that fall within the set of provider-specified IP ranges.
• the CSR SAN IP Address(es) must fall within a set of provider-specified IP ranges

With those verifications in place, it makes it quite hard for an attacker to get a forged hostname to be signed, it would indeed require:

• to impersonate a user on the Kubernetes API server with a Username that prefixes the SAN DNS Name request. \ concretely, if the attacker wants to forge a CSR for the auth.company.ch domain, s/he would need to create a CSR with the username system:node:a (remember, we only check the that the DNS name is prefixed by the node name) \ it might then be possible to create a CSR from a node a (not a smart name for a node, I agree), or a node auth, already more plausible
• to modify the provider-specific regex of the kubelet-csr-approver (requires API access or direct access to the node where the controller is running).
  however with API access, the attacker could as well also directly approve the CSR, and with full node access, the attacker could retrieve the controller's ServiceAccount and approve the CSR as well.

Is this CSR approver safe ?

Provided that the provider-specific regex is strict, that the IP ranges set is correctly specified, that the DNS system is not compromised, this automatic CSR approver would make it quite hard for an attacker to start forging CSRs.

Could this CSR approver be safer ?

For sure, this simply requires modifying the ProviderChecks(csr , x509csr)) function to implement additional checks (such as validating the node identity in an external inventory)

Build and development

When building locally to run the CSR approver on an actual cluster with e.g. the oidc authentication provider, you need to use the tag debug to import all authentication providers. You will then build as follows:

go build -tags debug ./cmd/kubelet-csr-approver/